TL;DR

I am pleased to announce the release of filepack 0.0.10.

Filepack is a content-addressed package format, metadata standard, command-line utility, and package server.

The filepack binary is written in Rust and can be used to create, sign, and verify packages, and includes a package server with an attractive front end, as well as commands to upload and download packages.

A demo server with a nice collection of packages is available at filepack.com.

It is already featureful and useful, but the package and metadata formats are not stable, and as yet there is no guarantee of backwards compatibility.

Lofty Ambitions

The goal of filepack is to make decentralized filesharing as appealing and easy to use as centralized services. When you share audio, images, and video they should look good and be easy to search and play.

Existing filesharing technologies are hard to use because they transfer unstructured data: individual files or directories of files. With no structure or metadata, many desirable features are hard or impossible to build. For example, structured search, graphical user interfaces with package artwork, and automatically locating, playing, and transcoding media.

Filepack packages include structured, machine-readable metadata describing the package and its contents. This allows them to be presented in an attractive way, supports structured search, and for any media in the package to be automatically previewed and played.

All metadata displayed by the filepack.com demo server is served directly from the metadata included in each package. This includes package titles, artwork, image thumbnails, and video placeholder images. Filepack metadata also identifies the media type of the package and each individual media item, including its file format, and details like resolution, codec, and duration.

Inscrutable Details

Filepack packages are content-addressed Merkle trees, identified by a root hash called the package fingerprint. The inner hashes of the Merkle tree are BLAKE3 hashes, itself a tree hash, allowing for incremental transfer and verification. Packages may be signed using Ed25519.

Packages include both content and metadata, so a package can be downloaded with filepack download --package FINGERPRINT and uploaded to a new server with filepack upload --server URL.

Because filepack packages are normal directories of files, they can be shared with other filesharing technologies, like BitTorrent, Usenet, and ordinary web servers.

Metadata is optional, so filepack can also be used as a general purpose directory hashing, signing, and verification tool.

Filepack includes a web package type, which can served by the filepack server and have access to the server API. This allows creating and distributing web apps that can access and display the packages on a server.

Exhortations

I've been slaving away for months on this thing, so I would absolutely love it, and you, if you gave filepack a whirl.

You can download the latest release from GitHub or build it from source with cargo install filepack.

Try creating a package and running the server, for yourself or others.

Filepack only supports a limited number of media types and file formats, but I'm very happy to add more if there's demand.

Please feel free to report issues or request new features on GitHub, or just email me.

Future Directions

The next thing I want to work on is guaranteeing backwards compatibility, or at least understanding what needs to be done before backwards compatibility can be guaranteed.

Filepack defines a bunch of different formats, including the manifest and metadata file formats, Merkle tree structure, and signature scheme. I've constantly been making backwards incompatible changes to these formats in the process of adding features and fixing bugs. But, for filepack to be useful, these formats must reach a point where I can guarantee that packages created by old versions of filepack will work with future versions of filepack.

After that, I want to write comprehensive documentation, including specifications of the data formats, explanations of the design choices, and guides for things like how to create good packages and run a server.

And finally, after that, there are a whole bunch of things I'd like to do. In no particular order:

• Reach out to filesharing and archival communities, and add features that make filepack useful for them.

• Add more media types and file formats, and expand existing media types.

• Add subtypes that change how media is displayed. For example, a webtoon subtype that can be applied to image packages which causes the images to be displayed with a continuous vertical scrolling layout.

• Improve the web package type, and implement a single-page application media player that can be served from the demo server.

Eventually, I'd like to create a filepack-native peer-to-peer filesharing network, with a rich client that's much more like a media player than a filesharing application. But that's a long way off.

Closing Remarks

Filepack is something that I've been thinking about and working on, in one form or another, for quite a while. I'm very happy to finally be able to release something that at least starts to articulate the vision of the project. There's still a long way to go, but hopefully it's useful in its current form and someone out there appreciates the ideas that it embodies.