multi-pack-index

A list of packfile names.

A sorted list of object IDs.

A list of metadata for the ith object ID including:

If large offsets are required, we use another list of large offsets similar to version 2 pack-indexes.

The MIDX is stored in a file named multi-pack-index in the .git/objects/pack directory. This could be stored in the pack directory of an alternate. It refers only to packfiles in that same directory.

The core.multiPackIndex config setting must be on (which is the default) to consume MIDX files. Setting it to false prevents Git from reading a MIDX file, even if one exists.

The file format includes parameters for the object ID hash function, so a future change of hash algorithm does not require a change in format.

The MIDX keeps only one record per object ID. If an object appears in multiple packfiles, then the MIDX selects the copy in the preferred packfile, otherwise selecting from the most-recently modified packfile.

If there exist packfiles in the pack directory not registered in the MIDX, then those packfiles are loaded into the packed_git list and packed_git_mru cache.

The pack-indexes (.idx files) remain in the pack directory so we can delete the MIDX file, set core.midx to false, or downgrade without any loss of information.

The MIDX file format uses a chunk-based approach (similar to the commit-graph file) that allows optional data to be added.

The multi-pack-index allows many packfiles, especially in a context where repacking is expensive (such as a very large repo), or unexpected maintenance time is unacceptable (such as a high-demand build machine). However, the multi-pack-index needs to be rewritten in full every time. We can extend the format to be incremental, so writes are fast. By storing a small "tip" multi-pack-index that points to large "base" MIDX files, we can keep writes fast while still reducing the number of binary searches required for object lookups.

If the multi-pack-index is extended to store a "stable object order" (a function Order(hash) = integer that is constant for a given hash, even as the multi-pack-index is updated) then MIDX bitmaps could be updated independently of the MIDX.

Packfiles can be marked as "special" using empty files that share the initial name but replace ".pack" with ".keep" or ".promisor". We can add an optional chunk of data to the multi-pack-index that records flags of information about the packfiles. This allows new states, such as repacked or redeltified, that can help with pack maintenance in a multi-pack environment. It may also be helpful to organize packfiles by object type (commit, tree, blob, etc.) and use this metadata to help that maintenance.