git-refs: shard every checkpoint ID on the last two chars (one rule for both formats) · Entire

git-refs: shard every checkpoint ID on the last two chars (one rule for both formats)

dd6fdeb→main·

Soph·2w ago·5 files·+31 added/-37 removed

ShardFor previously branched on the ID Kind — legacy hex on the first two chars, ULID on the last two — which meant the ref shard depended on format detection and could be computed inconsistently by an independent reimplementation (the e2e helper sharded any 26-char string as a ULID, diverging from production's strict check for a 26-char non-ULID).

Collapse it to a single positional rule: the last two characters, for both legacy 12-hex IDs and 26-char ULIDs. Trailing chars are uniformly random in both formats a ULID's leading chars are its timestamp, so distribution stays even, and there is no Kind branch to get wrong. RefName still rejects KindUnknown IDs, so both supported formats map to a proper ref while invalid ones are refused.

This only affects the git-refs ref namespace (no production data yet); the entire/checkpoints/v1 branch tree keeps its independent first-two Path() layout. The e2e checkpointShard helper is simplified to the same last-two rule so it can't diverge from production. Updates the affected shard expectations in the id, refs-naming, and refs-store tests.

Resolves the git-refs sharding review thread (e2e/testutil/backend.go).

Sessions

d17074281105View transcript

Changes

5

93 unmodified lines

// ShardFor returns the two-character shard for storing this ID under a
// per-checkpoint git ref (refs/entire/checkpoints/<shard>/<id>), chosen so
// checkpoints spread evenly across buckets:
// per-checkpoint git ref (refs/entire/checkpoints/<shard>/<id>): the LAST two
// characters of the ID, for BOTH supported formats.
//
//   - Legacy hex IDs shard on the FIRST two characters, preserving the existing
//     entire/checkpoints/v1 tree layout (see Path).
//   - ULIDs shard on the LAST two characters: a ULID's leading characters encode
//     its timestamp and barely vary between nearby checkpoints, while the trailing
//     characters are random, so the suffix spreads evenly while the ID itself
//     stays lexicographically sortable.
// A single positional rule (independent of the ID's Kind) keeps ref naming
// robust for legacy and ULID IDs alike and impossible to compute inconsistently
// between callers. The suffix spreads checkpoints evenly across buckets for
// either format: a legacy hex ID is random throughout, and a ULID's leading
// characters encode its timestamp (barely varying between nearby checkpoints)
// while its trailing characters are random — so sharding on the suffix keeps the
// distribution even while the ID itself stays lexicographically sortable.
//
// For an ID shorter than two characters the whole ID is returned; an unrecognized
// ID falls back to the first-two (prefix) layout.
// This is the git-refs ref namespace only; the entire/checkpoints/v1 branch tree
// keeps its own independent first-two layout (see Path). For an ID shorter than
// two characters the whole ID is returned.
func (id CheckpointID) ShardFor() string {
    s := string(id)
    if len(s) < 2 {
        return s
    }
    if id.Kind() == KindULID {
        return s[len(s)-2:]
    }
    return s[:2]
}

Mcmd/entire/cli/checkpoint/id/id.go+13/-14

158 unmodified lines

// Test cases for ShardFor function.
{
    "legacy": {"a1b2c3d4e5f6", "a1"},
    "legacy other": {"abcdef123456", "ab"},
    "ulid": {"0123456789ABCDEFGHJKMNPQRS", "RS"},
    "unknown": {"XYZ", "XY"},
    "empty": {"", ""},
    {"one char", "a", "a"}
}

Mcmd/entire/cli/checkpoint/id/id_test.go+4/-6

37 unmodified lines

// Struct for test case entries.
{
    {name: "legacy hex shards on first two", cid: "a1b2c3d4e5f6", want: "refs/entire/checkpoints/a1/a1b2c3d4e5f6"},
    {name: "legacy hex shards on last two", cid: "a1b2c3d4e5f6", want: "refs/entire/checkpoints/f6/a1b2c3d4e5f6"}
}

Mcmd/entire/cli/checkpoint/refs_naming_test.go+6/-6

37 unmodified lines

// New test cases for different ID types and their expected refs.
{
    {name: "legacy round-trip", ref: "refs/entire/checkpoints/a1/a1b2c3d4e5f6"},
    {name: "ulid round-trip", ref: "refs/entire/checkpoints/f6/a1b2c3d4e5f6"}
}

Mcmd/entire/cli/checkpoint/refs_store_test.go+5/-5

37 unmodified lines

// Function to determine if using Git refs.
func UsingGitRefs() bool { return checkpointStoreMode() == storeModeGitRefs }