git-refs: shard every checkpoint ID on the last two chars (one rule for both formats) · Entire

git-refs: shard every checkpoint ID on the last two chars (one rule for both formats)

02e2f08·

Soph·2w ago·5 files·+31 added/-37 removed

ShardFor previously branched on the ID Kind — legacy hex on the first two chars, ULID on the last two — which meant the ref shard depended on format detection and could be computed inconsistently by an independent reimplementation (the e2e helper sharded any 26-char string as a ULID, diverging from production's strict check for a 26-char non-ULID).

Collapse it to a single positional rule: the last two characters, for both legacy 12-hex IDs and 26-char ULIDs. Trailing chars are uniformly random in both formats (a ULID's leading chars are its timestamp), so distribution stays even, and there is no Kind branch to get wrong. RefName still rejects KindUnknown IDs, so both supported formats map to a proper ref while invalid ones are refused.

This only affects the git-refs ref namespace (no production data yet); the entire/checkpoints/v1 branch tree keeps its independent first-two Path() layout. The e2e checkpointShard helper is simplified to the same last-two rule so it can't diverge from production. Updates the affected shard expectations in the id, refs-naming, and refs-store tests.

Sessions

d17074281105View transcript

Changes

5

// ShardFor returns the two-character shard for storing this ID under a
// per-checkpoint git ref (refs/entire/checkpoints/<shard>/<id>), chosen so
// checkpoints spread evenly across buckets:
// per-checkpoint git ref (refs/entire/checkpoints/<shard>/<id>): the LAST two
// characters of the ID, for BOTH supported formats.
//   - Legacy hex IDs shard on the FIRST two characters, preserving the existing
//     entire/checkpoints/v1 tree layout (see Path).
//   - ULIDs shard on the LAST two characters: a ULID's leading characters encode
//     its timestamp and barely vary between nearby checkpoints, while the trailing
//     characters are random, so the suffix spreads evenly while the ID itself
//     stays lexicographically sortable.
// A single positional rule (independent of the ID's Kind) keeps ref naming
// robust for legacy and ULID IDs alike and impossible to compute inconsistently
// between callers. The suffix spreads checkpoints evenly across buckets for
// either format: a legacy hex ID is random throughout, and a ULID's leading
// characters encode its timestamp (barely varying between nearby checkpoints)
// while its trailing characters are random — so sharding on the suffix keeps the
// distribution even while the ID itself stays lexicographically sortable.
// For an ID shorter than two characters the whole ID is returned; an unrecognized
// ID falls back to the first-two (prefix) layout.
// This is the git-refs ref namespace only; the entire/checkpoints/v1 branch tree
// keeps its own independent first-two layout (see Path). For an ID shorter than
// two characters the whole ID is returned.
func (id CheckpointID) ShardFor() string {
    s := string(id)
    if len(s) < 2 {
        return s
    }
    if id.Kind() == KindULID {
        return s[len(s)-2:]
    }
    return s[:2]
}

// NewCheckpointID creates a CheckpointID from a string, validating its format.

    input string
    want  string
    }{
    // Legacy hex shards on the first two chars (preserves the v1 layout).
    {"legacy", "a1b2c3d4e5f6", "a1"},
    {"legacy other", "abcdef123456", "ab"},
    // ULID shards on the LAST two chars.
    // Every format shards on the LAST two chars (single positional rule).
    {"legacy", "a1b2c3d4e5f6", "f6"},
    {"legacy other", "abcdef123456", "56"},
    {"ulid", sampleULID, "ZN"},
    {"ulid trailing", "0123456789ABCDEFGHJKMNPQRS", "RS"},
    // Unknown falls back to the prefix (first-two) layout.
    {"unknown", "XYZ", "XY"},
    {"unknown", "XYZ", "YZ"},
    // Short-string fallbacks.
    {"empty", "", ""},
    {"one char", "a", "a"},

func UsingGitRefs() bool { return checkpointStoreMode() == storeModeGitRefs }

// checkpointShard returns the two-char ref shard for a checkpoint ID, matching
// cmd/entire/cli/checkpoint/id.ShardFor: the first two chars for a 12-hex legacy
// ID, the last two for a 26-char ULID.
// cmd/entire/cli/checkpoint/id.ShardFor: the last two chars, for both legacy
// 12-hex IDs and 26-char ULIDs.
func checkpointShard(id string) string {
    if len(id) < 2 {
        return id
    }
    if len(id) == 26 {
        return id[len(id)-2:]
    }
    return id[:2]
}