Free commit graph store immediately after extracting first-parent chain

05d5f1a→main·

Soph·3mo ago·2 files·+38 added/-0 removed

The commit graph fetch for linux stores 1.4M commit objects in memory.NewStorage() (4.6 GB). We only need the 75k first-parent chain hashes (3 MB) for checkpoint placement. Previously the store stayed alive for the entire bootstrap run because the function scope held it.

Now: extract the chain, nil the store, force GC. The 4.6 GB becomes garbage-collectible immediately after chain extraction (~22s into the run) instead of persisting for the full ~17 minutes.

The transient spike during the fetch is unavoidable with current tools:

• git protocol v2 has no "first-parent only" fetch command
• tree:0 filter gives all 1.4M commits, not just the 75k first-parent
• go-git's pack parser needs the full store for delta resolution between commits during pack decoding

A streaming parent-map extractor could eliminate the spike entirely but requires go-git packfile internals work (custom storer or scanner integration). The GC approach is the pragmatic fix.

Also adds FirstParentChainFromMap to planner/checkpoint.go for future use when a streaming extractor is implemented.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Sessions

56d371ce68d4View transcript

Changes

2

• internal

  • planner

    • Mcheckpoint.go+28

  • strategy/bootstrap

    • Mbootstrap.go+10

41 unmodified lines

return chain, nil
}

// FirstParentChainFromMap walks a first-parent map from tip back to root.
// The map key is a commit hash, the value is its first parent hash.
// A zero-value parent marks the root. Returns the chain in root-to-tip order.
func FirstParentChainFromMap(parents map[plumbing.Hash]plumbing.Hash, tip plumbing.Hash) ([]plumbing.Hash, error) {
    chain := make([]plumbing.Hash, 0, len(parents))
    current := tip
    seen := make(map[plumbing.Hash]struct{}, len(parents))
    for {
        if _, ok := seen[current]; ok {
            return nil, fmt.Errorf("cycle detected at %s", current)
        }
        seen[current] = struct{}{}
        chain = append(chain, current)
        parent, ok := parents[current]
        if !ok {
            return nil, fmt.Errorf("commit %s not found in parent map", current)
        }
        if parent.IsZero() {
            break
        }
        current = parent
        }
    for i, j := 0, len(chain)-1; i < j; i, j = i+1, j-1 {
        chain[i], chain[j] = chain[j], chain[i]
    }
    return chain, nil
}

// SampledCheckpointCandidates generates a set of candidate indices to probe,
// sorted from largest (preferred) to smallest.
func SampledCheckpointCandidates(lo, hi int, prevSpan int) []int {

Minternal/planner/checkpoint.go+28

11 unmodified lines

func planCheckpointsFromChain(ctx context.Context, p Params, ref planner.DesiredRef) ([]plumbing.Hash, []plumbing.Hash, error) {
    p.log("bootstrap batch fetching commit graph", "branch", ref.TargetRef.String())

// Fetch all commits (tree:0 filter) into a temporary in-memory store.
    // For linux this is ~1.4M commits at ~3.3 KiB each = ~4.6 GB transient.
    // We extract the first-parent chain (~75k hashes, ~3 MB) immediately
    // and discard the store so GC can reclaim the 4.6 GB. The transient
    // spike is unavoidable with git protocol v2 (no first-parent-only
    // fetch) and go-git's pack parser (needs full store for delta resolution).
    graphStore := memory.NewStorage()
    gpRef := gitproto.DesiredRef{SourceRef: ref.SourceRef, TargetRef: ref.TargetRef, SourceHash: ref.SourceHash}
    if err := p.SourceService.FetchCommitGraph(ctx, graphStore, p.SourceConn, gpRef); err != nil {
        return nil, nil, fmt.Errorf("fetch bootstrap planning graph for %s: %w", ref.TargetRef, err)
    }
    chain, err := planner.FirstParentChain(graphStore, ref.SourceHash)
    graphStore = nil // allow GC to reclaim the commit graph store (~4.6 GB for linux)
    runtime.GC()
    if err != nil {
        return nil, nil, fmt.Errorf("walk first-parent chain for %s: %w", ref.TargetRef, err)
    }
}

Minternal/strategy/bootstrap/bootstrap.go+10