Document bootstrap batching design

Sessions

a2d74df554b4View transcript

Changes

1

• docs

• Abootstrap-batching.md+264

Bootstrap Batching Design

bootstrap currently streams one source pack into one target push. That is good for many initial syncs, but it is not enough for very large single-branch repositories where one initial pack is itself too large for comfortable target-side unpacking and indexing.

This note sketches a batching design for large bootstrap jobs.

Goal

Reduce per-push size and target-side receive-pack / index-pack pressure for very large initial syncs, while preserving the main benefit of bootstrap:

• no full local object materialization in git-sync
• direct source-to-target relay
• clear operator-visible progress and restart points

Non-Goals

V1 batching should not try to solve every large-migration problem.

Out of scope:

• non-empty target incremental relay batching
• prune/delete behavior
• tag retargeting
• fully optimal pack-size planning
• arbitrary graph partitioning

Why Per-Ref Batching Is Not Enough

Per-ref batching is easy:

• push refs/heads/main
• then refs/heads/release
• then tags

That helps when there are many refs and each ref is moderate in size.

It does not help for repositories where a single branch is enormous. Linux master is the motivating example: even a single branch bootstrap can be too large for one target-side unpack/index step.

Preferred Model

Use branch checkpoint batching with temporary refs.

High-level idea:

1. Choose a sequence of ancestor checkpoints for a source branch.
2. Push them oldest to newest into a temporary target ref.
3. Once the final tip is present, create the real target ref.
4. Delete the temporary ref at the end.

This gives:

• bounded per-push transfer size
• bounded target-side unpack/index work per batch
• restart points between batches
• no partially initialized real branch refs visible unless the run finishes

Command Shape

Possible CLI extension:

git-sync bootstrap \
  --batch-max-pack-bytes 1073741824 \
<source-url> \
<target-url>

Possible related flags:

• --batch-max-pack-bytes
• --batch-ref-prefix refs/gitsync/bootstrap/
• --keep-temp-refs-on-failure

The first version should only need --batch-max-pack-bytes.

Temporary Ref Strategy

For each target branch:

• real target ref: refs/heads/main
• temp bootstrap ref: refs/gitsync/bootstrap/heads/main

Flow:

1. Create/update only the temp ref during intermediate batches.
2. After the final tip batch succeeds:
   • create the real ref at the final tip
   • delete the temp ref

This keeps the target repository in a cleaner state:

• before completion, the real branch is absent
• after completion, only the final branch remains

If failure happens mid-run:

• the temp ref records progress
• the real ref is still absent

Checkpoint Selection

The most practical first heuristic is first-parent checkpointing.

For a branch tip:

1. Walk first-parent ancestry backward.
2. Sample candidate checkpoint commits.
3. Starting from the oldest candidate, estimate each batch by doing a source fetch against the previous checkpoint as have.
4. Pick the largest checkpoint whose pack stays under --batch-max-pack-bytes.
5. Repeat until the branch tip is reached.

This is only a heuristic:

• actual pack size depends on delta choices and object reuse
• merges and deep side histories can make batch sizes uneven

But it is much simpler than exact graph partitioning and good enough for a first implementation.

Batch Flow For One Branch

Given checkpoints:

• C1
• C2
• C3
• tip

The flow is:

1. Fetch source pack for C1 with no have.
2. Push it to temp ref refs/gitsync/bootstrap/heads/main.
3. Fetch source pack for C2 with have=C1.
4. Push update of temp ref to C2.
5. Fetch source pack for C3 with have=C2.
6. Push update of temp ref to C3.
7. Fetch source pack for tip with have=C3.
8. Push update of temp ref to tip.
9. Create real target ref refs/heads/main at tip.
10. Delete temp ref.

The final ref creation can be:

• one separate tiny push with no pack
• or combined with the last batch if command ordering and push semantics stay clear

The first version should prefer the separate final ref creation because it is easier to reason about.

Tags

Tags should not be pushed during intermediate branch batches.

Recommended rule:

• branch batches first
• tag creation only after all referenced branch/object checkpoints complete

This avoids cases where a tag points at an object graph that is not yet fully present on target.

V1 batching should support:

• branch refs only

Tag batching can be added later.

Restart and Recovery

Batching is only worth doing if failures are restartable.

Minimum restart model:

1. Detect existing temp refs on target.
2. Resolve their current hashes.
3. Resume from the latest completed checkpoint instead of starting from zero.

If --keep-temp-refs-on-failure is false, cleanup can still happen on clean failures, but default restartability is more valuable than aggressive cleanup.

Safety Model

V1 batching should remain strict.

Allow only:

• empty managed target refs
• branch-only bootstrap
• no force
• no prune
• no existing real target refs for the managed branches

Fail if:

• temp refs already exist but do not match expected checkpoint progression
• estimated batch sizing cannot find a checkpoint under the configured limit
• final ref cutover fails

Implementation Shape

Suggested pieces:

• BootstrapBatch execution path in internal/syncer
• checkpoint planner:
  • first-parent ancestry walker
  • batch-size estimator
• temp ref naming helpers
• resume detector for existing temp refs
• final cutover helper

The estimator should reuse the existing relay mechanics:

• source fetch with have
• streamed push to target

But it will need one new planning pass to probe likely batch sizes before actual execution.

Operator Output

Batching should be explicit in output.

Text output should include:

• batching=true
• per-branch checkpoint count
• current batch number
• temp ref names when verbose

JSON should include:

• batching
• batch_count
• completed_batches
• temp_refs

Practical Risks

• pack-size estimation may require extra source fetches before actual execution
• checkpoint search may be slow on very deep histories
• target-side unpack/index cost may still be high even after batching, just smaller
• temp refs add cleanup and restart complexity

This is still likely worthwhile for very large initial migrations because it changes a single huge risky operation into several bounded ones.

Recommended Phases

Phase A:

• batch branch-only bootstrap
• no tags
• temp refs required
• no resume
• manual cleanup if interrupted

Phase B:

• add resume from existing temp refs
• add better progress reporting
• add batch-size estimation metrics

Phase C:

• consider tag creation after successful branch completion
• consider whether per-ref or per-branch parallelism is worth it

Phase D:

• only then consider using similar checkpoint batching ideas for non-empty target incremental relay