Document bootstrap relay design

1fb6bdc→main·

Soph·3mo ago·2 files·+157 added/-0 removed

Sessions

2dcf2f1a3166View transcript

Changes

2

• MREADME.md+6

• docs

• Abootstrap.md+151

217 unmodified lines

218
219
220
221
222
223
224
225
226

217 unmodified lines

That path exercises real smart HTTP fetch and push with a local bare source repo and a local bare target repo.

Planned Bootstrap Path

There is a planned bootstrap command path for large initial syncs into an empty target. The intent is to relay a fetched source pack directly into target receive-pack instead of decoding the full object graph into local memory first.

The design note is in docs/bootstrap.md.

MREADME.md+6

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151

Bootstrap Design

bootstrap is a planned command path for initial remote-to-remote seeding when the target does not yet contain the managed refs.

The goal is to avoid decoding the fetched source objects into the local in-memory object store during an initial sync. Instead, bootstrap should fetch a pack from the source and relay it directly into target receive-pack.

Why

The current sync path is optimized for general incremental reconciliation:

• it fetches from source with target tip hashes as have
• it builds plans locally
• it stores fetched source objects in a local object store
• it computes the object closure to push
• it encodes a new pack for target

That is a good general path, but it is a poor fit for very large initial syncs into an empty target because the missing object graph must fit in local memory.

bootstrap is meant to cover the opposite case:

• target refs are absent
• all actions are creates
• there is no need for fast-forward checks
• the main cost is moving a large pack from source to target efficiently

V1 Scope

bootstrap should be intentionally narrow:

• create-only
• fail if any managed target ref already exists
• branch refs by default
• optional --tags
• optional explicit --map
• no --force
• no --prune
• no mixed create and update runs
• no automatic fallback to normal sync
• smart HTTP only

This command is for first-time seeding. After that, operators should use sync.

Command Shape

Preferred CLI:

git-sync bootstrap [flags] <source-url> <target-url>

Expected v1 flags:

• --branch
• --map
• --tags
• --stats
• --json
• --protocol auto|v1|v2
• existing source and target auth flags

Intended Flow

1. List source refs.
2. List target refs.
3. Build the managed ref set from --branch, --map, and --tags.
4. Fail if any managed target ref already exists.
5. Build create commands for the target.
6. Ask source for a pack containing the selected source tips.
7. Strip protocol framing and sideband as needed.
8. Stream the resulting pack directly into target receive-pack.
9. Parse target report-status and return a create summary.

Why This Helps

The large memory cost in the current implementation comes from storing fetched source objects locally before re-encoding them.

bootstrap should avoid that cost for initial syncs by not materializing the object graph in local storage unless a fallback path is explicitly chosen later.

The expected wins are:

• much lower RAM usage for empty-target syncs
• less local CPU spent decoding and re-encoding large object graphs
• better fit for large repo migrations

Constraints

There are still some hard limits:

• source and target still need normal smart HTTP discovery
• target policy can still reject pushes
• push still depends on target receive-pack behavior and capabilities
• if a relay-safe path cannot be used, bootstrap should fail and tell the user to use sync

V1 should stay strict rather than trying to be clever.

Implementation Notes

The cleanest implementation shape is a separate code path, not an optimization hidden inside sync.

Suggested pieces:

• runBootstrap in cmd/git-sync/main.go
• syncer.Bootstrap(ctx, cfg) in internal/syncer
• source fetch helper that returns a pack stream instead of writing objects into storage
• target receive-pack helper that accepts an externally supplied pack stream
• bootstrap-specific result type or reuse Result with only create actions

The initial implementation should prefer:

• one multi-ref source fetch
• one multi-command target push

That keeps it efficient and conceptually simple.

Failure Rules

V1 should fail when:

• any managed target ref already exists
• no source refs matched
• the source fetch cannot be relayed cleanly
• target push fails

The error should explicitly recommend normal sync when the repository is no longer in bootstrap shape.

Follow-Up Steps

Phase 1:

• implement bootstrap for create-only branch refs
• support optional tag creation
• add JSON and stats output
• add in-process integration tests
• add git-http-backend integration coverage for empty-target bootstrap

Phase 2:

• allow relay-safe create-only runs with explicit mapped refs
• add better operator output for large initial transfers
• add safety thresholds for advertised/fetched bytes

Phase 3:

• investigate hybrid behavior: relay when the target is empty, otherwise fail fast into normal sync
• investigate whether target capability combinations require alternate pack handling
• measure source-to-target pack relay memory and CPU against current sync

Phase 4:

• consider a more advanced incremental relay mode for non-empty targets
• only pursue this if large migration workflows become important enough to justify the added protocol complexity