Document Phase A bootstrap batching · Entire
Document Phase A bootstrap batching
9e1e66d→main·
Soph·3mo ago·3 files·+51 added/-2 removed
Sessions
5cc48c743bfdView transcript
Changes
3
- MREADME.md+27/-1
- docs
- Mbootstrap-batching.md+6
- Mbootstrap.md+18/-1
76 unmodified lines
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
70 unmodified lines
170
171
172
173
174
175
176
97 unmodified lines
274
275
276
277
278
279
280
281
282
283
284
285
4 unmodified lines
290
291
292
269
293
294
295
296
297
76 unmodified lines
<target-url>
Add --batch-max-pack-bytes to split large branch bootstraps into multiple relay batches with temporary refs:
go run ./cmd/git-sync bootstrap \
--batch-max-pack-bytes 1073741824 \
<source-url> \
<target-url>
Current batching scope is intentionally narrow:
- protocol v2 only
- branch refs only
- no tags
- no resume
- temporary refs under
refs/gitsync/bootstrap/heads/
Add --measure-memory to bootstrap, sync, plan, probe, or fetch to sample elapsed time and Go heap usage:
70 unmodified lines
- `probe` returns top-level keys such as `source_url`, `target_url`, `protocol`, `ref_prefixes`, `source_capabilities`, `target_capabilities`, `refs`, and `stats`
- `fetch` returns top-level keys such as `source_url`, `protocol`, `wants`, `haves`, `fetched_objects`, and `stats`
- `bootstrap`, `plan`, and `sync` return top-level keys such as `plans`, `pushed`, `skipped`, `blocked`, `deleted`, `dry_run`, `protocol`, and `stats`
- `bootstrap`, `plan`, and `sync` also expose `relay`, `relay_mode`, `relay_reason`, `batching`, and `batch_count`
- each item in `plans` includes stable string fields such as `branch`, `source_ref`, `target_ref`, `source_hash`, `target_hash`, `kind`, `action`, and `reason`
Probe both source and target remotes to inspect source fetch capabilities and target `receive-pack` capabilities:
97 unmodified lines
That path exercises real smart HTTP fetch and push with a local bare source repo and a local bare target repo.
The Phase A batching path also has a dedicated `git-http-backend` test:
```bash
env GOCACHE=/tmp/go-build GITSYNC_E2E_GIT_HTTP_BACKEND=1 go test ./internal/syncer -run TestBootstrap_GitHTTPBackendBatchedBranch -v
There is also an optional live Linux bootstrap smoke against the public Linux repository:
4 unmodified lines
## Planned Bootstrap Path
There is a planned `bootstrap` command path for large initial syncs into an empty target. The intent is to relay a fetched source pack directly into target `receive-pack` instead of decoding the full object graph into local memory first.
There is a dedicated `bootstrap` command path for large initial syncs into an empty target. The intent is to relay a fetched source pack directly into target `receive-pack` instead of decoding the full object graph into local memory first.
The design note is in [docs/bootstrap.md](/Users/soph/Work/entire/devenv/git-sync/docs/bootstrap.md).
For very large single-branch repositories, there is also a batching design and initial implementation note in [docs/bootstrap-batching.md](/Users/soph/Work/entire/devenv/git-sync/docs/bootstrap-batching.md).
MREADME.md+27/-1
247 unmodified lines
248
249
250
251
252
253
254
255
256
257
258
259
247 unmodified lines
- no resume
- manual cleanup if interrupted
Progress:
- implemented via `git-sync bootstrap --batch-max-pack-bytes`
- currently requires source-side protocol v2 with fetch filter support
- exercised by `TestBootstrap_GitHTTPBackendBatchedBranch`
Phase B:
- add resume from existing temp refs
Mdocs/bootstrap-batching.md+6
1
2
3
3
4
5
6
127 unmodified lines
134
135
136
137
138
139
140
141
142
143
144
145
146
147
4 unmodified lines
152
153
154
155
156
157
158
6 unmodified lines
165
166
167
168
169
170
171
172
173
174
175
176
177
178
# Bootstrap Design
`bootstrap` is a planned command path for initial remote-to-remote seeding when the target does not yet contain the managed refs.
`bootstrap` is the dedicated command path for initial remote-to-remote seeding when the target does not yet contain the managed refs.
The goal is to avoid decoding the fetched source objects into the local in-memory object store during an initial sync. Instead, `bootstrap` should fetch a pack from the source and relay it directly into target `receive-pack`.
127 unmodified lines
- add in-process integration tests
- add `git-http-backend` integration coverage for empty-target bootstrap
Progress:
- `bootstrap` is implemented
- optional tag creation is supported on the non-batched path
- JSON and stats output are supported
- in-process integration coverage exists
- `git-http-backend` integration coverage exists
Phase 2:
- allow relay-safe create-only runs with explicit mapped refs
4 unmodified lines
- explicit mapped refs are supported
- `--max-pack-bytes` provides a first safety threshold for the streamed source pack during bootstrap
- `--batch-max-pack-bytes` now enables a Phase A batched branch-only bootstrap mode for large initial syncs
Phase 3:
6 unmodified lines
- `sync` now auto-selects the bootstrap relay path when all managed target refs are absent and the run matches bootstrap semantics
- dry-run `plan` surfaces a bootstrap suggestion for the same target shape
Batching note:
- the current batched bootstrap path is intentionally narrow
- it requires source-side protocol v2 with fetch filters
- it supports branch refs only
- it does not support tags or resume yet
- it uses temporary target refs under `refs/gitsync/bootstrap/heads/`
Phase 4:
- consider a more advanced incremental relay mode for non-empty targets