Polish bootstrap batch guidance · Entire

Polish bootstrap batch guidance

5806892→main·

Soph·3mo ago·4 files·+28 added/-9 removed

Sessions

2fd4174c6988View transcript

Changes

4

93 unmodified lines

94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117

93 unmodified lines

- temporary refs under `refs/gitsync/bootstrap/heads/`
- resume from existing temp refs is supported when they match a planned checkpoint

This mode is intended as an advanced large-repo fallback, not the default bootstrap path. Use plain `bootstrap` first when a single streamed initial sync is acceptable.

A practical starting point is:

- `--batch-max-pack-bytes 536870912` for a conservative `512 MiB` target-side batch size
- `--batch-max-pack-bytes 1073741824` when you want fewer, larger batches and the target has more headroom

For example:

```bash
go run ./cmd/git-sync bootstrap \
  --batch-max-pack-bytes 536870912 \
  --protocol v2 \
  -v \
<source-url> \
<target-url>
```

Add `--measure-memory` to `bootstrap`, `sync`, `plan`, `probe`, or `fetch` to sample elapsed time and Go heap usage:

```bash

MREADME.md+18

253 unmodified lines

254
255
256
257
258
259
260
261
262
263
264
265
266

253 unmodified lines

- currently requires source-side protocol v2 with fetch filter support
- resumes from an existing temp ref when that temp ref matches a planned checkpoint
- exercised by `TestBootstrap_GitHTTPBackendBatchedBranch`
- validated against `torvalds/linux` as a large-source manual stress path

Operator guidance:

- prefer plain `bootstrap` first
- use batching when a single large bootstrap push is too risky, too large, or fails on the target side
- start with `--batch-max-pack-bytes 536870912` and adjust upward only if the target has enough headroom

Phase B:

Mdocs/bootstrap-batching.md+7

171 unmodified lines

172
173
174
175
176
177
178

171 unmodified lines

- it batches branch refs and then optionally creates tags after the branch batches complete
- it resumes from an existing temp ref when that temp ref matches a planned checkpoint
- it uses temporary target refs under `refs/gitsync/bootstrap/heads/`
- it should be treated as an advanced large-repo fallback when one-shot bootstrap is too risky or fails on target-side unpack/index pressure

Phase 4:

Mdocs/bootstrap.md+1

1212 unmodified lines

1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1216
1217
1218
1219
1220
1228
1221
1222
1230
1223
1224
1225
1226

1212 unmodified lines

prevSpan int,
) (int, error) {
    return sampledCheckpointUnderLimitByProbe(chain, prevIdx, prevSpan, func(idx int) (bool, error) {
        progressf(
            cfg.Verbose,
            "bootstrap-batch: branch=%s probe checkpoint=%s base=%s",
            ref.TargetRef,
            shortHash(chain[idx]),
            shortHash(prevHash),
        )
        ooLarge, err := sourcePackExceedsLimit(ctx, sourceConn, sourceService, ref, chain[idx], prevHash, cfg.BatchMaxPackBytes)
        if err != nil {
            return false, fmt.Errorf("measure bootstrap batch for %s at %s: %w", ref.TargetRef, shortHash(chain[idx]), err)
        }
        if tooLarge {
            progressf(cfg.Verbose, "bootstrap-batch: checkpoint=%s exceeds limit=%d", shortHash(chain[idx]), cfg.BatchMaxPackBytes)
            progressf(cfg.Verbose, "bootstrap-batch: sample %s exceeds limit=%d", shortHash(chain[idx]), cfg.BatchMaxPackBytes)
        } else {
            progressf(cfg.Verbose, "bootstrap-batch: checkpoint=%s fits limit=%d", shortHash(chain[idx]), cfg.BatchMaxPackBytes)
            progressf(cfg.Verbose, "bootstrap-batch: sample %s fits limit=%d", shortHash(chain[idx]), cfg.BatchMaxPackBytes)
        }
        return tooLarge, nil
    })
}`

Minternal/syncer/syncer.go+2/-9