Merge pull request #4 from entirehq/soph/replicate · Entire
Log in
Merge pull request #4 from entirehq/soph/replicate
b0f3048→main·
Soph·3mo ago·36 files·+2,325 added/-821 removed
Add replicate mode: source-authoritative relay-only replication with adaptive batching
Changes
36
ACHANGELOG.md+56
MREADME.md+31/-8
cmd
git-sync-bench
Mmain.go+2/-2
git-sync
Mmain.go+45/-6
Mmain_test.go+164/-3
docs
Marchitecture.md+17/-7
Mbenchmarking.md+1/-1
Mbootstrap-batching.md+27/-17
Mbootstrap.md+1/-1
Membedding.md+5/-2
Mtesting.md+1/-1
internal
gitproto
Mfetch.go+80/-19
Mfetch_test.go+63/-13
Mpush.go+64/-9
Mpush_test.go+63/-1
Mrefs.go+4
planner
Mcheckpoint.go+28
Mplanner.go+99
Mplanner_test.go+126
Mrelay.go+46
strategy
bootstrap
Mbootstrap.go+245/-307
Mbootstrap_test.go+221/-340
replicate
Areplicate.go+107
Areplicate_test.go+118
syncer
Mentire_local_smoke_test.go+2/-2
Mgit_http_backend_test.go+5/-5
Mintegration_test.go+450/-56
Mlive_bootstrap_test.go+1/-1
Msyncer.go+171/-15
pkg/gitsync
Mclient.go+22
Mclient_test.go+11
internalbridge
Mconfig.go+13
Mmodel.go+4/-2
Mmodel_test.go+2/-1
Mtypes.go+9
unstable
Mclient.go+21/-2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# Changelog
Entries are grouped per release. Unreleased changes sit at the top.
## Unreleased
### Added
- `git-sync replicate` subcommand and `git-sync plan --mode replicate` for
source-authoritative, relay-only replication. Divergent branches and tags
are retargeted against the source; `--prune` deletes orphan managed refs.
Relay-only by design: no materialized fallback. Replicate works against
targets that advertise `no-thin` (including any server built on go-git's
receive-pack, e.g. `entire-server`) because the relayed pack is always
self-contained — our upload-pack client never requests the `thin-pack`
capability from the source.
- `gitsync.Client.Replicate` on the stable embedding surface.
- `gitsync.OperationMode`, `gitsync.ModeSync`, `gitsync.ModeReplicate`, and
`SyncPolicy.Mode` for selecting the mode from library callers.
- `--max-pack-bytes` and `--target-max-pack-bytes` flags on `sync`,
`replicate`, and `plan`. Previously only `bootstrap` exposed them, but
replicate's bootstrap-fallback path internally honors both — without
these flags, users couldn't split a huge initial replicate push into
tractable receive-pack POSTs for size-limited targets. The unstable
library `buildSyncConfig` now forwards `MaxPackBytes` and
`TargetMaxPackBytes` from `AdvancedOptions` to `syncer.Config`.
### Changed (stable API, breaking)
- The stable `SyncResult.Execution` summary has a new field layout:
- Added `execution.operation_mode` (`"sync"` or `"replicate"`) to describe
the high-level product mode the caller requested.
- Renamed `execution.mode` to `execution.transfer_mode` to describe the
low-level engine path that actually executed (`incremental-relay`,
`materialized`, `bootstrap-relay`, `replicate`, etc.). The previous
`mode` key is no longer emitted.
External embedders that parse `execution.mode` must switch to
`execution.transfer_mode`. There is no compatibility alias; the field was
renamed in place.
- `syncer.Result` (the CLI-level JSON shape) gains a top-level
`operation_mode` field alongside the existing relay fields. Callers
consuming the flat CLI JSON can use it to distinguish sync vs replicate
results.
### Fixed
- Work around a go-git v6 upload-pack bug where the server emits two
consecutive `NAK` pktlines in stateless-RPC mode when the client sends
haves that are not reachable from any want. The second NAK would
otherwise be misread by the sideband demuxer as a frame with channel
byte `'N'` ("unknown channel NAK"). `internal/gitproto.fetchPackV1` and
`fetchToStoreV1` now drain trailing NAKs before handing off to the
demuxer. See the corresponding go-git fix
(`plumbing/transport: don't emit a second NAK after encoding empty ACKs`).
ACHANGELOG.md+56
55 unmodified lines
56
57
58
59
59
60
61
62
63
64
2 unmodified lines
67
68
69
69
70
71
72
73
21 unmodified lines
95
96
97
98
99
100
101
31 unmodified lines
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
12 unmodified lines
172
173
174
152
175
176
177
178
156
179
180
181
182
10 unmodified lines
193
194
195
173
174
196
197
198
199
200
201
202
180
203
204
205
206
21 unmodified lines
228
229
230
208
231
232
233
234
55 unmodified lines
- `git-sync probe`: inspect a source remote, and optionally a target remote
- `git-sync fetch`: exercise source-side fetch negotiation without pushing
- `git-sync bootstrap`: seed an empty target with create-only relay behavior
- `git-sync plan`: compute source-to-target ref actions without pushing
- `git-sync plan`: compute source-to-target ref actions without pushing, with `--mode sync|replicate`
- `git-sync sync`: execute the planned changes against the target
- `git-sync replicate`: execute source-authoritative relay-only replication against the target
- `git-sync-bench`: run repeatable benchmark scenarios against fresh empty targets
## Library API
2 unmodified lines
- `pkg/gitsync`
- stable embedding surface for queue workers and other external callers
- typed `Probe`, `Plan`, and `Sync` requests/results
- typed `Probe`, `Plan`, `Sync`, and `Replicate` requests/results
- injected auth and HTTP client support
- `pkg/gitsync/unstable`
- explicitly non-stable surface for first-party tooling and advanced controls
21 unmodified lines
- Optional exact ref mapping with `--map`
- Fast-forward safety by default
- Optional forced retargeting with `--force`
- Optional source-authoritative relay-only replication with `replicate` / `plan --mode replicate`
- Optional managed-ref deletion with `--prune`
- Optional transfer stats output with `--stats`
- Optional machine-readable output with `--json`
31 unmodified lines
https://github.com/target-org/target-repo.git
```
Plan a source-authoritative replication without pushing anything:
```bash
go run ./cmd/git-sync plan \
--mode replicate \
--stats \
https://github.com/source-org/source-repo.git \
https://github.com/target-org/target-repo.git
```
Execute relay-only replication that overwrites differing managed refs and fails instead of materializing locally:
```bash
go run ./cmd/git-sync replicate \
--stats \
https://github.com/source-org/source-repo.git \
https://github.com/target-org/target-repo.git
```
If `replicate` cannot use relay against the target, it fails and tells you to rerun with `sync`.
Bootstrap an empty target without using the normal local object-store sync path:
```bash
12 unmodified lines
<target-url>
```
Add `--batch-max-pack-bytes` to split large branch bootstraps into multiple relay batches with temporary refs:
Add `--target-max-pack-bytes` to split large branch bootstraps into multiple relay batches with temporary refs:
```bash
go run ./cmd/git-sync bootstrap \
--batch-max-pack-bytes 1073741824 \
--target-max-pack-bytes 1073741824 \
<source-url> \
<target-url>
```
10 unmodified lines
A practical starting point is:
- `--batch-max-pack-bytes 536870912` for a conservative `512 MiB` target-side batch size
- `--batch-max-pack-bytes 1073741824` when you want fewer, larger batches and the target has more headroom
- `--target-max-pack-bytes 536870912` for a conservative `512 MiB` target-side batch size
- `--target-max-pack-bytes 1073741824` when you want fewer, larger batches and the target has more headroom
For example:
```bash
go run ./cmd/git-sync bootstrap \
--batch-max-pack-bytes 536870912 \
--target-max-pack-bytes 536870912 \
--protocol v2 \
-v \
<source-url> \
21 unmodified lines
--scenario bootstrap \
--source-url /tmp/git-sync-bench/kubernetes.git \
--repeat 3 \
--batch-max-pack-bytes 104857600 \
--target-max-pack-bytes 104857600 \
--stats \
--json
```
MREADME.md+31/-8
110 unmodified lines
111
112
113
114
114
115
116
117
324 unmodified lines
442
443
444
445
445
446
447
448
110 unmodified lines
fs.BoolVar(&cfg.Options.CollectStats, "stats", false, "collect transfer statistics")
fs.BoolVar(&cfg.Options.MeasureMemory, "measure-memory", true, "sample elapsed time and Go heap usage")
fs.Int64Var(&cfg.Options.MaxPackBytes, "max-pack-bytes", 0, "abort bootstrap if the streamed source pack exceeds this many bytes")
fs.Int64Var(&cfg.Options.BatchMaxPackBytes, "batch-max-pack-bytes", 0, "split branch bootstrap into relay batches capped at this many bytes per batch")
fs.Int64Var(&cfg.Options.TargetMaxPackBytes, "target-max-pack-bytes", 0, "target receive-pack body size limit; batches are planned and auto-subdivided to fit")
benchProtocol := benchProtocolModeFlag(benchProtocolMode(validation.ProtocolAuto))
fs.Var(&benchProtocol, "protocol", "protocol mode: auto, v1, or v2")
fs.BoolVar(&cfg.Options.Verbose, "v", false, "verbose logging")
324 unmodified lines
}
func usageError(message string) error {
usage := "usage:\n git-sync-bench --source-url <repo> [flags]\n\nflags:\n --scenario bootstrap|sync\n --repeat 3\n --work-dir /tmp/git-sync-bench\n --keep-targets\n --json\n --branch main,release\n --map main:stable\n --tags\n --force\n --prune\n --stats\n --measure-memory\n --max-pack-bytes 104857600\n --batch-max-pack-bytes 104857600\n --protocol auto|v1|v2\n -v\n"
usage := "usage:\n git-sync-bench --source-url <repo> [flags]\n\nflags:\n --scenario bootstrap|sync\n --repeat 3\n --work-dir /tmp/git-sync-bench\n --keep-targets\n --json\n --branch main,release\n --map main:stable\n --tags\n --force\n --prune\n --stats\n --measure-memory\n --max-pack-bytes 104857600\n --target-max-pack-bytes 104857600\n --protocol auto|v1|v2\n -v\n"
if message == "" {
return errors.New(strings.TrimSpace(usage))
}
Mcmd/git-sync-bench/main.go+2/-2
28 unmodified lines
29
30
31
32
32
33
34
35
34
36
37
38
39
7 unmodified lines
47
48
49
48
50
51
52
53
18 unmodified lines
72
73
74
75
76
77
78
79
80
81
1 unmodified line
83
84
85
86
87
88
89
90
1 unmodified line
92
93
94
95
96
97
98
35 unmodified lines
134
135
136
128
137
138
139
140
141
142
143
144
36 unmodified lines
181
182
183
171
184
185
186
187
221 unmodified lines
409
410
411
399
412
413
414
415
1 unmodified line
417
418
419
420
421
422
423
424
425
426
7 unmodified lines
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
28 unmodified lines
switch args[0] {
case "sync":
return runSyncLike(ctx, "sync", args[1:], false)
return runSyncLike(ctx, "sync", args[1:], false, gitsync.ModeSync)
case "replicate":
return runSyncLike(ctx, "replicate", args[1:], false, gitsync.ModeReplicate)
case "plan":
return runSyncLike(ctx, "plan", args[1:], true)
return runSyncLike(ctx, "plan", args[1:], true, "")
case "bootstrap":
return runBootstrap(ctx, args[1:])
case "probe":
7 unmodified lines
}
}
func runSyncLike(ctx context.Context, name string, args []string, dryRun bool) error {
func runSyncLike(ctx context.Context, name string, args []string, dryRun bool, defaultMode gitsync.OperationMode) error {
fs := flag.NewFlagSet(name, flag.ContinueOnError)
fs.SetOutput(os.Stderr)
18 unmodified lines
branches := fs.String("branch", "", "comma-separated branch list; default is all source branches")
fs.Var(&mappings, "map", "ref mapping in src:dst form; short names map branches, full refs map exact refs")
modeValue := operationModeFlag(defaultOperationMode(defaultMode))
if name == "plan" {
fs.Var(&modeValue, "mode", "operation mode: sync or replicate")
}
fs.BoolVar(&req.Policy.IncludeTags, "tags", false, "mirror tags")
fs.BoolVar(&req.Policy.Force, "force", false, "allow non-fast-forward branch updates and retarget tags")
fs.BoolVar(&req.Policy.Prune, "prune", false, "delete managed target refs that no longer exist on source")
1 unmodified line
fs.BoolVar(&req.Options.MeasureMemory, "measure-memory", false, "sample elapsed time and Go heap usage")
fs.BoolVar(&jsonOutput, "json", false, "print JSON output")
fs.IntVar(&req.Options.MaterializedMaxObjects, "materialized-max-objects", unstable.DefaultMaterializedMaxObjects, "abort non-relay materialized syncs above this many objects")
fs.Int64Var(&req.Options.MaxPackBytes, "max-pack-bytes", 0, "abort bootstrap-relay push if the streamed source pack exceeds this many bytes")
fs.Int64Var(&req.Options.TargetMaxPackBytes, "target-max-pack-bytes", 0, "target receive-pack body size limit; batches are planned and auto-subdivided to fit")
protocolValue := protocolModeFlag(protocolMode(envOr("GITSYNC_PROTOCOL", validation.ProtocolAuto)))
fs.Var(&protocolValue, "protocol", "protocol mode: auto, v1, or v2")
fs.BoolVar(&req.Options.Verbose, "v", false, "verbose logging")
1 unmodified line
if err := fs.Parse(args); err != nil {
return err
}
req.Policy.Mode = gitsync.OperationMode(modeValue)
req.Policy.Protocol = gitsync.ProtocolMode(protocolValue)
positional := fs.Args()
35 unmodified lines
if dryRun {
result, err = client.Plan(ctx, req)
} else {
result, err = client.Sync(ctx, req)
if req.Policy.Mode == gitsync.ModeReplicate {
result, err = client.Replicate(ctx, req)
} else {
result, err = client.Sync(ctx, req)
}
}
if err != nil {
return err
36 unmodified lines
fs.BoolVar(&req.Options.MeasureMemory, "measure-memory", false, "sample elapsed time and Go heap usage")
fs.BoolVar(&jsonOutput, "json", false, "print JSON output")
fs.Int64Var(&req.Options.MaxPackBytes, "max-pack-bytes", 0, "abort bootstrap if the streamed source pack exceeds this many bytes")
fs.Int64Var(&req.Options.BatchMaxPackBytes, "batch-max-pack-bytes", 0, "split branch bootstrap into relay batches capped at this many bytes per batch")
fs.Int64Var(&req.Options.TargetMaxPackBytes, "target-max-pack-bytes", 0, "target receive-pack body size limit; batches are planned and auto-subdivided to fit")
bootstrapProtocol := protocolModeFlag(protocolMode(envOr("GITSYNC_PROTOCOL", validation.ProtocolAuto)))
fs.Var(&bootstrapProtocol, "protocol", "protocol mode: auto, v1, or v2")
fs.BoolVar(&req.Options.Verbose, "v", false, "verbose logging")
221 unmodified lines
}
func usageError(message string) error {
usage := fmt.Sprintf("usage:\n git-sync sync [flags] <source-url> <target-url>\n git-sync plan [flags] <source-url> <target-url>\n git-sync bootstrap [flags] <source-url> <target-url>\n git-sync probe [flags] <source-url> [target-url]\n git-sync fetch [flags] <source-url>\n\nsync flags:\n --branch main,dev\n --map main:stable\n --tags\n --force\n --prune\n --stats\n --measure-memory\n --json\n --materialized-max-objects %d\n --protocol auto|v1|v2\n --source-token ...\n --target-token ...\n --source-username git\n --target-username git\n --source-bearer-token ...\n --target-bearer-token ...\n --source-insecure-skip-tls-verify\n --target-insecure-skip-tls-verify\n -v\n\nplan flags:\n --branch main,dev\n --map main:stable\n --tags\n --force\n --prune\n --stats\n --measure-memory\n --json\n --protocol auto|v1|v2\n --source-token ...\n --target-token ...\n --source-username git\n --target-username git\n --source-bearer-token ...\n --target-bearer-token ...\n --source-insecure-skip-tls-verify\n --target-insecure-skip-tls-verify\n -v\n\nbootstrap flags:\n --branch main,dev\n --map main:stable\n --tags\n --max-pack-bytes 104857600\n --batch-max-pack-bytes 1073741824\n --stats\n --measure-memory\n --json\n --protocol auto|v1|v2\n --source-token ...\n --target-token ...\n --source-username git\n --target-username git\n --source-bearer-token ...\n --target-bearer-token ...\n --source-insecure-skip-tls-verify\n --target-insecure-skip-tls-verify\n -v\n\nprobe flags:\n --tags\n --stats\n --measure-memory\n --json\n --protocol auto|v1|v2\n --source-token ...\n --source-username git\n --source-bearer-token ...\n --target-token ...\n --target-username git\n --target-bearer-token ...\n --source-insecure-skip-tls-verify\n --target-insecure-skip-tls-verify\n\nfetch flags:\n --branch main,dev\n --tags\n --stats\n --measure-memory\n --json\n --protocol auto|v1|v2\n --have-ref main\n --have <hash>\n --source-token ...\n --source-username git\n --source-bearer-token ...\n --source-insecure-skip-tls-verify\n", unstable.DefaultMaterializedMaxObjects)
usage := fmt.Sprintf("usage:\n git-sync sync [flags] <source-url> <target-url>\n git-sync replicate [flags] <source-url> <target-url>\n git-sync plan [flags] <source-url> <target-url>\n git-sync bootstrap [flags] <source-url> <target-url>\n git-sync probe [flags] <source-url> [target-url]\n git-sync fetch [flags] <source-url>\n\nsync flags:\n --branch main,dev\n --map main:stable\n --tags\n --force\n --prune\n --stats\n --measure-memory\n --json\n --materialized-max-objects %d\n --max-pack-bytes <bytes>\n --target-max-pack-bytes <bytes>\n --protocol auto|v1|v2\n --source-token ...\n --target-token ...\n --source-username git\n --target-username git\n --source-bearer-token ...\n --target-bearer-token ...\n --source-insecure-skip-tls-verify\n --target-insecure-skip-tls-verify\n -v\n\nreplicate flags:\n --branch main,dev\n --map main:stable\n --tags\n --prune\n --stats\n --measure-memory\n --json\n --max-pack-bytes <bytes>\n --target-max-pack-bytes <bytes>\n --protocol auto|v1|v2\n --source-token ...\n --target-token ...\n --source-username git\n --target-username git\n --source-bearer-token ...\n --target-bearer-token ...\n --source-insecure-skip-tls-verify\n --target-insecure-skip-tls-verify\n -v\n\nplan flags:\n --mode sync|replicate\n --branch main,dev\n --map main:stable\n --tags\n --force\n --prune\n --stats\n --measure-memory\n --json\n --max-pack-bytes <bytes>\n --target-max-pack-bytes <bytes>\n --protocol auto|v1|v2\n --source-token ...\n --target-token ...\n --source-username git\n --target-username git\n --source-bearer-token ...\n --target-bearer-token ...\n --source-insecure-skip-tls-verify\n --target-insecure-skip-tls-verify\n -v\n\nbootstrap flags:\n --branch main,dev\n --map main:stable\n --tags\n --max-pack-bytes 104857600\n --target-max-pack-bytes 1073741824\n --stats\n --measure-memory\n --json\n --protocol auto|v1|v2\n --source-token ...\n --target-token ...\n --source-username git\n --target-username git\n --source-bearer-token ...\n --target-bearer-token ...\n --source-insecure-skip-tls-verify\n --target-insecure-skip-tls-verify\n -v\n\nprobe flags:\n --tags\n --stats\n --measure-memory\n --json\n --protocol auto|v1|v2\n --source-token ...\n --source-username git\n --source-bearer-token ...\n --target-token ...\n --target-username git\n --target-bearer-token ...\n --source-insecure-skip-tls-verify\n --target-insecure-skip-tls-verify\n\nfetch flags:\n --branch main,dev\n --tags\n --stats\n --measure-memory\n --json\n --protocol auto|v1|v2\n --have-ref main\n --have <hash>\n --source-token ...\n --source-username git\n --source-bearer-token ...\n --source-insecure-skip-tls-verify\n", unstable.DefaultMaterializedMaxObjects)
if message == "" {
return errors.New(strings.TrimSpace(usage))
}
1 unmodified line
}
type protocolMode gitsync.ProtocolMode
type operationMode gitsync.OperationMode
type protocolModeFlag protocolMode
type operationModeFlag operationMode
func (p *protocolModeFlag) String() string {
return string(*p)
7 unmodified lines
*p = protocolModeFlag(protocolMode(gitsync.ProtocolMode(mode)))
return nil
}
func (m *operationModeFlag) String() string {
return string(*m)
}
func (m *operationModeFlag) Set(value string) error {
switch gitsync.OperationMode(value) {
case gitsync.ModeSync, gitsync.ModeReplicate:
*m = operationModeFlag(operationMode(value))
return nil
default:
return fmt.Errorf("unsupported mode %q", value)
}
}
// defaultOperationMode returns the starting value for the --mode flag.
// Subcommands that pin a mode (sync, replicate) pass it in; plan passes ""
// and gets sync as the default, letting --mode override it.
func defaultOperationMode(defaultMode gitsync.OperationMode) operationMode {
if defaultMode != "" {
return operationMode(defaultMode)
}
return operationMode(gitsync.ModeSync)
}
Mcmd/git-sync/main.go+45/-6
18 unmodified lines
19
20
21
22
23
24
25
26
89 unmodified lines
116
117
118
119
120
121
122
123
124
31 unmodified lines
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
79 unmodified lines
354
355
356
357
358
359
360
361
362
363
245
246
247
364
365
366
367
368
369
370
41 unmodified lines
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
1 unmodified line
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
18 unmodified lines
git "github.com/go-git/go-git/v6"
"github.com/go-git/go-git/v6/plumbing"
"github.com/go-git/go-git/v6/plumbing/object"
"github.com/go-git/go-git/v6/plumbing/protocol/packp"
"github.com/go-git/go-git/v6/plumbing/protocol/packp/capability"
"github.com/go-git/go-git/v6/plumbing/transport"
transporthttp "github.com/go-git/go-git/v6/plumbing/transport/http"
"github.com/go-git/go-git/v6/storage/memory"
89 unmodified lines
if result["dry_run"] != true {
t.Fatalf("expected dry_run=true, got %#v", result["dry_run"])
}
if result["operation_mode"] != "sync" {
t.Fatalf("expected operation_mode=sync, got %#v", result["operation_mode"])
}
if result["bootstrap_suggested"] != true {
t.Fatalf("expected bootstrap_suggested=true, got %#v", result["bootstrap_suggested"])
}
31 unmodified lines
}
}
func TestRun_Plan_ReplicateMode_JSONShowsReplicate(t *testing.T) {
sourceRepo, sourceFS := newSourceRepo(t)
makeCommits(t, sourceRepo, sourceFS, 1)
targetRepo, err := git.Init(memory.NewStorage())
if err != nil {
t.Fatalf("init target repo: %v", err)
}
sourceServer := newSmartHTTPRepoServer(t, sourceRepo)
targetServer := newSmartHTTPRepoServer(t, targetRepo)
defer sourceServer.Close()
defer targetServer.Close()
output, err := captureStdout(func() error {
return run(context.Background(), []string{
"plan",
"--mode", "replicate",
"--json",
sourceServer.RepoURL(),
targetServer.RepoURL(),
})
})
if err != nil {
t.Fatalf("run replicate plan: %v", err)
}
var result map[string]any
if err := json.Unmarshal([]byte(output), &result); err != nil {
t.Fatalf("decode plan json: %v\noutput=%s", err, output)
}
if result["operation_mode"] != "replicate" {
t.Fatalf("expected operation_mode=replicate, got %#v", result["operation_mode"])
}
}
func TestRun_Replicate_SubcommandExecutesAgainstEmptyTarget(t *testing.T) {
sourceRepo, sourceFS := newSourceRepo(t)
makeCommits(t, sourceRepo, sourceFS, 1)
targetRepo, err := git.Init(memory.NewStorage())
if err != nil {
t.Fatalf("init target repo: %v", err)
}
output, err := captureStdout(func() error {
return run(context.Background(), []string{
"replicate",
"--json",
sourceServer.RepoURL(),
targetServer.RepoURL(),
})
})
if err != nil {
t.Fatalf("run replicate subcommand: %v", err)
}
var result map[string]any
if err := json.Unmarshal([]byte(output), &result); err != nil {
t.Fatalf("decode replicate json: %v\noutput=%s", err, output)
}
if result["dry_run"] != false {
t.Fatalf("expected dry_run=false, got %#v", result["dry_run"])
}
if result["operation_mode"] != "replicate" {
t.Fatalf("expected operation_mode=replicate, got %#v", result["operation_mode"])
}
if result["pushed"] != float64(1) {
t.Fatalf("expected pushed=1, got %#v", result["pushed"])
}
if result["relay"] != true {
t.Fatalf("expected relay=true, got %#v", result["relay"])
}
if result["relay_reason"] != "empty-target-managed-refs" {
t.Fatalf("expected relay_reason=empty-target-managed-refs, got %#v", result["relay_reason"])
}
if got := targetServer.Count("git-receive-pack"); got != 1 {
t.Fatalf("expected one receive-pack POST, got %d", got)
}
sourceHead, err := sourceRepo.Reference(plumbing.NewBranchReferenceName(testBranch), true)
if err != nil {
t.Fatalf("source head: %v", err)
}
targetHead, err := targetRepo.Reference(plumbing.NewBranchReferenceName(testBranch), true)
if err != nil {
t.Fatalf("target head: %v", err)
}
if sourceHead.Hash() != targetHead.Hash() {
t.Fatalf("expected target head %s to match source head %s", targetHead.Hash(), sourceHead.Hash())
}
}
func TestRun_Replicate_SubcommandRejectsForce(t *testing.T) {
err := run(context.Background(), []string{
"replicate",
"--force",
"http://127.0.0.1:1/source.git",
"http://127.0.0.1:1/target.git",
})
if err == nil {
t.Fatal("expected replicate --force to be rejected")
}
if !strings.Contains(err.Error(), "replicate does not support --force") {
t.Fatalf("unexpected error: %v", err)
}
}
func captureStdout(fn func() error) (string, error) {
old := os.Stdout
r, w, err := os.Pipe()
79 unmodified lines
mu sync.Mutex
receivePacks int
thinCapable bool
}
func newSmartHTTPRepoServer(t *testing.T, repo *git.Repository) *smartHTTPRepoServer {
t.Helper()
s := &smartHTTPRepoServer{
t: t,
repo: repo,
repoPath: "/repo.git",
t: t,
repo: repo,
repoPath: "/repo.git",
thinCapable: true,
}
s.server = httptest.NewServer(http.HandlerFunc(s.handle))
return s
41 unmodified lines
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if service == transport.ReceivePackService && s.thinCapable {
rewritten, err := rewriteReceivePackAdvertisement(buf.Bytes(), func(caps *capability.List) {
caps.Delete(capability.Capability("no-thin"))
})
if err != nil {
s.t.Fatalf("rewrite receive-pack advertisement: %v", err)
}
buf.Reset()
_, _ = buf.Write(rewritten)
}
w.Header().Set("Content-Type", fmt.Sprintf("application/x-%s-advertisement", service))
if _, err := w.Write(buf.Bytes()); err != nil {
1 unmodified line
}
}
func rewriteReceivePackAdvertisement(data []byte, mutate func(*capability.List)) ([]byte, error) {
ar := packp.NewAdvRefs()
if err := ar.Decode(bytes.NewReader(data)); err == nil {
mutate(ar.Capabilities)
var buf bytes.Buffer
if err := ar.Encode(&buf); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
rd := bytes.NewReader(data)
var smart packp.SmartReply
if err := smart.Decode(rd); err != nil {
return nil, err
}
ar = packp.NewAdvRefs()
if err := ar.Decode(rd); err != nil {
return nil, err
}
mutate(ar.Capabilities)
var buf bytes.Buffer
if err := smart.Encode(&buf); err != nil {
return nil, err
}
if err := ar.Encode(&buf); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func (s *smartHTTPRepoServer) handleUploadPack(w http.ResponseWriter, r *http.Request) {
var buf bytes.Buffer
wc := nopWriteCloser{&buf}
Mcmd/git-sync/main_test.go+164/-3
22 unmodified lines
23
24
25
26
26
27
28
29
34 unmodified lines
64
65
66
67
67
68
69
70
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
6 unmodified lines
92
93
94
85
95
96
97
98
22 unmodified lines
121
122
123
114
124
125
126
127
4 unmodified lines
132
133
134
125
135
136
137
138
22 unmodified lines
- relay-first strategies
- front-loaded validation
- typed results and JSON output
- explicit execution modes instead of a single opaque "mirror" operation
- explicit operation modes instead of a single opaque "mirror" operation
## When To Use `git-sync`
34 unmodified lines
### Explicit Strategy Split
The current execution modes are:
The current product modes are:
- `bootstrap`
- empty-target relay
- `sync`
- planning plus reconciliation
- `replicate`
- source-authoritative overwrite planning
- relay-only execution
- no materialized fallback
- works against targets that advertise `no-thin` (the relayed pack is
always self-contained because our upload-pack client does not request
the `thin-pack` capability)
The current transfer modes are:
- `bootstrap`
- empty-target relay
- incremental relay
- narrow fast path for safe updates
- materialized fallback
6 unmodified lines
- `pkg/gitsync`
- stable public embedding API
- typed `Probe`, `Plan`, and `Sync` requests/results
- typed `Probe`, `Plan`, `Sync`, and `Replicate` requests/results
- auth and HTTP client injection for worker-style callers
- `pkg/gitsync/unstable`
- explicitly non-stable first-party tooling surface
22 unmodified lines
The project now separates embedding concerns from first-party tooling concerns:
- `pkg/gitsync` is the stable library boundary.
Callers express orchestration intent through typed probe, plan, and sync requests. Auth and transport are injected. Execution strategy remains internal.
Callers express orchestration intent through typed probe, plan, sync, and replicate requests. Auth and transport are injected. Execution strategy remains internal.
- `pkg/gitsync/unstable` is the escape hatch for advanced controls.
It exists so the CLI and benchmark tool can use batching limits, memory measurement, verbose progress, bootstrap, and fetch without widening the stable API prematurely.
4 unmodified lines
- `Counts`
aggregate applied/skipped/blocked/deleted counts
- `Execution`
protocol, relay summary, execution mode, and batch summary
protocol, operation mode, transfer summary, and batch summary
That split is intentional:
Mdocs/architecture.md+17/-7
23 unmodified lines
24
25
26
27
27
28
29
30
23 unmodified lines
Mdocs/benchmarking.md+1/-1
59 unmodified lines
60 61 62 63 63 64 65 66 67 68 69 70 70 71 72 73 74 74 75 76 77 21 unmodified lines
99 100 101 102 102 103 104 105 106 107 108 109 110 106 107 108 109 110 111 112 112 113 114 115 114 115 117 116 117 118 119 120 121 122 123 124 125 126 127 128 129 110 unmodified lines
240 241 242 234 235 243 244 245 246 247 248 249 250 9 unmodified lines
260 261 262 253 263 264 265 266 3 unmodified lines
270 271 272 263 273 274 275 276
59 unmodified lines
git-sync bootstrap \
--batch-max-pack-bytes 1073741824 \
--target-max-pack-bytes 1073741824 \
<source-url> \
<target-url>
Possible related flags:
--batch-max-pack-bytes--target-max-pack-bytes--batch-ref-prefix refs/gitsync/bootstrap/--keep-temp-refs-on-failure
The first version should only need --batch-max-pack-bytes.
The first version should only need --target-max-pack-bytes.
Temporary Ref Strategy
21 unmodified lines
Checkpoint Selection
The most practical first heuristic is first-parent checkpointing. Checkpoints are placed using a commit-count estimate, not measured pack sizes.
For a branch tip:
- Walk first-parent ancestry backward.
- Sample candidate checkpoint commits.
- Starting from the oldest candidate, estimate each batch by doing a source fetch against the previous checkpoint as
have. - Pick the largest checkpoint whose pack stays under
--batch-max-pack-bytes. - Repeat until the branch tip is reached.
- Fetch the commit graph (tree:0 filter, one round-trip — commits only, no blobs/trees).
- Walk first-parent ancestry backward to get the chain length.
- Estimate total pack size:
chainLen × 8 KiB/commit. - Compute number of batches:
ceil(estimated / --target-max-pack-bytes). - Place checkpoints evenly along the first-parent chain.
This is only a heuristic: This is a heuristic — real bytes-per-commit varies widely (2–100+ KiB depending on blob churn). The estimate intentionally errs toward more batches.
- actual pack size depends on delta choices and object reuse
- merges and deep side histories can make batch sizes uneven
Adaptive size correction
But it is much simpler than exact graph partitioning and good enough for a first implementation. If the estimate is too optimistic (fewer batches than needed), two safeguards catch it:
PACK header pre-check: after starting a fetch, peek at the first 12 bytes of the pack to read the object count. Multiply by ~750 bytes/object. If the estimate exceeds
--target-max-pack-bytes, abort the fetch (12 bytes wasted, not gigabytes), insert a midpoint checkpoint, and retry. This avoids a full transfer for obviously-oversized batches.Target rejection retry: if the target's receive-pack rejects a push for exceeding its body-size limit, detect the error, insert a midpoint checkpoint from the stored chain, and retry. This catches cases where the PACK header estimate was close but the real pack was slightly over.
Both safeguards converge in O(log n) splits — each failure halves the commit range.
Why not probe (the previous design)
The previous implementation did full FetchPack round-trips per probe candidate to measure actual pack sizes. For linux/master (75k commits) this required 13+ fetch-and-discard cycles, downloading gigabytes of throwaway data and taking minutes before any real push started. The estimate approach reduces planning to one commit-graph fetch (~20 seconds) plus arithmetic.
Batch Flow For One Branch
110 unmodified lines
Practical Risks
- pack-size estimation may require extra source fetches before actual execution
- checkpoint search may be slow on very deep histories
- the 8 KiB/commit estimate can be significantly off for blob-heavy repos (linux is ~66 KiB/commit); the PACK header pre-check and target-rejection retry handle this adaptively
- object density is not uniform along the commit chain — recent history often has more objects per commit than early history, so evenly-spaced checkpoints produce uneven pack sizes
- target-side unpack/index cost may still be high even after batching, just smaller
- temp refs add cleanup and restart complexity
- the source builds the full pack even if we abort after the PACK header; this wastes source CPU but not network
This is still likely worthwhile for very large initial migrations because it changes a single huge risky operation into several bounded ones.
9 unmodified lines
Progress:
implemented via
git-sync bootstrap --batch-max-pack-bytesimplemented via
git-sync bootstrap --target-max-pack-bytescurrently requires source-side protocol v2 with fetch filter support
resumes from an existing temp ref when that temp ref matches a planned checkpoint
exercised by
TestBootstrap_GitHTTPBackendBatchedBranch3 unmodified linesprefer plain
bootstrapfirstuse batching when a single large bootstrap push is too risky, too large, or fails on the target side
start with
--batch-max-pack-bytes 536870912and adjust upward only if the target has enough headroomstart with
--target-max-pack-bytes 536870912and adjust upward only if the target has enough headroom
Phase B:
Mdocs/bootstrap-batching.md+27/-17
```
151 unmodified lines
152
153
154
155
155
156
157
158
151 unmodified lines
- explicit mapped refs are supported
- `--max-pack-bytes` provides a first safety threshold for the streamed source pack during bootstrap
- `--batch-max-pack-bytes` now enables a Phase A batched branch-only bootstrap mode for large initial syncs
- `--target-max-pack-bytes` now enables a Phase A batched branch-only bootstrap mode for large initial syncs
Phase 3:
```
Mdocs/bootstrap.md+1/-1
```
13 unmodified lines
14
15
16
17
18
19
20
98 unmodified lines
119
120
121
121
122
123
124
125
126
21 unmodified lines
148
149
150
149
151
152
153
154
155
13 unmodified lines
- `Probe`
- `Plan`
- `Sync`
- `Replicate`
- typed requests and results
- injected auth and HTTP client support
98 unmodified lines
- `Counts`
aggregate applied/skipped/blocked/deleted totals
- `Execution`
protocol, relay summary, execution mode, and batch summary
protocol, `operation_mode` (sync or replicate), relay summary,
`transfer_mode` (the engine path that executed), and batch summary
- `Stats`
transfer counters when requested
- `Measurement`
21 unmodified lines
- retry on returned `error`
- do not blindly retry on `Counts.Blocked > 0`
- log `Execution.Mode` and `Execution.Reason` for operator visibility
- log `Execution.OperationMode`, `Execution.TransferMode`, and
`Execution.Reason` for operator visibility
## What Not To Depend On
```
Mdocs/embedding.md+5/-2
29 unmodified lines
30 31 32 33 33 34 35 36
29 unmodified lines
env GOCACHE=/tmp/go-build GITSYNC_E2E_GIT_HTTP_BACKEND=1 go test ./internal/syncer -run TestBootstrap_GitHTTPBackendBatchedPlanningTracksBatchLimit -v
That test uses a real `git-http-backend` source/target pair and checks that a smaller `--batch-max-pack-bytes` planning limit produces at least as many planned checkpoints as a larger one, while still planning to the branch tip.
That test uses a real `git-http-backend` source/target pair and checks that a smaller `--target-max-pack-bytes` planning limit produces at least as many planned checkpoints as a larger one, while still planning to the branch tip.
## Live Linux Smokes
Mdocs/testing.md+1/-1
1
2
3
4
5
6
7
53 unmodified lines
61
62
63
63
64
65
65
66
67
68
69
9 unmodified lines
79
80
81
81
82
83
83
84
85
86
87
30 unmodified lines
118
119
120
120
121
122
123
124
125
17 unmodified lines
143
144
145
146
147
148
149
2 unmodified lines
152
153
154
152
155
156
157
158
159
160
161
162
163
164
165
11 unmodified lines
177
178
179
170
180
181
182
183
2 unmodified lines
186
187
188
189
190
191
192
2 unmodified lines
195
196
197
187
198
199
200
201
202
203
204
205
206
207
208
14 unmodified lines
223
224
225
208
226
227
228
229
1 unmodified line
231
232
233
216
234
235
236
237
13 unmodified lines
251
252
253
254
255
256
257
6 unmodified lines
264
265
266
248
267
268
269
270
12 unmodified lines
283
284
285
286
287
288
268
289
290
291
292
15 unmodified lines
308
309
310
311
312
313
314
3 unmodified lines
318
319
320
299
321
322
323
324
6 unmodified lines
331
332
333
334
335
336
337
338
339
340
341
13 unmodified lines
355
356
357
358
359
332
360
361
362
363
4 unmodified lines
368
369
370
371
372
344
373
374
375
347
376
377
378
379
380
381
382
3 unmodified lines
386
387
388
389
390
358
391
392
393
394
2 unmodified lines
397
398
399
400
401
368
402
403
404
405
406
407
408
409
410
373
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
package gitproto
import (
"bufio"
"bytes"
"context"
"errors"
53 unmodified lines
) error {
switch s.Protocol {
case "v2":
return fetchToStoreV2(ctx, store, conn, s.V2Caps, desired, targetRefs)
return fetchToStoreV2(ctx, store, conn, s.V2Caps, desired, targetRefs, s.Verbose)
case "v1":
return fetchToStoreV1(ctx, store, conn, s.V1Adv, desired, targetRefs)
return fetchToStoreV1(ctx, store, conn, s.V1Adv, desired, targetRefs, s.Verbose)
default:
return fmt.Errorf("unsupported source protocol %q", s.Protocol)
}
9 unmodified lines
) (io.ReadCloser, error) {
switch s.Protocol {
case "v2":
return fetchPackV2(ctx, conn, s.V2Caps, desired, targetRefs)
return fetchPackV2(ctx, conn, s.V2Caps, desired, targetRefs, s.Verbose)
case "v1":
return fetchPackV1(ctx, conn, s.V1Adv, desired, targetRefs)
return fetchPackV1(ctx, conn, s.V1Adv, desired, targetRefs, s.Verbose)
default:
return nil, fmt.Errorf("unsupported source protocol %q", s.Protocol)
}
30 unmodified lines
return err
}
defer ioutil.CheckClose(reader, &err)
return storeV2FetchPack(store, reader)
// Commit-graph fetches are short and not user-facing; skip progress.
return storeV2FetchPack(store, reader, false)
}
// Capabilities returns the sorted capability list for display.
17 unmodified lines
caps *V2Capabilities,
desired map[plumbing.ReferenceName]DesiredRef,
targetRefs map[plumbing.ReferenceName]plumbing.Hash,
verbose bool,
) error {
wants := collectWants(desired)
haves := SortedUniqueHashes(refValues(targetRefs))
2 unmodified lines
}
cmdArgs := make([]string, 0, len(wants)+len(haves)+4)
cmdArgs = append(cmdArgs, "ofs-delta", "no-progress")
// NOTE: no "thin-pack" argument. The relayed pack must stay
// self-contained so callers (e.g. replicate) can forward it to
// receive-pack servers that may advertise "no-thin". See
// planner.SupportsReplicateRelay for the matching invariant.
cmdArgs = append(cmdArgs, "ofs-delta")
if !verbose {
cmdArgs = append(cmdArgs, "no-progress")
}
for _, h := range wants {
cmdArgs = append(cmdArgs, "want "+h.String())
}
11 unmodified lines
return err
}
defer ioutil.CheckClose(reader, &err)
return storeV2FetchPack(store, reader)
return storeV2FetchPack(store, reader, verbose)
}
func fetchPackV2(
2 unmodified lines
caps *V2Capabilities,
desired map[plumbing.ReferenceName]DesiredRef,
targetRefs map[plumbing.ReferenceName]plumbing.Hash,
verbose bool,
) (io.ReadCloser, error) {
wants := collectWants(desired)
haves := SortedUniqueHashes(refValues(targetRefs))
2 unmodified lines
}
cmdArgs := make([]string, 0, len(wants)+len(haves)+4)
cmdArgs = append(cmdArgs, "ofs-delta", "no-progress")
// NOTE: no "thin-pack" argument. The relayed pack must stay
// self-contained so callers (e.g. replicate) can forward it to
// receive-pack servers that may advertise "no-thin". See
// planner.SupportsReplicateRelay for the matching invariant.
cmdArgs = append(cmdArgs, "ofs-delta")
if !verbose {
cmdArgs = append(cmdArgs, "no-progress")
}
// Only request include-tag if the server supports it (issue #6).
if hasTag(desired) && caps.FetchSupports("include-tag") {
cmdArgs = append(cmdArgs, "include-tag")
14 unmodified lines
if err != nil {
return nil, err
}
packStream, err := openV2PackStream(reader)
packStream, err := openV2PackStream(reader, verbose)
if err != nil {
_ = reader.Close()
return nil, err
1 unmodified line
return packStream, nil
}
func storeV2FetchPack(store storer.Storer, r io.Reader) error {
func storeV2FetchPack(store storer.Storer, r io.Reader, verbose bool) error {
reader := NewPacketReader(r)
for {
kind, payload, err := reader.ReadPacket()
13 unmodified lines
switch line {
case "packfile\n":
demux := sideband.NewDemuxer(sideband.Sideband64k, reader.BufReader())
demux.Progress = progressSink(verbose, "source: ")
return packfile.UpdateObjectStorage(store, demux)
case "acknowledgments\n", "shallow-info\n":
if err := SkipSection(reader); err != nil {
6 unmodified lines
}
}
func openV2PackStream(body io.ReadCloser) (io.ReadCloser, error) {
func openV2PackStream(body io.ReadCloser, verbose bool) (io.ReadCloser, error) {
reader := NewPacketReader(body)
for {
kind, payload, err := reader.ReadPacket()
12 unmodified lines
line := string(payload)
switch line {
case "packfile\n":
demux := sideband.NewDemuxer(sideband.Sideband64k, reader.BufReader())
demux.Progress = progressSink(verbose, "source: ")
return &wrappedRC{
Reader: sideband.NewDemuxer(sideband.Sideband64k, reader.BufReader()),
Reader: demux,
Closer: body,
}, nil
case "acknowledgments\n", "shallow-info\n":
15 unmodified lines
desired map[plumbing.ReferenceName]DesiredRef,
targetRefs map[plumbing.ReferenceName]plumbing.Hash,
includeTags bool,
verbose bool,
) ([]byte, *capability.List, error) {
wants := collectWants(desired)
haves := SortedUniqueHashes(refValues(targetRefs))
3 unmodified lines
req := packp.NewUploadRequest()
req.Wants = wants
if adv.Capabilities.Supports(capability.NoProgress) {
if !verbose && adv.Capabilities.Supports(capability.NoProgress) {
_ = req.Capabilities.Set(capability.NoProgress)
}
if includeTags && adv.Capabilities.Supports(capability.IncludeTag) {
6 unmodified lines
if adv.Capabilities.Supports(capability.OFSDelta) {
_ = req.Capabilities.Set(capability.OFSDelta)
}
// NOTE: we intentionally do not request capability.ThinPack. The relayed
// pack must stay self-contained because callers (e.g. replicate) forward
// it to receive-pack servers that may advertise "no-thin".
// planner.SupportsReplicateRelay depends on this invariant — if you add
// thin-pack support here, update that check to gate on target NoThin.
var buf bytes.Buffer
if err := req.Encode(&buf); err != nil {
13 unmodified lines
adv *packp.AdvRefs,
desired map[plumbing.ReferenceName]DesiredRef,
targetRefs map[plumbing.ReferenceName]plumbing.Hash,
verbose bool,
) error {
body, caps, err := buildV1UploadPackBody(adv, desired, targetRefs, hasTag(desired))
body, caps, err := buildV1UploadPackBody(adv, desired, targetRefs, hasTag(desired), verbose)
if err != nil {
return err
}
4 unmodified lines
defer ioutil.CheckClose(reader, &err)
// Decode server response (ACK/NAK) then read pack with sideband demux.
buffered := bufio.NewReader(reader)
var srvResp packp.ServerResponse
if decErr := srvResp.Decode(reader); decErr != nil {
if decErr := srvResp.Decode(buffered); decErr != nil {
return fmt.Errorf("decode server response: %w", decErr)
}
sbReader := buildSidebandReader(caps, reader, nil)
if drainErr := drainTrailingNAKs(buffered); drainErr != nil {
return fmt.Errorf("drain server response: %w", drainErr)
}
sbReader := buildSidebandReader(caps, buffered, progressSink(verbose, "source: "))
return packfile.UpdateObjectStorage(store, sbReader)
}
3 unmodified lines
adv *packp.AdvRefs,
desired map[plumbing.ReferenceName]DesiredRef,
targetRefs map[plumbing.ReferenceName]plumbing.Hash,
verbose bool,
) (io.ReadCloser, error) {
body, caps, err := buildV1UploadPackBody(adv, desired, targetRefs, hasTag(desired))
body, caps, err := buildV1UploadPackBody(adv, desired, targetRefs, hasTag(desired), verbose)
if err != nil {
return nil, err
}
2 unmodified lines
return nil, fmt.Errorf("source upload-pack: %w", err)
}
buffered := bufio.NewReader(reader)
var srvResp packp.ServerResponse
if decErr := srvResp.Decode(reader); decErr != nil {
if decErr := srvResp.Decode(buffered); decErr != nil {
_ = reader.Close()
return nil, fmt.Errorf("decode server response: %w", decErr)
}
if drainErr := drainTrailingNAKs(buffered); drainErr != nil {
_ = reader.Close()
return nil, fmt.Errorf("drain server response: %w", drainErr)
}
return &wrappedRC{
Reader: buildSidebandReader(caps, reader, nil),
Reader: buildSidebandReader(caps, buffered, progressSink(verbose, "source: ")),
Closer: reader,
}, nil
}
// drainTrailingNAKs consumes any extra "NAK\n" pktlines left in the stream
// after ServerResponse.Decode. go-git's upload-pack server emits a second NAK
// when haves were sent but none were reachable from the wants (see
// plumbing/transport/upload_pack.go), while go-git's ServerResponse.Decode
// stops after the first NAK. The remainder would otherwise be misread by the
// sideband demuxer as a frame with channel byte 'N' ("unknown channel NAK").
//
// A stream that runs out before we can peek 8 bytes carries no trailing NAK
// to drain, so we silently stop. The downstream consumer observes the same
// underlying read error on its first read.
func drainTrailingNAKs(r *bufio.Reader) error {
for {
header, err := r.Peek(8)
if len(header) < 8 || !bytes.Equal(header, []byte("0008NAK\n")) {
_ = err
return nil
}
if _, err := r.Discard(8); err != nil {
return err
}
}
}
// buildSidebandReader wraps a reader with sideband demuxing if the negotiated
// capabilities include sideband support. Delegates to PreferredSideband (issue #4).
func buildSidebandReader(caps *capability.List, reader io.Reader, progress sideband.Progress) io.Reader {
Minternal/gitproto/fetch.go+80/-19
330 unmodified lines
331
332
333
334
334
335
336
337
45 unmodified lines
383
384
385
386
386
387
388
389
45 unmodified lines
435
436
437
438
438
439
440
441
43 unmodified lines
485
486
487
488
488
489
490
491
38 unmodified lines
530
531
532
533
533
534
535
536
39 unmodified lines
576
577
578
579
579
580
581
582
28 unmodified lines
611
612
613
614
614
615
616
617
36 unmodified lines
654
655
656
657
657
658
659
660
9 unmodified lines
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
21 unmodified lines
747
748
749
700
750
751
752
753
39 unmodified lines
793
794
795
746
796
797
798
799
39 unmodified lines
839
840
841
792
842
843
844
845
36 unmodified lines
882
883
884
835
885
886
887
888
4 unmodified lines
893
894
895
846
896
897
898
899
330 unmodified lines
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() {
_, err := fetchPackV1(ctx, conn, adv, desired, nil)
_, err := fetchPackV1(ctx, conn, adv, desired, nil, false)
done <- err
}()
45 unmodified lines
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() {
_, err := fetchPackV2(ctx, conn, caps, desired, nil)
_, err := fetchPackV2(ctx, conn, caps, desired, nil, false)
done <- err
}()
45 unmodified lines
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() {
done <- fetchToStoreV2(ctx, memory.NewStorage(), conn, caps, desired, nil)
done <- fetchToStoreV2(ctx, memory.NewStorage(), conn, caps, desired, nil, false)
}()
select {
43 unmodified lines
},
}
err = fetchToStoreV2(context.Background(), memory.NewStorage(), conn, caps, desired, nil)
err = fetchToStoreV2(context.Background(), memory.NewStorage(), conn, caps, desired, nil, false)
if err == nil {
t.Fatal("expected decode error")
}
38 unmodified lines
done := make(chan error, 1)
go func() {
done <- fetchToStoreV2(ctx, memory.NewStorage(), conn, caps, desired, nil)
done <- fetchToStoreV2(ctx, memory.NewStorage(), conn, caps, desired, nil, false)
}()
select {
39 unmodified lines
},
}
_, err = fetchPackV1(context.Background(), conn, adv, desired, nil)
_, err = fetchPackV1(context.Background(), conn, adv, desired, nil, false)
if err == nil {
t.Fatal("expected decode error")
}
28 unmodified lines
},
}
rc, err := fetchPackV1(context.Background(), conn, adv, desired, nil)
rc, err := fetchPackV1(context.Background(), conn, adv, desired, nil, false)
if err != nil {
t.Fatalf("fetchPackV1: %v", err)
}
36 unmodified lines
},
}
rc, err := fetchPackV1(context.Background(), conn, adv, desired, nil)
rc, err := fetchPackV1(context.Background(), conn, adv, desired, nil, false)
if err != nil {
t.Fatalf("fetchPackV1: %v", err)
}
9 unmodified lines
}
}
func TestFetchPackV1DrainsSecondNAK(t *testing.T) {
// go-git's upload-pack server emits two NAKs when haves were sent but none
// were reachable from the wants. ServerResponse.Decode only consumes the
// first; the second must be drained before sideband demux or the demuxer
// misreads "NAK\n" as a sideband frame with channel 'N'.
// This simulates that double-NAK followed by a sideband-wrapped PACK header
// (channel 0x01 + "PACK" magic).
payload := []byte("0008NAK\n0008NAK\n0009\x01PACK")
ep, err := transport.NewEndpoint("https://example.com/repo.git")
if err != nil {
t.Fatalf("parse endpoint: %v", err)
}
body := &trackingReadCloser{ReadCloser: io.NopCloser(bytes.NewReader(payload))}
conn := NewConn(ep, "source", nil, roundTripperFunc(func(req *http.Request) (*http.Response, error) {
return &http.Response{
StatusCode: http.StatusOK,
Request: req,
Body: body,
}, nil
}))
adv := packp.NewAdvRefs()
_ = adv.Capabilities.Set(capability.Sideband64k)
desired := map[plumbing.ReferenceName]DesiredRef{
plumbing.NewBranchReferenceName("main"): {
SourceRef: plumbing.NewBranchReferenceName("main"),
TargetRef: plumbing.NewBranchReferenceName("main"),
SourceHash: plumbing.NewHash("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"),
},
}
rc, err := fetchPackV1(context.Background(), conn, adv, desired, nil, false)
if err != nil {
t.Fatalf("fetchPackV1 should tolerate double-NAK preamble, got: %v", err)
}
got, err := io.ReadAll(rc)
// Reader will error (or hit EOF) after the truncated sideband frame; what
// we care about is that the demuxed payload started with PACK, proving the
// second NAK was drained rather than fed into the demuxer.
if !bytes.HasPrefix(got, []byte("PACK")) {
t.Fatalf("expected demuxed pack bytes to start with PACK, got %q (err=%v)", got, err)
}
if closeErr := rc.Close(); closeErr != nil {
t.Fatalf("close returned reader: %v", closeErr)
}
if !body.closed {
t.Fatal("expected returned reader close to close underlying body")
}
}
func TestFetchPackV2ClosesBodyOnDecodeError(t *testing.T) {
ep, err := transport.NewEndpoint("https://example.com/repo.git")
if err != nil {
21 unmodified lines
},
}
_, err = fetchPackV2(context.Background(), conn, caps, desired, nil)
_, err = fetchPackV2(context.Background(), conn, caps, desired, nil, false)
if err == nil {
t.Fatal("expected decode error")
}
39 unmodified lines
},
}
rc, err := fetchPackV2(context.Background(), conn, caps, desired, nil)
rc, err := fetchPackV2(context.Background(), conn, caps, desired, nil, false)
if err != nil {
t.Fatalf("fetchPackV2: %v", err)
}
39 unmodified lines
},
}
rc, err := fetchPackV2(context.Background(), conn, caps, desired, nil)
rc, err := fetchPackV2(context.Background(), conn, caps, desired, nil, false)
if err != nil {
t.Fatalf("fetchPackV2: %v", err)
}
36 unmodified lines
},
}
err = fetchToStoreV2(context.Background(), memory.NewStorage(), conn, caps, desired, nil)
err = fetchToStoreV2(context.Background(), memory.NewStorage(), conn, caps, desired, nil, false)
if err == nil {
t.Fatal("expected malformed mid-stream sideband error")
}
4 unmodified lines
func TestBuildV1UploadPackBodyEmptyWantSet(t *testing.T) {
adv := packp.NewAdvRefs()
_, _, err := buildV1UploadPackBody(adv, nil, nil, false)
_, _, err := buildV1UploadPackBody(adv, nil, nil, false, false)
if !errors.Is(err, git.NoErrAlreadyUpToDate) {
t.Fatalf("expected NoErrAlreadyUpToDate, got %v", err)
}
Minternal/gitproto/fetch_test.go+63/-13
95 unmodified lines
96
97
98
99
100
101
102
9 unmodified lines
112
113
114
114
115
116
117
116
117
118
119
118
119
120
121
122
123
124
125
126
127
128
129
23 unmodified lines
153
154
155
149
156
157
158
159
9 unmodified lines
169
170
171
165
172
173
174
175
24 unmodified lines
200
201
202
196
203
204
205
206
13 unmodified lines
220
221
222
216
223
224
225
226
2 unmodified lines
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
95 unmodified lines
conn *Conn,
req *packp.UpdateRequests,
packData io.Reader,
verbose bool,
) error {
var header bytes.Buffer
if err := req.Encode(&header); err != nil {
9 unmodified lines
}
defer reader.Close()
// Unwrap sideband if negotiated.
// Unwrap sideband if negotiated; stream server-side progress to stderr
// when verbose so long-running pushes show "Resolving deltas ..." etc.
var respReader io.Reader = reader
if req.Capabilities.Supports(capability.Sideband64k) {
respReader = sideband.NewDemuxer(sideband.Sideband64k, reader)
} else if req.Capabilities.Supports(capability.Sideband) {
respReader = sideband.NewDemuxer(sideband.Sideband, reader)
switch {
case req.Capabilities.Supports(capability.Sideband64k):
dem := sideband.NewDemuxer(sideband.Sideband64k, reader)
dem.Progress = progressSink(verbose, "target: ")
respReader = dem
case req.Capabilities.Supports(capability.Sideband):
dem := sideband.NewDemuxer(sideband.Sideband, reader)
dem.Progress = progressSink(verbose, "target: ")
respReader = dem
}
if req.Capabilities.Supports(capability.ReportStatus) {
23 unmodified lines
return err
}
if !hasUpdates {
return sendReceivePack(ctx, conn, req, nil)
return sendReceivePack(ctx, conn, req, nil, verbose)
}
useRefDeltas := !adv.Capabilities.Supports(capability.OFSDelta)
9 unmodified lines
done <- pw.Close()
}()
err = sendReceivePack(ctx, conn, req, pr)
err = sendReceivePack(ctx, conn, req, pr, verbose)
_ = pr.Close()
encodeErr := <-done
if err != nil {
24 unmodified lines
return err
}
err = sendReceivePack(ctx, conn, req, pack)
err = sendReceivePack(ctx, conn, req, pack, verbose)
closeErr := pack.Close()
if err != nil {
return err
13 unmodified lines
if err != nil {
return err
}
return sendReceivePack(ctx, conn, req, nil)
return sendReceivePack(ctx, conn, req, nil, verbose)
}
func progressWriter(verbose bool) io.Writer {
2 unmodified lines
}
return os.Stderr
}
// progressSink returns a line-prefixing io.Writer suitable for
// sideband.Demuxer.Progress. When verbose is false it returns nil so the
// demuxer discards progress frames without allocating.
func progressSink(verbose bool, prefix string) io.Writer {
if !verbose {
return nil
}
return &prefixedLineWriter{w: os.Stderr, prefix: prefix, atLineStart: true}
}
// prefixedLineWriter prepends a fixed prefix to each line of input written
// to the wrapped writer. Git sideband progress arrives as chunks that may
// contain '\n' between full lines or '\r' for in-place updates ("Resolving
// deltas: 12%\r"); both are treated as line terminators so the next chunk
// gets a fresh prefix.
type prefixedLineWriter struct {
w io.Writer
prefix string
atLineStart bool
}
func (p *prefixedLineWriter) Write(b []byte) (int, error) {
consumed := 0
for len(b) > 0 {
if p.atLineStart {
if _, err := io.WriteString(p.w, p.prefix); err != nil {
return consumed, err
}
p.atLineStart = false
}
i := bytes.IndexAny(b, "\r\n")
var chunk []byte
if i < 0 {
chunk = b
} else {
chunk = b[:i+1]
p.atLineStart = true
}
n, err := p.w.Write(chunk)
consumed += n
if err != nil {
return consumed, err
}
b = b[len(chunk):]
}
return consumed, nil
}
Minternal/gitproto/push.go+64/-9
15 unmodified lines
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
1 unmodified line
85
86
87
26
88
89
90
91
15 unmodified lines
"github.com/go-git/go-git/v6/plumbing/transport"
)
func TestPrefixedLineWriter(t *testing.T) {
tests := []struct {
name string
writes []string
want string
}{
{
name: "single line with newline",
writes: []string{"counting objects: 42\n"},
want: "target: counting objects: 42\n",
},
{
name: "carriage returns are line terminators for in-place updates",
writes: []string{"resolving deltas: 10%\rresolving deltas: 50%\rresolving deltas: 100%\n"},
want: "target: resolving deltas: 10%\rtarget: resolving deltas: 50%\rtarget: resolving deltas: 100%\n",
},
{
name: "split across multiple writes",
writes: []string{"count", "ing ", "objects: 100\nresolving "},
want: "target: counting objects: 100\ntarget: resolving ",
},
{
name: "no trailing prefix when stream ends mid-line",
writes: []string{"partial progress"},
want: "target: partial progress",
},
{
name: "empty write is a noop",
writes: []string{"", "visible\n"},
want: "target: visible\n",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
var buf bytes.Buffer
pw := &prefixedLineWriter{w: &buf, prefix: "target: ", atLineStart: true}
for _, chunk := range tc.writes {
n, err := pw.Write([]byte(chunk))
if err != nil {
t.Fatalf("Write(%q): %v", chunk, err)
}
if n != len(chunk) {
t.Fatalf("Write(%q) consumed %d, want %d", chunk, n, len(chunk))
}
}
if got := buf.String(); got != tc.want {
t.Fatalf("output = %q, want %q", got, tc.want)
}
})
}
}
func TestProgressSinkNilWhenNotVerbose(t *testing.T) {
if got := progressSink(false, "anything: "); got != nil {
t.Fatalf("progressSink(false) = %T, want nil", got)
}
if got := progressSink(true, "source: "); got == nil {
t.Fatal("progressSink(true) returned nil, want non-nil writer")
}
}
func TestOpenV2PackStreamCloseClosesBody(t *testing.T) {
body := &trackingReadCloser{
ReadCloser: io.NopCloser(bytes.NewBufferString(
1 unmodified line
)),
}
rc, err := openV2PackStream(body)
rc, err := openV2PackStream(body, false)
if err != nil {
t.Fatalf("openV2PackStream: %v", err)
}
Minternal/gitproto/push_test.go+63/-1
20 unmodified lines
21
22
23
24
25
26
27
28
29
30
20 unmodified lines
Protocol string // "v1" or "v2"
V1Adv *packp.AdvRefs
V2Caps *V2Capabilities
// Verbose, when true, streams source-side sideband progress ("Counting
// objects", "Compressing objects", ...) to stderr and asks the source
// upload-pack to emit progress by not sending the no-progress option.
Verbose bool
}
// ListSourceRefs discovers refs from the source using the configured protocol mode.
Minternal/gitproto/refs.go+4
41 unmodified lines
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
41 unmodified lines
return chain, nil
}
// FirstParentChainFromMap walks a first-parent map from tip back to root.
// The map key is a commit hash, the value is its first parent hash.
// A zero-value parent marks the root. Returns the chain in root-to-tip order.
func FirstParentChainFromMap(parents map[plumbing.Hash]plumbing.Hash, tip plumbing.Hash) ([]plumbing.Hash, error) {
chain := make([]plumbing.Hash, 0, len(parents))
current := tip
seen := make(map[plumbing.Hash]struct{}, len(parents))
for {
if _, ok := seen[current]; ok {
return nil, fmt.Errorf("cycle detected at %s", current)
}
seen[current] = struct{}{}
chain = append(chain, current)
parent, ok := parents[current]
if !ok {
return nil, fmt.Errorf("commit %s not found in parent map", current)
}
if parent.IsZero() {
break
}
current = parent
}
for i, j := 0, len(chain)-1; i < j; i, j = i+1, j-1 {
chain[i], chain[j] = chain[j], chain[i]
}
return chain, nil
}
// SampledCheckpointCandidates generates a set of candidate indices to probe,
// sorted from largest (preferred) to smallest.
func SampledCheckpointCandidates(lo, hi int, prevSpan int) []int {
Minternal/planner/checkpoint.go+28
160 unmodified lines
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
81 unmodified lines
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
160 unmodified lines
return plans, nil
}
// BuildReplicationPlans generates overwrite-oriented plans for replication mode.
// Divergent refs are updated directly rather than blocked.
func BuildReplicationPlans(
desired map[plumbing.ReferenceName]DesiredRef,
targetRefs map[plumbing.ReferenceName]plumbing.Hash,
managed map[plumbing.ReferenceName]ManagedTarget,
cfg PlanConfig,
) ([]BranchPlan, error) {
managed = copyManagedTargets(managed)
if cfg.Prune {
for targetRef := range targetRefs {
if _, ok := managed[targetRef]; ok {
continue
}
switch {
case targetRef.IsTag() && cfg.IncludeTags:
managed[targetRef] = ManagedTarget{Kind: RefKindTag, Label: targetRef.Short()}
case targetRef.IsBranch() && len(cfg.Mappings) == 0 && len(cfg.Branches) == 0:
managed[targetRef] = ManagedTarget{Kind: RefKindBranch, Label: targetRef.Short()}
}
}
}
targetNames := make([]plumbing.ReferenceName, 0, len(managed))
for name := range managed {
targetNames = append(targetNames, name)
}
sort.Slice(targetNames, func(i, j int) bool { return targetNames[i] < targetNames[j] })
plans := make([]BranchPlan, 0, len(targetNames))
for _, targetRef := range targetNames {
info := managed[targetRef]
want, existsInDesired := desired[targetRef]
targetHash, existsOnTarget := targetRefs[targetRef]
if !existsInDesired {
if cfg.Prune && existsOnTarget {
plans = append(plans, BranchPlan{
Branch: info.Label,
TargetRef: targetRef,
TargetHash: targetHash,
Kind: info.Kind,
Action: ActionDelete,
Reason: fmt.Sprintf("%s -> <deleted>", ShortHash(targetHash)),
})
}
continue
}
plans = append(plans, PlanReplicationRef(want, targetHash, existsOnTarget))
}
sort.Slice(plans, func(i, j int) bool {
return plans[i].TargetRef.String() < plans[j].TargetRef.String()
})
return plans, nil
}
func copyManagedTargets(input map[plumbing.ReferenceName]ManagedTarget) map[plumbing.ReferenceName]ManagedTarget {
out := make(map[plumbing.ReferenceName]ManagedTarget, len(input))
for k, v := range input {
out[k] = v
}
return out
}
// BuildBootstrapPlans creates plans for an empty-target bootstrap.
func BuildBootstrapPlans(
desired map[plumbing.ReferenceName]DesiredRef,
81 unmodified lines
return plan, nil
}
// PlanReplicationRef determines the overwrite-oriented action for a single ref.
func PlanReplicationRef(want DesiredRef, targetHash plumbing.Hash, existsOnTarget bool) BranchPlan {
plan := BranchPlan{
Branch: want.Label,
SourceRef: want.SourceRef,
TargetRef: want.TargetRef,
SourceHash: want.SourceHash,
TargetHash: targetHash,
Kind: want.Kind,
}
if want.SourceHash == targetHash {
plan.Action = ActionSkip
plan.Reason = fmt.Sprintf("%s already current", ShortHash(want.SourceHash))
return plan
}
if !existsOnTarget || targetHash.IsZero() {
plan.Action = ActionCreate
plan.Reason = fmt.Sprintf("%s -> <new>", ShortHash(want.SourceHash))
return plan
}
plan.Action = ActionUpdate
switch want.Kind {
case RefKindTag:
plan.Reason = fmt.Sprintf("%s -> %s (replicate tag overwrite)", ShortHash(targetHash), ShortHash(want.SourceHash))
default:
plan.Reason = fmt.Sprintf("%s -> %s (replicate overwrite)", ShortHash(targetHash), ShortHash(want.SourceHash))
}
return plan
}
// MaxAncestryDepth is the maximum number of commits to visit during a
// fast-forward ancestry check. This prevents full graph walks on very
// large histories (issue #16). Set high enough to avoid false negatives
Minternal/planner/planner.go+99
75 unmodified lines
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
681 unmodified lines
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
75 unmodified lines
}
}
func TestPlanReplicationRefOverwritesDivergence(t *testing.T) {
target := plumbing.NewHash("1111111111111111111111111111111111111111")
source := plumbing.NewHash("2222222222222222222222222222222222222222")
plan := PlanReplicationRef(DesiredRef{
Kind: RefKindBranch,
Label: "main",
SourceRef: plumbing.NewBranchReferenceName("main"),
TargetRef: plumbing.NewBranchReferenceName("main"),
SourceHash: source,
}, target, true)
if plan.Action != ActionUpdate {
t.Fatalf("expected update, got %s", plan.Action)
}
if plan.Reason == "" {
t.Fatalf("expected overwrite reason")
}
}
func TestPlanReplicationRefOverwritesTagRetarget(t *testing.T) {
target := plumbing.NewHash("1111111111111111111111111111111111111111")
source := plumbing.NewHash("2222222222222222222222222222222222222222")
plan := PlanReplicationRef(DesiredRef{
Kind: RefKindTag,
Label: "v1",
SourceRef: plumbing.NewTagReferenceName("v1"),
TargetRef: plumbing.NewTagReferenceName("v1"),
SourceHash: source,
}, target, true)
if plan.Action != ActionUpdate {
t.Fatalf("expected update, got %s", plan.Action)
}
if plan.Reason != "11111111 -> 22222222 (replicate tag overwrite)" {
t.Fatalf("unexpected reason: %s", plan.Reason)
}
}
func TestBuildReplicationPlansDoesNotMutateManaged(t *testing.T) {
// BuildReplicationPlans inserts prune-eligible orphan refs into a local
// copy of `managed`. Regression guard: it must not mutate the caller's map.
orphan := plumbing.NewBranchReferenceName("stale")
main := plumbing.NewBranchReferenceName("main")
managed := map[plumbing.ReferenceName]ManagedTarget{
main: {Kind: RefKindBranch, Label: "main"},
}
desired := map[plumbing.ReferenceName]DesiredRef{
main: {
Kind: RefKindBranch,
Label: "main",
SourceRef: main,
TargetRef: main,
SourceHash: plumbing.NewHash("2222222222222222222222222222222222222222"),
},
}
targetRefs := map[plumbing.ReferenceName]plumbing.Hash{
main: plumbing.NewHash("1111111111111111111111111111111111111111"),
orphan: plumbing.NewHash("3333333333333333333333333333333333333333"),
}
plans, err := BuildReplicationPlans(desired, targetRefs, managed, PlanConfig{Prune: true})
if err != nil {
t.Fatalf("BuildReplicationPlans: %v", err)
}
// The returned plans should include the orphan delete...
var sawDelete bool
for _, p := range plans {
if p.TargetRef == orphan && p.Action == ActionDelete {
sawDelete = true
}
}
if !sawDelete {
t.Fatalf("expected prune delete for orphan, got plans=%+v", plans)
}
// ...but the caller's managed map must remain unchanged.
if len(managed) != 1 {
t.Fatalf("caller's managed map was mutated: %+v", managed)
}
if _, ok := managed[orphan]; ok {
t.Fatalf("orphan leaked into caller's managed map")
}
}
func TestValidateMappingsRejectsDuplicateTargets(t *testing.T) {
_, err := validation.ValidateMappings([]RefMapping{
{Source: "main", Target: "stable"},
681 unmodified lines
}
}
func TestSupportsReplicateRelayToleratesNoThin(t *testing.T) {
// replicate tolerates "no-thin" targets because gitproto.FetchPack never
// requests the thin-pack capability, so the relayed pack is always
// self-contained and safe for no-thin receive-pack servers.
ok, reason := SupportsReplicateRelay(RelayTargetPolicy{CapabilitiesKnown: true, NoThin: true})
if !ok {
t.Fatalf("expected SupportsReplicateRelay to accept no-thin target, got reason=%s", reason)
}
if reason != "replicate-target-capable-no-thin" {
t.Fatalf("unexpected reason: %s", reason)
}
}
func TestSupportsReplicateRelayRejectsUnknownCapabilities(t *testing.T) {
ok, reason := SupportsReplicateRelay(RelayTargetPolicy{CapabilitiesKnown: false})
if ok {
t.Fatal("expected SupportsReplicateRelay=false when target capabilities are unknown")
}
if reason != "replicate-missing-target-capabilities" {
t.Fatalf("unexpected reason: %s", reason)
}
}
func TestCanReplicateRelayRejectsInvalidPlanAction(t *testing.T) {
plans := []BranchPlan{{
Branch: "main",
SourceRef: "refs/heads/main",
TargetRef: "refs/heads/main",
SourceHash: plumbing.NewHash("1111111111111111111111111111111111111111"),
TargetHash: plumbing.NewHash("2222222222222222222222222222222222222222"),
Kind: RefKindBranch,
Action: ActionDelete,
}}
ok, reason := CanReplicateRelay(plans)
if ok {
t.Fatal("expected CanReplicateRelay=false for delete action")
}
if reason != "replicate-branch-action-not-create-or-update" {
t.Fatalf("unexpected reason: %s", reason)
}
}
func TestCanFullTagCreateRelay(t *testing.T) {
plans := []BranchPlan{{
Branch: "v1.0",
Minternal/planner/planner_test.go+126
10 unmodified lines
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
87 unmodified lines
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
10 unmodified lines
NoThin bool
}
// SupportsReplicateRelay checks target-side capabilities required by
// replication mode before looking at planned ref actions.
//
// Replicate tolerates targets that advertise "no-thin" because our
// upload-pack client (gitproto.FetchPack) never requests the "thin-pack"
// capability, so the source never emits a thin pack. The relayed pack is
// always self-contained and safe to push to a no-thin receive-pack.
// If gitproto.FetchPack ever begins requesting thin-pack, this check must
// gain a matching fallback (omit the request, or explicitly advertise
// NoThin on the source request) when target.NoThin is set.
func SupportsReplicateRelay(target RelayTargetPolicy) (bool, string) {
if !target.CapabilitiesKnown {
return false, "replicate-missing-target-capabilities"
}
if target.NoThin {
return true, "replicate-target-capable-no-thin"
}
return true, "replicate-target-capable"
}
// CanBootstrapRelay checks whether all desired target refs are absent on the target,
// making a bootstrap relay possible.
func CanBootstrapRelay(
87 unmodified lines
return reason
}
}
// CanReplicateRelay checks whether replication mode can execute as a relay-only
// overwrite sync against the target.
func CanReplicateRelay(plans []BranchPlan) (bool, string) {
for _, plan := range plans {
switch plan.Kind {
case RefKindBranch:
if !plan.SourceRef.IsBranch() || !plan.TargetRef.IsBranch() {
return false, "replicate-non-branch-mapping"
}
if plan.Action != ActionCreate && plan.Action != ActionUpdate {
return false, "replicate-branch-action-not-create-or-update"
}
case RefKindTag:
if !plan.SourceRef.IsTag() || !plan.TargetRef.IsTag() {
return false, "replicate-non-tag-mapping"
}
if plan.Action != ActionCreate && plan.Action != ActionUpdate {
return false, "replicate-tag-action-not-create-or-update"
}
default:
return false, "replicate-unsupported-ref-kind"
}
}
return true, "replicate-overwrite-relay"
}
Minternal/planner/relay.go+46
11 unmodified lines
12
13
14
15
16
17
18
10 unmodified lines
29
30
31
31
32
33
34
35
17 unmodified lines
53
54
55
55
56
57
58
59
13 unmodified lines
73
74
75
75
76
77
78
79
4 unmodified lines
84
85
86
86
87
88
88
89
90
91
92
92
93
94
95
96
5 unmodified lines
102
103
104
104
105
106
107
108
15 unmodified lines
124
125
126
126
127
128
129
130
131
131
132
132
133
134
135
136
63 unmodified lines
200
201
202
202
203
204
205
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
6 unmodified lines
224
225
226
218
219
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
222
261
262
263
264
265
266
267
18 unmodified lines
286
287
288
247
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
3 unmodified lines
343
344
345
346
347
348
349
10 unmodified lines
360
361
362
363
364
365
366
367
37 unmodified lines
405
406
407
323
408
409
410
411
8 unmodified lines
420
421
422
338
423
424
425
426
1 unmodified line
428
429
430
346
431
432
433
434
350
351
352
353
354
355
356
435
436
437
438
439
440
441
442
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
443
444
445
446
447
448
449
450
451
452
453
454
377
455
456
457
458
459
460
381
461
462
463
384
464
465
386
387
388
389
390
391
392
393
394
466
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
467
468
469
470
471
472
473
474
475
476
477
430
431
432
433
434
435
436
437
438
439
440
441
442
443
478
479
480
481
445
446
482
483
484
485
486
448
487
488
489
451
452
453
454
455
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
457
509
510
511
512
459
460
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
463
464
465
530
531
532
533
534
468
469
470
471
472
473
474
535
536
537
538
539
540
541
542
476
477
478
479
480
481
482
483
484
543
486
544
545
546
489
490
491
492
493
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
496
497
498
499
564
565
566
567
568
569
570
571
501
502
503
504
505
572
573
574
575
576
507
508
509
510
577
578
579
580
581
582
513
514
515
516
517
518
519
520
521
583
584
585
586
587
588
589
590
591
592
593
524
525
526
527
528
529
530
531
532
533
594
595
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
596
597
598
599
600
2 unmodified lines
603
604
605
606
607
608
669
670
671
672
673
674
675
676
609
678
610
611
612
613
614
615
616
617
618
619
4 unmodified lines
624
625
626
689
627
628
629
630
3 unmodified lines
634
635
636
699
637
638
639
640
54 unmodified lines
695
696
697
760
761
698
699
700
701
702
703
704
767
705
706
707
708
11 unmodified lines
"log/slog"
"net/http"
"regexp"
"runtime"
"strconv"
"strings"
"sync"
10 unmodified lines
)
const (
defaultAutoBatchMaxPackBytes = 512 * 1024 * 1024
defaultTargetMaxPackBytes = 512 * 1024 * 1024
githubLargeRepoThresholdKB = 1536 * 1024
)
17 unmodified lines
DesiredRefs map[plumbing.ReferenceName]planner.DesiredRef
TargetRefs map[plumbing.ReferenceName]plumbing.Hash
MaxPackBytes int64
BatchMaxPack int64
TargetMaxPack int64
Verbose bool
Logger *slog.Logger
}
13 unmodified lines
type plannedBatch struct {
planner.BootstrapBatch
prefetchedPacks map[plumbing.Hash][]byte
chain []plumbing.Hash // full first-parent chain (root→tip) for subdividing on push failure
}
// Execute runs the bootstrap strategy (one-shot or batched).
4 unmodified lines
// GitHub large-repo preflight
if batchLimit, ok := githubBatchLimit(ctx, p); ok {
p.BatchMaxPack = batchLimit
p.TargetMaxPack = batchLimit
p.log("bootstrap github preflight selected batched mode",
"batch_max_pack_bytes", p.BatchMaxPack)
"target_max_pack_bytes", p.TargetMaxPack)
}
planTargetRefs := p.TargetRefs
if p.BatchMaxPack > 0 {
if p.TargetMaxPack > 0 {
planTargetRefs = adjustedBootstrapTargetRefs(p.DesiredRefs, p.TargetRefs)
}
plans, err := planner.BuildBootstrapPlans(p.DesiredRefs, planTargetRefs)
5 unmodified lines
Plans: plans, Relay: true, RelayMode: "bootstrap", RelayReason: relayReason,
}
if p.BatchMaxPack > 0 {
if p.TargetMaxPack > 0 {
return executeBatched(ctx, p, plans, result)
}
15 unmodified lines
pushErr := p.TargetPusher.PushPack(ctx, cmds, packReader)
_ = packReader.Close()
if pushErr != nil {
autoBatch, ok := autoBatchMaxPackBytes(p, pushErr)
autoBatch, ok := autoTargetMaxPackBytes(p, pushErr)
if !ok {
return result, fmt.Errorf("push target refs: %w", pushErr)
}
p.log("bootstrap retrying with batched mode after target rejection",
"batch_max_pack_bytes", autoBatch)
p.BatchMaxPack = autoBatch
"target_max_pack_bytes", autoBatch)
p.TargetMaxPack = autoBatch
return executeBatched(ctx, p, plans, result)
}
63 unmodified lines
}
}
batchLimit := p.BatchMaxPack
if p.MaxPackBytes > 0 && (batchLimit == 0 || p.MaxPackBytes < batchLimit) {
batchLimit = p.MaxPackBytes
}
// MaxPackBytes is the hard abort threshold for any single source fetch.
// TargetMaxPack controls checkpoint *placement* (how many batches) but
// should not cap individual fetches — the estimate may undercount, and
// the actual pack for a batch can legitimately exceed the planning
// heuristic. If the resulting pack is too large for the target's
// receive-pack, the push itself fails and resume handles retry.
fetchLimit := p.MaxPackBytes
// Track branch tips already pushed to the target so subsequent branches
// can advertise them as haves. Without this, the second branch in a
// multi-branch bootstrap re-sends all shared objects (e.g., linux's
// master and nocache-cleanup share ~99% of history).
completedRefs := planner.CopyRefHashMap(p.TargetRefs)
for _, batch := range batches {
result.PlannedBatchCount += len(batch.Checkpoints)
6 unmodified lines
current := batch.ResumeHash
startIdx, err := planner.BootstrapResumeIndex(batch.Checkpoints, batch.ResumeHash)
if err != nil {
return result, fmt.Errorf("resume bootstrap batch for %s: %w", batch.Plan.TargetRef, err)
if err != nil && !batch.ResumeHash.IsZero() && len(batch.chain) > 0 {
// Temp ref doesn't match any planned checkpoint (e.g., the user
// changed --target-max-pack-bytes between runs). If the hash is
// in the commit chain, reuse it as the starting point and re-plan
// remaining checkpoints — preserving already-pushed data.
if chainIdx := chainPosition(batch.chain, batch.ResumeHash); chainIdx >= 0 {
remaining := batch.chain[chainIdx+1:]
if len(remaining) > 0 {
numBatches := estimateBatchCount(int64(len(remaining)), p.TargetMaxPack)
batch.Checkpoints = evenCheckpoints(remaining, numBatches)
p.log("bootstrap batch resuming from stale temp ref",
"branch", batch.Plan.TargetRef.String(),
"resume_hash", planner.ShortHash(batch.ResumeHash),
"remaining_commits", len(remaining),
"new_batches", len(batch.Checkpoints))
startIdx = 0
err = nil
}
}
}
if err != nil && !batch.ResumeHash.IsZero() {
// Temp ref hash not in the chain at all — truly stale. Delete and start fresh.
p.log("bootstrap batch clearing stale temp ref",
"branch", batch.Plan.TargetRef.String(),
"temp_ref", batch.TempRef.String(),
"stale_hash", planner.ShortHash(batch.ResumeHash))
delCmds := []gitproto.PushCommand{{Name: batch.TempRef, Old: batch.ResumeHash, Delete: true}}
if delErr := p.TargetPusher.PushCommands(ctx, delCmds); delErr != nil {
return result, fmt.Errorf("delete stale temp ref %s: %w (original: %w)", batch.TempRef, delErr, err)
}
current = plumbing.ZeroHash
startIdx = 0
}
for idx := startIdx; idx < len(batch.Checkpoints); idx++ {
// Manual index loop: subdivide may insert checkpoints at the current
// index, so we must not auto-increment after a retry.
idx := startIdx
for idx < len(batch.Checkpoints) {
checkpoint := batch.Checkpoints[idx]
p.log("bootstrap batch push checkpoint",
"branch", batch.Plan.TargetRef.String(),
18 unmodified lines
})
}
packReader, err := packReaderForCheckpoint(ctx, p, batch, checkpoint, current, batchLimit)
packReader, err := packReaderForCheckpoint(ctx, p, batch, checkpoint, current, completedRefs, fetchLimit)
if err != nil {
return result, fmt.Errorf("fetch source batch pack for %s: %w", batch.Plan.TargetRef, err)
}
packReader = closeOnce(packReader)
// Peek at the PACK header (12 bytes) to get the object count.
// If the estimated pack size exceeds the batch limit, subdivide
// immediately instead of pushing a pack the target will reject.
// This avoids wasting a multi-GiB transfer on a doomed push.
if p.TargetMaxPack > 0 && len(batch.chain) > 0 {
subdivided := false
packReader, err = checkPackSizeAndSubdivide(packReader, p.TargetMaxPack, func() bool {
expanded := subdivideCheckpoints(batch.chain, current, batch.Checkpoints[idx:])
if len(expanded) > len(batch.Checkpoints[idx:]) {
p.log("bootstrap batch subdividing before push (pack header estimate)",
"branch", batch.Plan.TargetRef.String(),
"old_remaining", len(batch.Checkpoints[idx:]),
"new_remaining", len(expanded))
batch.Checkpoints = append(batch.Checkpoints[:idx], expanded...)
subdivided = true
return true
}
return false
})
if err != nil {
return result, fmt.Errorf("check pack size for %s: %w", batch.Plan.TargetRef, err)
}
if subdivided {
continue // retry at same idx with new (smaller) checkpoint
}
}
cmds := convert.PlansToPushCommands(stagePlans)
if err := p.TargetPusher.PushPack(ctx, cmds, packReader); err != nil {
_ = packReader.Close()
if isTargetBodyLimitError(err) && len(batch.chain) > 0 {
expanded := subdivideCheckpoints(batch.chain, current, batch.Checkpoints[idx:])
if len(expanded) > len(batch.Checkpoints[idx:]) {
p.log("bootstrap batch subdividing after target size rejection",
"branch", batch.Plan.TargetRef.String(),
"old_remaining", len(batch.Checkpoints[idx:]),
"new_remaining", len(expanded),
"error", err.Error())
batch.Checkpoints = append(batch.Checkpoints[:idx], expanded...)
continue // retry at same idx with new (smaller) checkpoint
}
}
return result, fmt.Errorf("push bootstrap batch for %s: %w", batch.Plan.TargetRef, err)
}
_ = packReader.Close()
3 unmodified lines
"batch_total", len(batch.Checkpoints))
current = checkpoint
result.BatchCount++
idx++
}
if current.IsZero() {
10 unmodified lines
if err := p.TargetPusher.PushCommands(ctx, cmds); err != nil {
return result, fmt.Errorf("delete bootstrap temp ref for %s: %w", batch.Plan.TargetRef, err)
}
completedRefs[batch.Plan.TargetRef] = batch.Plan.SourceHash
completedRefs[batch.TempRef] = batch.Plan.SourceHash
p.log("bootstrap batch branch finalized", "branch", batch.Plan.TargetRef.String())
}
37 unmodified lines
func planBatches(ctx context.Context, p Params, desired []planner.DesiredRef) ([]plannedBatch, error) {
out := make([]plannedBatch, 0, len(desired))
for _, ref := range desired {
checkpoints, prefetched, err := planCheckpointsWithCache(ctx, p, ref)
checkpoints, chain, err := planCheckpointsFromChain(ctx, p, ref)
if err != nil {
return nil, err
}
8 unmodified lines
ResumeHash: p.TargetRefs[planner.BootstrapTempRef(ref.TargetRef)],
Checkpoints: checkpoints,
},
prefetchedPacks: prefetched,
chain: chain,
})
}
return out, nil
1 unmodified line
// PlanCheckpoints plans the checkpoint hashes for a single branch during batched bootstrap.
func PlanCheckpoints(ctx context.Context, p Params, ref planner.DesiredRef) ([]plumbing.Hash, error) {
checkpoints, _, err := planCheckpointsWithCache(ctx, p, ref)
checkpoints, _, err := planCheckpointsFromChain(ctx, p, ref)
return checkpoints, err
}
func planCheckpointsWithCache(ctx context.Context, p Params, ref planner.DesiredRef) ([]plumbing.Hash, map[plumbing.Hash][]byte, error) {
cp, err := newCheckpointPlanner(ctx, p, ref)
if err != nil {
return nil, nil, err
}
return cp.plan()
}
// estimatedBytesPerCommit is the heuristic for estimating pack size from commit
// count. Real repos range from ~5 KiB/commit (small web apps) to ~120 KiB
// (blob-heavy monorepos); most mature repos fall in 20–80 KiB. 64 KiB produces
// accurate batch counts for large repos (linux is ~66 KiB/commit) while
// slightly overestimating for small ones (harmless — extra batches finish fast).
// The PACK header pre-check and target-rejection retry catch remaining error.
const estimatedBytesPerCommit = 65536
type probeResult struct {
tooLarge bool
data []byte
}
type checkpointPlanner struct {
ctx context.Context
params Params
ref planner.DesiredRef
chain []plumbing.Hash
probeCache map[string]probeResult
prefetched map[plumbing.Hash][]byte
}
func newCheckpointPlanner(ctx context.Context, p Params, ref planner.DesiredRef) (*checkpointPlanner, error) {
func planCheckpointsFromChain(ctx context.Context, p Params, ref planner.DesiredRef) ([]plumbing.Hash, []plumbing.Hash, error) {
p.log("bootstrap batch fetching commit graph", "branch", ref.TargetRef.String())
// Fetch all commits (tree:0 filter) into a temporary in-memory store.
// For linux this is ~1.4M commits at ~3.3 KiB each = ~4.6 GB transient.
// We extract the first-parent chain (~75k hashes, ~3 MB) immediately
// and discard the store so GC can reclaim the 4.6 GB. The transient
// spike is unavoidable with git protocol v2 (no first-parent-only
// fetch) and go-git's pack parser (needs full store for delta resolution).
graphStore := memory.NewStorage()
gpRef := gitproto.DesiredRef{SourceRef: ref.SourceRef, TargetRef: ref.TargetRef, SourceHash: ref.SourceHash}
if err := p.SourceService.FetchCommitGraph(ctx, graphStore, p.SourceConn, gpRef); err != nil {
return nil, fmt.Errorf("fetch bootstrap planning graph for %s: %w", ref.TargetRef, err)
return nil, nil, fmt.Errorf("fetch bootstrap planning graph for %s: %w", ref.TargetRef, err)
}
chain, err := planner.FirstParentChain(graphStore, ref.SourceHash)
graphStore = nil // allow GC to reclaim the commit graph store (~4.6 GB for linux)
runtime.GC()
if err != nil {
return nil, fmt.Errorf("walk first-parent chain for %s: %w", ref.TargetRef, err)
return nil, nil, fmt.Errorf("walk first-parent chain for %s: %w", ref.TargetRef, err)
}
if len(chain) == 0 {
return nil, fmt.Errorf("empty first-parent chain for %s", ref.TargetRef)
return nil, nil, fmt.Errorf("empty first-parent chain for %s", ref.TargetRef)
}
return &checkpointPlanner{
ctx: ctx,
params: p,
ref: ref,
chain: chain,
probeCache: make(map[string]probeResult),
prefetched: make(map[plumbing.Hash][]byte),
}, nil
}
func (p *checkpointPlanner) plan() ([]plumbing.Hash, map[plumbing.Hash][]byte, error) {
checkpoints := make([]plumbing.Hash, 0, len(p.chain))
prevIdx := -1
prevHash := plumbing.ZeroHash
prevSpan := initialCheckpointSpan(p.params.BatchMaxPack, len(p.chain))
prevMeasuredBytes := 0
for prevIdx < len(p.chain)-1 {
bestIdx, err := p.selectNextCheckpoint(prevIdx, prevHash, prevSpan, prevMeasuredBytes)
if err != nil {
return nil, nil, err
}
if bestIdx <= prevIdx {
return nil, nil, fmt.Errorf("could not find bootstrap checkpoint for %s under batch-max-pack-bytes=%d", p.ref.TargetRef, p.params.BatchMaxPack)
}
if result, ok := p.cachedProbe(prevHash, bestIdx); ok && !result.tooLarge && len(result.data) > 0 {
p.prefetched[p.chain[bestIdx]] = result.data
prevSpan = adaptiveNextProbeSpan(p.params.BatchMaxPack, len(result.data), bestIdx-prevIdx, len(p.chain)-1-bestIdx)
prevMeasuredBytes = len(result.data)
} else {
prevSpan = bestIdx - prevIdx
prevMeasuredBytes = 0
}
prevIdx = bestIdx
prevHash = p.chain[bestIdx]
checkpoints = append(checkpoints, prevHash)
p.params.log("bootstrap batch planned checkpoint",
"branch", p.ref.TargetRef.String(),
"checkpoint", planner.ShortHash(prevHash),
"selected", len(checkpoints),
"chain_len", len(p.chain))
}
return checkpoints, p.prefetched, nil
numBatches := estimateBatchCount(int64(len(chain)), p.TargetMaxPack)
checkpoints := evenCheckpoints(chain, numBatches)
p.log("bootstrap batch planned checkpoints",
"branch", ref.TargetRef.String(),
"chain_len", len(chain),
"estimated_batches", len(checkpoints))
return checkpoints, chain, nil
}
func (p *checkpointPlanner) selectNextCheckpoint(prevIdx int, prevHash plumbing.Hash, prevSpan int, prevMeasuredBytes int) (int, error) {
bestIdx := -1
remaining := len(p.chain) - 1 - prevIdx
if shouldSelectTipWithoutProbe(p.params.BatchMaxPack, prevMeasuredBytes, prevSpan, remaining) {
bestIdx = len(p.chain) - 1
} else if shouldProbeTipFirst(p.params.BatchMaxPack, prevMeasuredBytes, prevSpan, remaining) {
tipIdx := len(p.chain) - 1
tooLarge, err := p.probe(prevHash, tipIdx)
if err != nil {
return -1, err
}
if !tooLarge {
bestIdx = tipIdx
}
func estimateBatchCount(chainLen int64, batchMaxPack int64) int {
if batchMaxPack <= 0 || chainLen <= 0 {
return 1
}
if bestIdx != -1 {
return bestIdx, nil
estimated := chainLen * estimatedBytesPerCommit
n := int((estimated + batchMaxPack - 1) / batchMaxPack)
if n < 1 {
n = 1
}
return p.searchCheckpointUnderLimit(prevIdx, prevHash, prevSpan)
return n
}
func (p *checkpointPlanner) probe(prevHash plumbing.Hash, idx int) (bool, error) {
if result, ok := p.cachedProbe(prevHash, idx); ok {
return result.tooLarge, nil
}
data, tooLarge, err := fetchPackForProbe(p.ctx, p.params, p.ref, p.chain[idx], prevHash, p.params.BatchMaxPack)
// estimatedBytesPerObject is a conservative average for compressed git objects
// in a packfile. Used with the PACK header's object count to estimate total
// pack size before streaming the full pack. Real values range from ~200 bytes
// (tiny commits in a sparse repo) to ~2 KiB (blob-heavy repos), with most
// mature repos averaging 500–1000 bytes. 750 is a reasonable middle ground.
const estimatedBytesPerObject = 750
// checkPackSizeAndSubdivide reads the 12-byte PACK header to get the object
// count, estimates total pack size, and if it exceeds batchLimit, closes the
// reader and calls subdivide(). Returns (nil, nil) when subdivided (caller
// should continue to retry), or (prepended reader, nil) to proceed with push.
func checkPackSizeAndSubdivide(
r io.ReadCloser,
batchLimit int64,
subdivide func() bool,
) (io.ReadCloser, error) {
var header [12]byte
n, err := io.ReadFull(r, header[:])
if err != nil {
return false, fmt.Errorf("measure bootstrap batch for %s at %s: %w", p.ref.TargetRef, planner.ShortHash(p.chain[idx]), err)
// Short pack or error — let the push handle it
prefixed := io.MultiReader(bytes.NewReader(header[:n]), r)
return &wrappedMultiRC{Reader: prefixed, Closer: r}, nil
}
p.probeCache[probeKey(prevHash, idx)] = probeResult{tooLarge: tooLarge, data: data}
return tooLarge, nil
if string(header[:4]) != "PACK" {
// Not a standard packfile — can't estimate, proceed
prefixed := io.MultiReader(bytes.NewReader(header[:]), r)
return &wrappedMultiRC{Reader: prefixed, Closer: r}, nil
}
objectCount := int64(header[8])<<24 | int64(header[9])<<16 | int64(header[10])<<8 | int64(header[11])
estimated := objectCount * estimatedBytesPerObject
if estimated > batchLimit && subdivide() {
_ = r.Close()
return nil, nil
}
prefixed := io.MultiReader(bytes.NewReader(header[:]), r)
return &wrappedMultiRC{Reader: prefixed, Closer: r}, nil
}
func (p *checkpointPlanner) cachedProbe(prevHash plumbing.Hash, idx int) (probeResult, bool) {
result, ok := p.probeCache[probeKey(prevHash, idx)]
return result, ok
type wrappedMultiRC struct {
io.Reader
io.Closer
}
func (p *checkpointPlanner) probeBounds(prevHash plumbing.Hash, lo, hi int) (largestFit int, smallestTooLarge int) {
largestFit = lo - 1
smallestTooLarge = hi + 1
for idx := lo; idx <= hi; idx++ {
result, ok := p.cachedProbe(prevHash, idx)
if !ok {
continue
func (w *wrappedMultiRC) Read(p []byte) (int, error) { return w.Reader.Read(p) }
// chainPosition returns the index of hash in chain, or -1 if not found.
func chainPosition(chain []plumbing.Hash, hash plumbing.Hash) int {
for i, h := range chain {
if h == hash {
return i
}
if result.tooLarge {
if idx < smallestTooLarge {
smallestTooLarge = idx
}
continue
}
if idx > largestFit {
largestFit = idx
}
}
return largestFit, smallestTooLarge
return -1
}
func (p *checkpointPlanner) searchCheckpointUnderLimit(prevIdx int, prevHash plumbing.Hash, prevSpan int) (int, error) {
lo := prevIdx + 1
hi := len(p.chain) - 1
if lo > hi {
return -1, nil
// subdivideCheckpoints splits each remaining checkpoint range in half using
// the full commit chain. Called when a batch push is rejected for exceeding
// the target's body-size limit. Returns the expanded checkpoint list; if no
// split is possible (ranges are already 1 commit), returns the input unchanged.
func subdivideCheckpoints(chain []plumbing.Hash, current plumbing.Hash, remaining []plumbing.Hash) []plumbing.Hash {
chainIdx := make(map[plumbing.Hash]int, len(chain))
for i, h := range chain {
chainIdx[h] = i
}
curIdx, ok := chainIdx[current]
if !ok && !current.IsZero() {
return remaining
}
if current.IsZero() {
curIdx = -1
}
largestFit, smallestTooLarge := p.probeBounds(prevHash, lo, hi)
if smallestTooLarge > hi {
if largestFit >= lo {
return largestFit, nil
expanded := make([]plumbing.Hash, 0, len(remaining)*2)
prev := curIdx
for _, cp := range remaining {
cpIdx, ok := chainIdx[cp]
if !ok {
expanded = append(expanded, cp)
continue
}
idx := nextCheckpointProbeCandidate(lo, hi, prevSpan, largestFit, smallestTooLarge)
tooLarge, err := p.probe(prevHash, idx)
if err != nil {
return -1, err
gap := cpIdx - prev
if gap > 1 {
midIdx := prev + gap/2
expanded = append(expanded, chain[midIdx])
}
if !tooLarge {
return idx, nil
}
smallestTooLarge = idx
expanded = append(expanded, cp)
prev = cpIdx
}
return expanded
}
for smallestTooLarge-largestFit > 1 {
idx := nextCheckpointProbeCandidate(lo, hi, prevSpan, largestFit, smallestTooLarge)
if idx <= largestFit {
idx = largestFit + 1
}
if idx >= smallestTooLarge {
idx = smallestTooLarge - 1
}
if idx < lo || idx > hi || idx <= largestFit || idx >= smallestTooLarge {
func evenCheckpoints(chain []plumbing.Hash, numBatches int) []plumbing.Hash {
if numBatches <= 1 || len(chain) <= 1 || numBatches >= len(chain) {
return []plumbing.Hash{chain[len(chain)-1]}
}
checkpoints := make([]plumbing.Hash, 0, numBatches)
batchSize := len(chain) / numBatches
for i := 0; i < numBatches-1; i++ {
idx := (i+1)*batchSize - 1
if idx >= len(chain)-1 {
break
}
tooLarge, err := p.probe(prevHash, idx)
if err != nil {
return -1, err
}
if tooLarge {
smallestTooLarge = idx
continue
}
largestFit = idx
checkpoints = append(checkpoints, chain[idx])
}
if largestFit >= lo {
return largestFit, nil
}
tooLarge, err := p.probe(prevHash, lo)
if err != nil {
return -1, err
}
if tooLarge {
return -1, nil
}
return lo, nil
}
func nextCheckpointProbeCandidate(lo, hi, prevSpan, largestFit, smallestTooLarge int) int {
if smallestTooLarge <= hi {
idx := largestFit + (smallestTooLarge-largestFit)/2
if idx <= largestFit {
return largestFit + 1
}
if idx >= smallestTooLarge {
return smallestTooLarge - 1
}
return idx
}
projected := lo
if prevSpan > 0 {
projected = lo + prevSpan - 1
}
if projected < lo {
projected = lo
}
if projected > hi {
projected = hi
}
if projected <= largestFit {
return hi
}
return projected
}
func probeKey(prevHash plumbing.Hash, idx int) string {
return prevHash.String() + ":" + strconv.Itoa(idx)
}
func initialCheckpointSpan(batchMaxPack int64, chainLen int) int {
// Issue #14: Use a commit-count heuristic for the initial span estimate
// to reduce expensive fetch-and-discard probes. Typical compressed commit
// + tree overhead averages ~2-5 KiB per commit in a mature repo.
const avgBytesPerCommit = 4096
if batchMaxPack <= 0 {
return 0
}
initialSpan := int(batchMaxPack / avgBytesPerCommit)
if initialSpan > chainLen {
initialSpan = chainLen
}
if initialSpan < 1 {
initialSpan = 1
}
return initialSpan
}
func adaptiveNextProbeSpan(limit int64, measuredBytes int, selectedSpan int, remaining int) int {
if selectedSpan < 1 {
selectedSpan = 1
}
if remaining < 1 {
return selectedSpan
}
if limit <= 0 || measuredBytes <= 0 {
if selectedSpan > remaining {
return remaining
}
return selectedSpan
}
next := int((int64(selectedSpan) * limit) / int64(measuredBytes))
if next < 1 {
next = 1
}
if next > remaining {
next = remaining
}
return next
}
func shouldProbeTipFirst(limit int64, measuredBytes int, measuredSpan int, remaining int) bool {
if limit <= 0 || measuredBytes <= 0 || measuredSpan <= 0 || remaining <= measuredSpan {
return false
}
estimated := (int64(measuredBytes) * int64(remaining)) / int64(measuredSpan)
return estimated <= (limit*9)/10
}
func shouldSelectTipWithoutProbe(limit int64, measuredBytes int, measuredSpan int, remaining int) bool {
if limit <= 0 || measuredBytes <= 0 || measuredSpan <= 0 || remaining <= 0 {
return false
}
if remaining <= measuredSpan {
return int64(measuredBytes) <= limit/2
}
estimated := (int64(measuredBytes) * int64(remaining)) / int64(measuredSpan)
return estimated <= limit/2
}
func fetchPackForProbe(ctx context.Context, p Params, ref planner.DesiredRef, want, have plumbing.Hash, limit int64) ([]byte, bool, error) {
desired := singleGP(ref.SourceRef, ref.TargetRef, want)
haves := planner.SingleHaveMap(have)
packReader, err := p.SourceService.FetchPack(ctx, p.SourceConn, desired, haves)
if err != nil {
return nil, false, err
}
defer packReader.Close()
data, err := io.ReadAll(gitproto.LimitPackReader(packReader, limit))
if err == nil {
return data, false, nil
}
if strings.Contains(err.Error(), "source pack exceeded max-pack-bytes limit") {
return nil, true, nil
}
return nil, false, err
checkpoints = append(checkpoints, chain[len(chain)-1])
return checkpoints
}
func packReaderForCheckpoint(
2 unmodified lines
batch plannedBatch,
checkpoint plumbing.Hash,
current plumbing.Hash,
completedRefs map[plumbing.ReferenceName]plumbing.Hash,
batchLimit int64,
) (io.ReadCloser, error) {
if data, ok := batch.prefetchedPacks[checkpoint]; ok && len(data) > 0 {
p.log("bootstrap batch reusing prefetched probe pack",
"branch", batch.Plan.TargetRef.String(),
"checkpoint", planner.ShortHash(checkpoint),
"bytes", len(data))
return io.NopCloser(bytes.NewReader(data)), nil
}
desired := singleGP(batch.Plan.SourceRef, batch.TempRef, checkpoint)
haves := planner.SingleHaveMap(current)
// Merge the current checkpoint (within this branch) with any
// already-completed branch tips so the source can delta-compress
// against objects the target already has from previous branches.
haves := planner.CopyRefHashMap(completedRefs)
if !current.IsZero() {
haves[batch.TempRef] = current
}
packReader, err := p.SourceService.FetchPack(ctx, p.SourceConn, desired, haves)
if err != nil {
return nil, err
4 unmodified lines
// --- GitHub preflight ---
func githubBatchLimit(ctx context.Context, p Params) (int64, bool) {
if p.BatchMaxPack > 0 || p.SourceConn == nil || p.SourceConn.Endpoint == nil {
if p.TargetMaxPack > 0 || p.SourceConn == nil || p.SourceConn.Endpoint == nil {
return 0, false
}
if p.SourceService == nil || !p.SourceService.SupportsBootstrapBatch() {
3 unmodified lines
if !ok || repoSizeKB < githubLargeRepoThresholdKB {
return 0, false
}
limit := int64(defaultAutoBatchMaxPackBytes)
limit := int64(defaultTargetMaxPackBytes)
if p.MaxPackBytes > 0 && p.MaxPackBytes < limit {
limit = p.MaxPackBytes
}
54 unmodified lines
return parts[0], parts[1], true
}
func autoBatchMaxPackBytes(p Params, err error) (int64, bool) {
if p.BatchMaxPack > 0 || !isTargetBodyLimitError(err) {
func autoTargetMaxPackBytes(p Params, err error) (int64, bool) {
if p.TargetMaxPack > 0 || !isTargetBodyLimitError(err) {
return 0, false
}
if p.SourceService == nil || !p.SourceService.SupportsBootstrapBatch() {
return 0, false
}
limit := int64(defaultAutoBatchMaxPackBytes)
limit := int64(defaultTargetMaxPackBytes)
if targetLimit := targetBodyLimit(err); targetLimit > 0 {
derived := targetLimit / 2
if derived <= 0 {
Minternal/strategy/bootstrap/bootstrap.go+245/-307
127 unmodified lines
128
129
130
131
131
132
133
134
135
136
137
134
135
136
137
138
141
142
143
144
145
146
139
140
141
142
143
144
149
150
151
152
153
154
145
146
147
148
149
150
157
158
159
160
161
151
152
153
154
155
156
165
166
167
168
169
170
157
158
159
160
161
162
173
174
175
176
177
178
163
164
165
166
167
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
168
203
204
205
206
207
208
169
170
171
172
173
174
211
212
213
214
215
216
175
176
177
178
179
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
180
181
182
183
238
184
185
240
241
186
187
188
189
190
191
192
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
193
194
195
196
197
198
199
200
201
282
283
284
285
286
287
288
289
290
291
202
203
204
205
206
207
208
209
210
211
212
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
332
333
334
335
336
337
338
339
340
341
232
233
234
235
236
237
238
239
240
241
242
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
364
365
366
367
368
369
370
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
312
313
314
315
316
317
423
318
319
320
321
322
323
324
325
326
426
427
428
429
430
327
328
329
330
331
332
333
334
335
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
336
337
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
338
339
340
341
342
343
344
345
346
347
348
349
350
351
478
479
480
481
482
483
484
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
191 unmodified lines
586
587
588
682
589
590
591
2 unmodified lines
594
595
596
691
692
693
694
695
696
697
698
699
700
597
598
599
600
11 unmodified lines
612
613
614
718
615
616
617
618
619
723
724
725
620
621
622
3 unmodified lines
626
627
628
735
629
630
631
2 unmodified lines
634
635
636
744
745
746
747
748
749
750
751
752
753
637
638
639
640
12 unmodified lines
653
654
655
772
656
657
658
659
660
777
778
779
661
662
663
30 unmodified lines
694
695
696
816
697
698
699
700
127 unmodified lines
}
}
func TestAdaptiveNextProbeSpan(t *testing.T) {
func TestEstimateBatchCount(t *testing.T) {
tests := []struct {
name string
limit int64
measured int
selectedSpan int
remaining int
chainLen int64
batchMaxPack int64
want int
}{
{
name: "grows span when measured pack is far below limit",
limit: 1000,
measured: 250,
selectedSpan: 4,
remaining: 20,
want: 16,
name: "zero chain length returns 1",
chainLen: 0,
batchMaxPack: 1024 * 1024,
want: 1,
},
{
name: "caps growth at remaining commits",
limit: 1000,
measured: 200,
selectedSpan: 4,
remaining: 6,
want: 6,
name: "negative chain length returns 1",
chainLen: -5,
batchMaxPack: 1024 * 1024,
want: 1,
},
{
name: "keeps minimum span of one",
limit: 1000,
measured: 4000,
selectedSpan: 1,
remaining: 10,
name: "zero batch max pack returns 1",
chainLen: 100,
batchMaxPack: 0,
want: 1,
},
{
name: "falls back to selected span without measurements",
limit: 1000,
measured: 0,
selectedSpan: 3,
remaining: 10,
want: 3,
name: "negative batch max pack returns 1",
chainLen: 100,
batchMaxPack: -1,
want: 1,
},
{
name: "caps fallback span at remaining",
limit: 0,
measured: 0,
selectedSpan: 8,
remaining: 5,
want: 5,
name: "small chain fitting in one batch",
chainLen: 10,
batchMaxPack: 10 * estimatedBytesPerCommit,
want: 1,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := adaptiveNextProbeSpan(tt.limit, tt.measured, tt.selectedSpan, tt.remaining)
if got != tt.want {
t.Fatalf("adaptiveNextProbeSpan(%d, %d, %d, %d) = %d, want %d",
tt.limit, tt.measured, tt.selectedSpan, tt.remaining, got, tt.want)
}
})
}
}
func TestShouldProbeTipFirst(t *testing.T) {
tests := []struct {
name string
limit int64
measured int
measuredSpan int
remaining int
want bool
}{
{
name: "probes tip when estimate is comfortably under limit",
limit: 1000,
measured: 200,
measuredSpan: 4,
remaining: 10,
want: true,
name: "large chain needing multiple batches",
chainLen: 1000,
batchMaxPack: 100 * estimatedBytesPerCommit,
want: 10,
},
{
name: "does not probe tip when estimate is too close to limit",
limit: 1000,
measured: 400,
measuredSpan: 4,
remaining: 10,
want: false,
name: "ceil division when not evenly divisible",
chainLen: 101,
batchMaxPack: 100 * estimatedBytesPerCommit,
want: 2,
},
{
name: "does not probe tip without measurements",
limit: 1000,
measured: 0,
measuredSpan: 4,
remaining: 10,
want: false,
},
{
name: "does not probe tip when remaining span is not larger",
limit: 1000,
measured: 200,
measuredSpan: 4,
remaining: 4,
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := shouldProbeTipFirst(tt.limit, tt.measured, tt.measuredSpan, tt.remaining)
got := estimateBatchCount(tt.chainLen, tt.batchMaxPack)
if got != tt.want {
t.Fatalf("shouldProbeTipFirst(%d, %d, %d, %d) = %v, want %v",
tt.limit, tt.measured, tt.measuredSpan, tt.remaining, got, tt.want)
t.Fatalf("estimateBatchCount(%d, %d) = %d, want %d",
tt.chainLen, tt.batchMaxPack, got, tt.want)
}
})
}
}
func TestShouldSelectTipWithoutProbe(t *testing.T) {
tests := []struct {
name string
limit int64
measured int
measuredSpan int
remaining int
want bool
}{
{
name: "selects tip when estimate is far below limit",
limit: 1000,
measured: 150,
measuredSpan: 4,
remaining: 10,
want: true,
},
{
name: "does not select tip when estimate is only moderately below limit",
limit: 1000,
measured: 220,
measuredSpan: 4,
remaining: 10,
want: false,
},
{
name: "does not select tip without measurements",
limit: 1000,
measured: 0,
measuredSpan: 4,
remaining: 10,
want: false,
},
func TestEvenCheckpoints(t *testing.T) {
makeHashes := func(n int) []plumbing.Hash {
hashes := make([]plumbing.Hash, n)
for i := range hashes {
hashes[i] = plumbing.NewHash(fmt.Sprintf("%040d", i))
}
return hashes
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := shouldSelectTipWithoutProbe(tt.limit, tt.measured, tt.measuredSpan, tt.remaining)
if got != tt.want {
t.Fatalf("shouldSelectTipWithoutProbe(%d, %d, %d, %d) = %v, want %v",
tt.limit, tt.measured, tt.measuredSpan, tt.remaining, got, tt.want)
}
})
}
}
t.Run("1 batch returns just tip", func(t *testing.T) {
chain := makeHashes(10)
got := evenCheckpoints(chain, 1)
if len(got) != 1 {
t.Fatalf("len = %d, want 1", len(got))
}
if got[0] != chain[9] {
t.Fatalf("got %s, want tip %s", got[0], chain[9])
}
})
func TestNextCheckpointProbeCandidate(t *testing.T) {
tests := []struct {
name string
lo int
hi int
prevSpan int
largestFit int
smallestTooLarge int
want int
}{
{
name: "probes midpoint between known fit and too large bounds",
lo: 0,
hi: 19,
prevSpan: 8,
largestFit: 7,
smallestTooLarge: 15,
want: 11,
},
{
name: "projects previous span when no upper bound exists",
lo: 5,
hi: 19,
prevSpan: 4,
largestFit: 4,
smallestTooLarge: 20,
want: 8,
},
{
name: "falls back to hi when projection is behind known fit",
lo: 5,
hi: 19,
prevSpan: 1,
largestFit: 12,
smallestTooLarge: 20,
want: 19,
},
}
t.Run("3 batches on 10-element chain", func(t *testing.T) {
chain := makeHashes(10)
got := evenCheckpoints(chain, 3)
// batchSize = 10/3 = 3
// checkpoints at indices: (1)*3-1=2, (2)*3-1=5, then tip=9
if len(got) != 3 {
t.Fatalf("len = %d, want 3", len(got))
}
if got[0] != chain[2] {
t.Fatalf("got[0] = %s, want chain[2] = %s", got[0], chain[2])
}
if got[1] != chain[5] {
t.Fatalf("got[1] = %s, want chain[5] = %s", got[1], chain[5])
}
if got[2] != chain[9] {
t.Fatalf("got[2] = %s, want chain[9] (tip) = %s", got[2], chain[9])
}
})
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := nextCheckpointProbeCandidate(tt.lo, tt.hi, tt.prevSpan, tt.largestFit, tt.smallestTooLarge)
if got != tt.want {
t.Fatalf("nextCheckpointProbeCandidate(%d, %d, %d, %d, %d) = %d, want %d",
tt.lo, tt.hi, tt.prevSpan, tt.largestFit, tt.smallestTooLarge, got, tt.want)
}
})
}
}
t.Run("more batches than chain with single element returns just tip", func(t *testing.T) {
chain := makeHashes(1)
got := evenCheckpoints(chain, 5)
if len(got) != 1 {
t.Fatalf("len = %d, want 1", len(got))
}
if got[0] != chain[0] {
t.Fatalf("got %s, want tip %s", got[0], chain[0])
}
})
func TestPlanCheckpointsProbeCountVisibility(t *testing.T) {
tests := []struct {
name string
batchLimit int64
spanSizes map[int]int
wantProbes int
}{
{
name: "small early sample selects tip without a second probe",
batchLimit: 20_000,
spanSizes: map[int]int{1: 1_000, 2: 2_000, 3: 3_000, 4: 4_000, 5: 5_000, 6: 6_000, 7: 7_000, 8: 8_000, 9: 9_000, 10: 10_000},
wantProbes: 1,
},
{
name: "mid-sized spans require repeated exact probes",
batchLimit: 10_000,
spanSizes: map[int]int{1: 4_500, 2: 9_000, 3: 20_000, 4: 20_000, 5: 20_000, 6: 20_000, 7: 20_000, 8: 20_000, 9: 20_000, 10: 20_000},
wantProbes: 5,
},
t.Run("more batches than chain with multi-element chain returns just tip", func(t *testing.T) {
chain := makeHashes(3)
got := evenCheckpoints(chain, 10)
if len(got) != 1 {
t.Fatalf("len = %d, want 1", len(got))
}
if got[0] != chain[2] {
t.Fatalf("got %s, want tip %s", got[0], chain[2])
}
})
}
func TestCheckPackSizeAndSubdivide(t *testing.T) {
// Build a minimal PACK header: "PACK" + version 2 + object count
makePackHeader := func(objectCount uint32) []byte {
var h [12]byte
copy(h[:4], "PACK")
h[4], h[5], h[6], h[7] = 0, 0, 0, 2 // version 2
h[8] = byte(objectCount >> 24)
h[9] = byte(objectCount >> 16)
h[10] = byte(objectCount >> 8)
h[11] = byte(objectCount)
return h[:]
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
hashes := makeLinearCommitChain(t, 10)
indices := make(map[plumbing.Hash]int, len(hashes))
for i, h := range hashes {
indices[h] = i
}
t.Run("small pack proceeds without subdivide", func(t *testing.T) {
header := makePackHeader(100) // 100 * 750 = 75000 bytes estimated
body := append(header, []byte("packdata")...)
r := io.NopCloser(bytes.NewReader(body))
subdivided := false
got, err := checkPackSizeAndSubdivide(r, 1_000_000, func() bool {
subdivided = true
return true
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got == nil {
t.Fatal("expected non-nil reader")
}
if subdivided {
t.Fatal("should not subdivide small pack")
}
// Verify the PACK header was prepended back
out, _ := io.ReadAll(got)
if string(out[:4]) != "PACK" {
t.Fatalf("expected PACK header preserved, got %q", out[:4])
}
})
probes := 0
source := fakeBootstrapSource{
fetchCommitGraph: func(_ context.Context, store storer.Storer, _ *gitproto.Conn, _ gitproto.DesiredRef) error {
_ = writeLinearCommitChain(t, store, len(hashes))
return nil
},
fetchPack: func(_ context.Context, _ *gitproto.Conn, desired map[plumbing.ReferenceName]gitproto.DesiredRef, haves map[plumbing.ReferenceName]plumbing.Hash) (io.ReadCloser, error) {
probes++
for _, ref := range desired {
wantIdx, ok := indices[ref.SourceHash]
if !ok {
t.Fatalf("unexpected probe hash %s", ref.SourceHash)
}
haveIdx := -1
for _, h := range haves {
if h.IsZero() {
continue
}
if idx, ok := indices[h]; ok {
haveIdx = idx
}
}
span := wantIdx - haveIdx
size, ok := tt.spanSizes[span]
if !ok {
t.Fatalf("missing synthetic pack size for span %d", span)
}
return io.NopCloser(bytes.NewReader(make([]byte, size))), nil
}
t.Fatal("expected single desired ref")
return nil, nil
},
}
t.Run("large pack triggers subdivide", func(t *testing.T) {
header := makePackHeader(5_000_000) // 5M * 750 = 3.75 GiB estimated
r := io.NopCloser(bytes.NewReader(header))
subdivided := false
got, err := checkPackSizeAndSubdivide(r, 2_000_000_000, func() bool {
subdivided = true
return true
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got != nil {
t.Fatal("expected nil reader after subdivide")
}
if !subdivided {
t.Fatal("expected subdivide for large pack")
}
})
_, err := PlanCheckpoints(context.Background(), Params{
SourceService: source,
BatchMaxPack: tt.batchLimit,
}, planner.DesiredRef{
Label: "main",
Kind: planner.RefKindBranch,
SourceRef: plumbing.NewBranchReferenceName("main"),
TargetRef: plumbing.NewBranchReferenceName("main"),
SourceHash: hashes[len(hashes)-1],
})
if err != nil {
t.Fatalf("PlanCheckpoints() error = %v", err)
}
if probes != tt.wantProbes {
t.Fatalf("FetchPack probe count = %d, want %d", probes, tt.wantProbes)
}
t.Run("non-PACK data proceeds without subdivide", func(t *testing.T) {
r := io.NopCloser(bytes.NewReader([]byte("not a pack file at all")))
got, err := checkPackSizeAndSubdivide(r, 100, func() bool {
t.Fatal("should not subdivide non-pack data")
return true
})
}
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got == nil {
t.Fatal("expected non-nil reader for non-pack data")
}
})
}
func TestSearchCheckpointUnderLimitFindsLargestFittingCheckpoint(t *testing.T) {
hashes := makeLinearCommitChain(t, 20)
indices := make(map[plumbing.Hash]int, len(hashes))
for i, h := range hashes {
indices[h] = i
func TestSubdivideCheckpoints(t *testing.T) {
makeHashes := func(n int) []plumbing.Hash {
hashes := make([]plumbing.Hash, n)
for i := range hashes {
hashes[i] = plumbing.NewHash(fmt.Sprintf("%040d", i))
}
return hashes
}
source := fakeBootstrapSource{
fetchCommitGraph: func(_ context.Context, store storer.Storer, _ *gitproto.Conn, _ gitproto.DesiredRef) error {
_ = writeLinearCommitChain(t, store, len(hashes))
return nil
},
fetchPack: func(_ context.Context, _ *gitproto.Conn, desired map[plumbing.ReferenceName]gitproto.DesiredRef, haves map[plumbing.ReferenceName]plumbing.Hash) (io.ReadCloser, error) {
for _, ref := range desired {
wantIdx := indices[ref.SourceHash]
haveIdx := -1
for _, h := range haves {
if h.IsZero() {
continue
}
haveIdx = indices[h]
}
span := wantIdx - haveIdx
size := span * 1000
if span > 12 {
size = 50_000
}
return io.NopCloser(bytes.NewReader(make([]byte, size))), nil
}
t.Fatal("expected single desired ref")
return nil, nil
},
}
chain := makeHashes(10) // indices 0..9
cp := &checkpointPlanner{
ctx: context.Background(),
params: Params{
SourceService: source,
BatchMaxPack: 12_000,
},
ref: planner.DesiredRef{
Label: "main",
Kind: planner.RefKindBranch,
SourceRef: plumbing.NewBranchReferenceName("main"),
TargetRef: plumbing.NewBranchReferenceName("main"),
SourceHash: hashes[len(hashes)-1],
},
chain: hashes,
probeCache: make(map[string]probeResult),
prefetched: make(map[plumbing.Hash][]byte),
}
t.Run("splits single checkpoint at midpoint", func(t *testing.T) {
// current=chain[0], remaining=[chain[9]] → insert chain[4] as midpoint
got := subdivideCheckpoints(chain, chain[0], []plumbing.Hash{chain[9]})
if len(got) != 2 {
t.Fatalf("len = %d, want 2: %v", len(got), got)
}
if got[0] != chain[4] {
t.Fatalf("got[0] = %s, want midpoint chain[4] = %s", got[0], chain[4])
}
if got[1] != chain[9] {
t.Fatalf("got[1] = %s, want tip chain[9] = %s", got[1], chain[9])
}
})
bestIdx, err := cp.searchCheckpointUnderLimit(-1, plumbing.ZeroHash, 12)
if err != nil {
t.Fatalf("searchCheckpointUnderLimit() error = %v", err)
}
if bestIdx != 11 {
t.Fatalf("best checkpoint index = %d, want 11 (largest fitting span of 12 commits)", bestIdx)
}
t.Run("zero current starts from beginning", func(t *testing.T) {
// current=zero, remaining=[chain[9]] → insert chain[4] as midpoint
got := subdivideCheckpoints(chain, plumbing.ZeroHash, []plumbing.Hash{chain[9]})
if len(got) != 2 {
t.Fatalf("len = %d, want 2", len(got))
}
if got[0] != chain[4] {
t.Fatalf("got[0] = %s, want chain[4]", got[0])
}
})
t.Run("adjacent commits cannot split further", func(t *testing.T) {
// current=chain[8], remaining=[chain[9]] → gap=1, no midpoint
got := subdivideCheckpoints(chain, chain[8], []plumbing.Hash{chain[9]})
if len(got) != 1 {
t.Fatalf("len = %d, want 1", len(got))
}
})
t.Run("multiple remaining checkpoints each get split", func(t *testing.T) {
// current=zero, remaining=[chain[4], chain[9]]
// first: gap(-1→4)=5, mid=chain[1]
// second: gap(4→9)=5, mid=chain[6]
got := subdivideCheckpoints(chain, plumbing.ZeroHash, []plumbing.Hash{chain[4], chain[9]})
if len(got) != 4 {
t.Fatalf("len = %d, want 4: %v", len(got), got)
}
if got[0] != chain[1] {
t.Fatalf("got[0] = %s, want chain[1]", got[0])
}
if got[1] != chain[4] {
t.Fatalf("got[1] = %s, want chain[4]", got[1])
}
if got[2] != chain[6] {
t.Fatalf("got[2] = %s, want chain[6]", got[2])
}
if got[3] != chain[9] {
t.Fatalf("got[3] = %s, want chain[9]", got[3])
}
})
}
type fakeBootstrapSource struct {
191 unmodified lines
mainRef := plumbing.NewBranchReferenceName("main")
hashes := makeLinearCommitChain(t, 1)
pack := &trackingReadCloser{Reader: bytes.NewReader([]byte("PACK"))}
packFetches := 0
_, err := Execute(context.Background(), Params{
SourceService: fakeBootstrapSource{
2 unmodified lines
return nil
},
fetchPack: func(_ context.Context, _ *gitproto.Conn, desired map[plumbing.ReferenceName]gitproto.DesiredRef, _ map[plumbing.ReferenceName]plumbing.Hash) (io.ReadCloser, error) {
packFetches++
for _, ref := range desired {
if ref.SourceHash == hashes[len(hashes)-1] {
if packFetches == 1 {
return io.NopCloser(bytes.NewReader(nil)), nil
}
return pack, nil
}
}
return io.NopCloser(bytes.NewReader(nil)), nil
return pack, nil
},
},
TargetPusher: fakeBootstrapPusher{
11 unmodified lines
},
},
TargetRefs: map[plumbing.ReferenceName]plumbing.Hash{},
BatchMaxPack: 10,
TargetMaxPack: 10,
}, "empty target")
if err == nil || !strings.Contains(err.Error(), "push bootstrap batch") {
t.Fatalf("unexpected error: %v", err)
}
if packFetches < 2 {
t.Fatalf("expected separate probe and execution fetches, got %d", packFetches)
}
if !pack.closed {
t.Fatal("expected strategy to close checkpoint pack on push error")
}
3 unmodified lines
mainRef := plumbing.NewBranchReferenceName("main")
hashes := makeLinearCommitChain(t, 1)
pack := &interruptedReadCloser{first: []byte("PACK"), err: io.ErrUnexpectedEOF}
packFetches := 0
_, err := Execute(context.Background(), Params{
SourceService: fakeBootstrapSource{
2 unmodified lines
return nil
},
fetchPack: func(_ context.Context, _ *gitproto.Conn, desired map[plumbing.ReferenceName]gitproto.DesiredRef, _ map[plumbing.ReferenceName]plumbing.Hash) (io.ReadCloser, error) {
packFetches++
for _, ref := range desired {
if ref.SourceHash == hashes[len(hashes)-1] {
if packFetches == 1 {
return io.NopCloser(bytes.NewReader(nil)), nil
}
return pack, nil
}
}
return io.NopCloser(bytes.NewReader(nil)), nil
return pack, nil
},
},
TargetPusher: fakeBootstrapPusher{
12 unmodified lines
},
},
TargetRefs: map[plumbing.ReferenceName]plumbing.Hash{},
BatchMaxPack: 10,
TargetMaxPack: 10,
}, "empty target")
if !errors.Is(err, io.ErrUnexpectedEOF) {
t.Fatalf("expected interrupted read error, got %v", err)
}
if packFetches < 2 {
t.Fatalf("expected separate probe and execution fetches, got %d", packFetches)
}
if !pack.closed {
t.Fatal("expected strategy to close checkpoint pack after read interruption")
}
30 unmodified lines
},
},
TargetRefs: map[plumbing.ReferenceName]plumbing.Hash{},
BatchMaxPack: tt.batchMaxPack,
TargetMaxPack: tt.batchMaxPack,
}, "missing pusher")
if err == nil || err.Error() != "bootstrap strategy requires TargetPusher" {
t.Fatalf("Execute() error = %v, want missing TargetPusher", err)
Minternal/strategy/bootstrap/bootstrap_test.go+221/-340
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
// Package replicate implements relay-only source-authoritative replication.
package replicate
import (
"context"
"fmt"
"io"
"sync"
"github.com/go-git/go-git/v6/plumbing"
"github.com/soph/git-sync/internal/convert"
"github.com/soph/git-sync/internal/gitproto"
"github.com/soph/git-sync/internal/planner"
)
// Params holds the inputs for a replication relay execution.
type Params struct {
SourceConn *gitproto.Conn
SourceService interface {
FetchPack(context.Context, *gitproto.Conn, map[plumbing.ReferenceName]gitproto.DesiredRef, map[plumbing.ReferenceName]plumbing.Hash) (io.ReadCloser, error)
}
TargetPusher interface {
PushPack(context.Context, []gitproto.PushCommand, io.ReadCloser) error
PushCommands(context.Context, []gitproto.PushCommand) error
}
DesiredRefs map[plumbing.ReferenceName]planner.DesiredRef
TargetRefs map[plumbing.ReferenceName]plumbing.Hash
PushPlans []planner.BranchPlan
MaxPackBytes int64
}
// Result holds the outcome of a replication relay.
type Result struct {
Relay bool
RelayMode string
RelayReason string
}
// Execute runs relay-only replication. Create/update refs are pushed via pack
// relay and deletes are sent afterwards as ref-only commands.
func Execute(ctx context.Context, p Params) (Result, error) {
if p.TargetPusher == nil {
return Result{}, fmt.Errorf("replicate strategy requires TargetPusher")
}
updatePlans := make([]planner.BranchPlan, 0, len(p.PushPlans))
deletePlans := make([]planner.BranchPlan, 0, len(p.PushPlans))
for _, plan := range p.PushPlans {
switch plan.Action {
case planner.ActionCreate, planner.ActionUpdate:
updatePlans = append(updatePlans, plan)
case planner.ActionDelete:
deletePlans = append(deletePlans, plan)
default:
return Result{}, fmt.Errorf("replicate strategy does not support %s actions", plan.Action)
}
}
if len(updatePlans) > 0 {
desired := convert.DesiredRefsForPlans(p.DesiredRefs, updatePlans)
packReader, err := p.SourceService.FetchPack(ctx, p.SourceConn, desired, p.TargetRefs)
if err != nil {
return Result{}, fmt.Errorf("fetch source pack: %w", err)
}
packReader = gitproto.LimitPackReader(packReader, p.MaxPackBytes)
packReader = closeOnce(packReader)
if err := p.TargetPusher.PushPack(ctx, convert.PlansToPushCommands(updatePlans), packReader); err != nil {
_ = packReader.Close()
return Result{}, fmt.Errorf("push target refs: %w", err)
}
_ = packReader.Close()
}
if len(deletePlans) > 0 {
if err := p.TargetPusher.PushCommands(ctx, convert.PlansToPushCommands(deletePlans)); err != nil {
return Result{}, fmt.Errorf("delete target refs: %w", err)
}
}
return Result{Relay: true, RelayMode: "replicate", RelayReason: "replicate-overwrite-relay"}, nil
}
type closeOnceReadCloser struct {
io.ReadCloser
once sync.Once
}
func (c *closeOnceReadCloser) Close() error {
var err error
c.once.Do(func() {
err = c.ReadCloser.Close()
})
return err
}
func closeOnce(rc io.ReadCloser) io.ReadCloser {
if rc == nil {
return nil
}
if _, ok := rc.(*closeOnceReadCloser); ok {
return rc
}
// PushPack and the caller both close the relay reader; wrap it so retries and
// error cleanup do not surface spurious double-close failures.
return &closeOnceReadCloser{ReadCloser: rc}
}
Ainternal/strategy/replicate/replicate.go+107
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
package replicate
import (
"bytes"
"context"
"io"
"testing"
"github.com/go-git/go-git/v6/plumbing"
"github.com/soph/git-sync/internal/gitproto"
"github.com/soph/git-sync/internal/planner"
)
type fakeSourceService struct {
fetchPack func(context.Context, *gitproto.Conn, map[plumbing.ReferenceName]gitproto.DesiredRef, map[plumbing.ReferenceName]plumbing.Hash) (io.ReadCloser, error)
}
func (f fakeSourceService) FetchPack(
ctx context.Context,
conn *gitproto.Conn,
desired map[plumbing.ReferenceName]gitproto.DesiredRef,
targetRefs map[plumbing.ReferenceName]plumbing.Hash,
) (io.ReadCloser, error) {
return f.fetchPack(ctx, conn, desired, targetRefs)
}
type fakeTargetPusher struct {
pushPack func(context.Context, []gitproto.PushCommand, io.ReadCloser) error
pushCommands func(context.Context, []gitproto.PushCommand) error
}
func (f fakeTargetPusher) PushPack(ctx context.Context, cmds []gitproto.PushCommand, pack io.ReadCloser) error {
return f.pushPack(ctx, cmds, pack)
}
func (f fakeTargetPusher) PushCommands(ctx context.Context, cmds []gitproto.PushCommand) error {
return f.pushCommands(ctx, cmds)
}
func TestExecuteReplicateRelaysUpdatesAndDeletesSeparately(t *testing.T) {
mainRef := plumbing.NewBranchReferenceName("main")
oldRef := plumbing.NewBranchReferenceName("old")
oldHash := plumbing.NewHash("1111111111111111111111111111111111111111")
newHash := plumbing.NewHash("2222222222222222222222222222222222222222")
var gotDesired map[plumbing.ReferenceName]gitproto.DesiredRef
var gotHaves map[plumbing.ReferenceName]plumbing.Hash
var pushed []gitproto.PushCommand
var deleted []gitproto.PushCommand
result, err := Execute(context.Background(), Params{
SourceService: fakeSourceService{
fetchPack: func(_ context.Context, _ *gitproto.Conn, desired map[plumbing.ReferenceName]gitproto.DesiredRef, targetRefs map[plumbing.ReferenceName]plumbing.Hash) (io.ReadCloser, error) {
gotDesired = desired
gotHaves = targetRefs
return io.NopCloser(bytes.NewReader([]byte("PACK"))), nil
},
},
TargetPusher: fakeTargetPusher{
pushPack: func(_ context.Context, cmds []gitproto.PushCommand, pack io.ReadCloser) error {
defer pack.Close()
pushed = append([]gitproto.PushCommand(nil), cmds...)
return nil
},
pushCommands: func(_ context.Context, cmds []gitproto.PushCommand) error {
deleted = append([]gitproto.PushCommand(nil), cmds...)
return nil
},
},
DesiredRefs: map[plumbing.ReferenceName]planner.DesiredRef{
mainRef: {
SourceRef: mainRef,
TargetRef: mainRef,
SourceHash: newHash,
Kind: planner.RefKindBranch,
},
},
TargetRefs: map[plumbing.ReferenceName]plumbing.Hash{
mainRef: oldHash,
oldRef: oldHash,
},
PushPlans: []planner.BranchPlan{
{
SourceRef: mainRef,
TargetRef: mainRef,
SourceHash: newHash,
TargetHash: oldHash,
Kind: planner.RefKindBranch,
Action: planner.ActionUpdate,
},
{
TargetRef: oldRef,
TargetHash: oldHash,
Kind: planner.RefKindBranch,
Action: planner.ActionDelete,
},
},
})
if err != nil {
t.Fatalf("Execute() error = %v", err)
}
if !result.Relay || result.RelayMode != "replicate" || result.RelayReason != "replicate-overwrite-relay" {
t.Fatalf("unexpected result: %+v", result)
}
if len(gotDesired) != 1 || gotDesired[mainRef].SourceHash != newHash {
t.Fatalf("unexpected desired refs: %+v", gotDesired)
}
if gotHaves[mainRef] != oldHash {
t.Fatalf("expected target ref haves, got %+v", gotHaves)
}
if len(pushed) != 1 || pushed[0].Name != mainRef || pushed[0].New != newHash {
t.Fatalf("unexpected pack push commands: %+v", pushed)
}
if len(deleted) != 1 || !deleted[0].Delete || deleted[0].Name != oldRef {
t.Fatalf("unexpected delete commands: %+v", deleted)
}
}
Ainternal/strategy/replicate/replicate_test.go+118
97 unmodified lines
98
99
100
101
101
102
103
104
13 unmodified lines
118
119
120
121
121
122
123
124
97 unmodified lines
targetURL := strings.TrimRight(baseURL, "/") + "/git/" + username + "/" + repoName
t.Logf(
"Entire local smoke config: source=%s branch=%s target=%s repo=%s protocol=%s max_pack_bytes=%d batch_max_pack_bytes=%d",
"Entire local smoke config: source=%s branch=%s target=%s repo=%s protocol=%s max_pack_bytes=%d target_max_pack_bytes=%d",
sourceURL,
branch,
targetURL,
13 unmodified lines
Branches: []string{branch},
Verbose: true,
MaxPackBytes: maxPackBytes,
BatchMaxPackBytes: batchMaxPackBytes,
TargetMaxPackBytes: batchMaxPackBytes,
ProtocolMode: protocolMode,
})
if err != nil {
Minternal/syncer/entire_local_smoke_test.go+2/-2
445 unmodified lines
446
447
448
449
449
450
451
452
57 unmodified lines
510
511
512
513
513
514
515
516
16 unmodified lines
533
534
535
536
536
537
538
539
94 unmodified lines
634
635
636
637
637
638
639
640
62 unmodified lines
703
704
705
706
706
707
708
709
445 unmodified lines
result, err := Bootstrap(context.Background(), Config{
Source: Endpoint{URL: sourceURL},
Target: Endpoint{URL: targetURL},
BatchMaxPackBytes: 350_000,
TargetMaxPackBytes: 350_000,
})
if err != nil {
t.Fatalf("batched bootstrap failed: %v\nbackend-stderr:\n%s", err, server.Stderr())
57 unmodified lines
cfg := Config{
Source: Endpoint{URL: sourceURL},
Target: Endpoint{URL: targetURL},
BatchMaxPackBytes: 350_000,
TargetMaxPackBytes: 350_000,
ProtocolMode: protocolModeAuto,
}
16 unmodified lines
ref := desired[plumbing.NewBranchReferenceName(testBranch)]
bParams := bstrap.Params{
SourceConn: sourceConn, SourceService: sourceService,
BatchMaxPack: cfg.BatchMaxPackBytes, Verbose: cfg.Verbose,
TargetMaxPack: cfg.TargetMaxPackBytes, Verbose: cfg.Verbose,
}
checkpoints, err := bstrap.PlanCheckpoints(context.Background(), bParams, ref)
if err != nil {
94 unmodified lines
checkpoints, err := bstrap.PlanCheckpoints(context.Background(), bstrap.Params{
SourceConn: sourceConn,
SourceService: sourceService,
BatchMaxPack: limit,
TargetMaxPack: limit,
Verbose: cfg.Verbose,
}, ref)
if err != nil {
62 unmodified lines
Source: Endpoint{URL: sourceURL},
Target: Endpoint{URL: targetURL},
IncludeTags: true,
BatchMaxPackBytes: 350_000,
TargetMaxPackBytes: 350_000,
})
if err != nil {
t.Fatalf("batched bootstrap with tags failed: %v\nbackend-stderr:\n%s", err, server.Stderr())
Minternal/syncer/git_http_backend_test.go+5/-5
30 unmodified lines
31
32
33
34
34
35
36
10 unmodified lines
47
48
49
50
51
52
53
32 unmodified lines
86
87
88
89
89
90
91
92
2 unmodified lines
95
96
97
98
98
99
100
101
554 unmodified lines
656
657
658
659
659
660
661
662
663
664
661
665
666
667
668
9 unmodified lines
678
679
680
677
681
679
680
682
683
684
685
686
687
688
689
690
691
688
689
690
691
692
693
694
695
692
693
694
695
696
697
698
697
699
700
699
700
701
701
702
703
703
704
705
706
707
708
709
710
704
705
706
712
713
714
715
716
717
718
719
707
721
722
723
724
725
726
708
709
710
711
712
713
714
715
26 unmodified lines
742
743
744
759
745
746
747
748
46 unmodified lines
795
796
797
812
798
799
800
801
62 unmodified lines
864
865
866
881
867
868
869
870
24 unmodified lines
895
896
897
912
898
899
900
901
9 unmodified lines
911
912
913
928
914
915
916
917
64 unmodified lines
982
983
984
999
985
986
987
988
17 unmodified lines
1006
1007
1008
1023
1009
1010
1011
1012
134 unmodified lines
1147
1148
1149
1150
1151
1152
1153
410 unmodified lines
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
158 unmodified lines
2097
2098
2099
2100
2101
2102
2103
146 unmodified lines
2250
2251
2252
1896
1897
1898
1899
1900
1901
1902
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
2266
2267
2268
2269
4 unmodified lines
2274
2275
2276
2277
2278
2279
2280
2281
2282
2283
2284
2285
2286
2287
2288
2289
2290
2291
2292
2293
2294
2295
2296
2297
2298
2299
2300
2301
2302
2303
2304
2305
2306
2307
2308
2309
2310
30 unmodified lines
"github.com/soph/git-sync/internal/auth"
"github.com/soph/git-sync/internal/gitproto"
"github.com/soph/git-sync/internal/planner"
bstrap "github.com/soph/git-sync/internal/strategy/bootstrap"
"github.com/soph/git-sync/internal/syncertest"
)
10 unmodified lines
sourceServer := newSmartHTTPRepoServer(t, sourceRepo)
targetServer := newSmartHTTPRepoServer(t, targetRepo)
targetServer.receivePackThinCap = true
defer sourceServer.Close()
defer targetServer.Close()
32 unmodified lines
func TestRun_IntegrationInitialSyncAutoFallsBackToBatchedBootstrapOnTargetBodyLimit(t *testing.T) {
sourceRepo, sourceFS := newSourceRepo(t)
makeLargeCommits(t, sourceRepo, sourceFS, 20, 200_000)
makeLargeCommits(t, sourceRepo, sourceFS, 100, 5_000)
targetRepo, err := git.Init(memory.NewStorage())
if err != nil {
2 unmodified lines
sourceServer := newSmartHTTPRepoServerV2(t, sourceRepo)
targetServer := newSmartHTTPRepoServer(t, targetRepo)
targetServer.receivePackBodyLimit = 1_000_000
targetServer.receivePackBodyLimit = 300_000
defer sourceServer.Close()
defer targetServer.Close()
554 unmodified lines
}
}
func TestBootstrap_IntegrationBatchedResumeMismatchFails(t *testing.T) {
func TestBootstrap_IntegrationBatchedResumeMismatchClearsAndRetries(t *testing.T) {
// When a temp ref from a previous run doesn't match any planned checkpoint
// (e.g., the user changed --target-max-pack-bytes between runs), bootstrap
// should delete the stale temp ref and start the branch fresh rather than
// failing with a resume mismatch error.
sourceRepo, sourceFS := newSourceRepo(t)
makeLargeCommits(t, sourceRepo, sourceFS, 5, 200_000)
makeLargeCommits(t, sourceRepo, sourceFS, 100, 5_000)
unrelatedRepo, unrelatedFS := newSourceRepo(t)
makeCommits(t, unrelatedRepo, unrelatedFS, 1)
9 unmodified lines
if err := copyRefsAndObjects(unrelatedRepo.Storer, targetRepo.Storer, nil); err != nil {
t.Fatalf("copy unrelated objects: %v", err)
}
targetHead := unrelatedHead
tempRef := planner.BootstrapTempRef(plumbing.NewBranchReferenceName(testBranch))
if err := targetRepo.Storer.SetReference(plumbing.NewHashReference(tempRef, targetHead.Hash())); err != nil {
t.Fatalf("set temp ref: %v", err)
if err := targetRepo.Storer.SetReference(plumbing.NewHashReference(tempRef, unrelatedHead.Hash())); err != nil {
t.Fatalf("set stale temp ref: %v", err)
}
sourceServer := newSmartHTTPRepoServerV2(t, sourceRepo)
targetServer := newSmartHTTPRepoServer(t, targetRepo)
targetServer.receivePackThinCap = true
defer sourceServer.Close()
defer targetServer.Close()
cfg := Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
ProtocolMode: protocolModeAuto,
BatchMaxPackBytes: 350_000,
}
s, err := newSession(context.Background(), cfg, false)
result, err := Bootstrap(context.Background(), Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
ProtocolMode: protocolModeAuto,
TargetMaxPackBytes: 350_000,
})
if err != nil {
t.Fatalf("new session: %v", err)
t.Fatalf("expected bootstrap to clear stale temp ref and succeed, got: %v", err)
}
desired, _, err := planner.BuildDesiredRefs(s.sourceRefMap, planConfig(cfg))
if err != nil {
t.Fatalf("build desired refs: %v", err)
if !result.Batching {
t.Fatalf("expected batched bootstrap, got %+v", result)
}
ref := desired[plumbing.NewBranchReferenceName(testBranch)]
checkpoints, err := bstrap.PlanCheckpoints(context.Background(), bstrap.Params{
SourceConn: s.sourceConn,
SourceService: s.sourceService,
BatchMaxPack: cfg.BatchMaxPackBytes,
}, ref)
if err != nil {
t.Fatalf("plan checkpoints: %v", err)
if result.Pushed == 0 {
t.Fatalf("expected pushed > 0, got %+v", result)
}
if len(checkpoints) == 0 {
t.Fatal("expected checkpoints for batched bootstrap")
}
for _, checkpoint := range checkpoints {
if checkpoint == targetHead.Hash() {
t.Fatalf("expected unrelated temp ref hash, got checkpoint collision at %s", checkpoint)
}
}
_, err = Bootstrap(context.Background(), cfg)
if err == nil {
t.Fatal("expected batched bootstrap resume mismatch to fail")
}
if !strings.Contains(err.Error(), "does not match any planned checkpoint") {
t.Fatalf("unexpected error: %v", err)
assertHeadsMatch(t, sourceRepo, targetRepo, testBranch)
// Stale temp ref should have been cleaned up.
if _, err := targetRepo.Reference(tempRef, true); err != plumbing.ErrReferenceNotFound {
t.Fatalf("expected stale temp ref to be deleted, got err=%v", err)
}
}
26 unmodified lines
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
ProtocolMode: protocolModeAuto,
BatchMaxPackBytes: 350_000,
TargetMaxPackBytes: 350_000,
})
if err != nil {
t.Fatalf("batched bootstrap final-tip cutover failed: %v", err)
46 unmodified lines
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
ProtocolMode: protocolModeAuto,
BatchMaxPackBytes: 350_000,
TargetMaxPackBytes: 350_000,
})
if err != nil {
t.Fatalf("batched bootstrap cleanup rerun failed: %v", err)
62 unmodified lines
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
ProtocolMode: protocolModeAuto,
BatchMaxPackBytes: 350_000,
TargetMaxPackBytes: 350_000,
}
if _, err := Bootstrap(context.Background(), cfg); err == nil {
24 unmodified lines
func TestBootstrap_IntegrationBatchedPackFailureResumesOnRetry(t *testing.T) {
sourceRepo, sourceFS := newSourceRepo(t)
makeLargeCommits(t, sourceRepo, sourceFS, 5, 200_000)
makeLargeCommits(t, sourceRepo, sourceFS, 80, 5_000)
targetRepo, err := git.Init(memory.NewStorage())
if err != nil {
9 unmodified lines
targetRef := plumbing.NewBranchReferenceName(testBranch)
64 unmodified lines
func TestBootstrap_IntegrationBatchedLightweightTagCreatesWithoutExtraPack(t *testing.T) {
sourceRepo, sourceFS := newSourceRepo(t)
makeLargeCommits(t, sourceRepo, sourceFS, 5, 200_000)
makeLargeCommits(t, sourceRepo, sourceFS, 80, 5_000)
sourceHead, err := sourceRepo.Reference(plumbing.NewBranchReferenceName(testBranch), true)
if err != nil {
t.Fatalf("resolve source head: %v", err)
17 unmodified lines
Target: Endpoint{URL: targetServer.RepoURL()},
ProtocolMode: protocolModeAuto,
IncludeTags: true,
BatchMaxPackBytes: 350_000,
TargetMaxPackBytes: 350_000,
})
if err != nil {
t.Fatalf("batched bootstrap with lightweight tag failed: %v", err)
134 unmodified lines
410 unmodified lines
assertHeadsMatch(t, sourceRepo, targetRepo, testBranch)
}
func TestRun_IntegrationReplicateAgainstNoThinTarget(t *testing.T) {
// Replicate must tolerate targets that advertise no-thin. Source upload-pack
// never receives a thin-pack request from us (see gitproto/fetch.go), so
// the relayed pack is self-contained and acceptable to a no-thin
// receive-pack. This is the main reason the capability was reconsidered:
// go-git's own receive-pack advertises no-thin, and so does any server
// built on it (e.g. entire-server).
sourceRepo, sourceFS := newSourceRepo(t)
makeCommits(t, sourceRepo, sourceFS, 2)
targetRepo, err := git.Init(memory.NewStorage())
if err != nil {
t.Fatalf("init target repo: %v", err)
}
sourceServer := newSmartHTTPRepoServer(t, sourceRepo)
targetServer := newSmartHTTPRepoServer(t, targetRepo)
targetServer.receivePackNoThin = true
defer sourceServer.Close()
defer targetServer.Close()
result, err := Run(context.Background(), Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
Mode: modeReplicate,
})
if err != nil {
t.Fatalf("replicate against no-thin target failed: %v", err)
}
if result.OperationMode != modeReplicate {
t.Fatalf("expected operation_mode=replicate, got %q", result.OperationMode)
}
if !result.Relay {
t.Fatalf("expected relay execution against no-thin target, got %+v", result)
}
assertHeadsMatch(t, sourceRepo, targetRepo, testBranch)
}
func TestReplicateCanBootstrapRejectsPruneDeletes(t *testing.T) {
s := &syncSession{
cfg: Config{
Prune: true,
},
target: &targetSession{
refMap: map[plumbing.ReferenceName]plumbing.Hash{
plumbing.NewBranchReferenceName("stale"): plumbing.NewHash("1111111111111111111111111111111111111111"),
},
},
}
desired := map[plumbing.ReferenceName]planner.DesiredRef{
plumbing.NewBranchReferenceName("main"): {
SourceRef: plumbing.NewBranchReferenceName("main"),
TargetRef: plumbing.NewBranchReferenceName("main"),
SourceHash: plumbing.NewHash("2222222222222222222222222222222222222222"),
Kind: planner.RefKindBranch,
},
}
if s.replicateCanBootstrap(desired) {
t.Fatal("expected bootstrap shortcut to be disabled when prune would delete managed refs")
}
}
func TestRun_IntegrationReplicateOverwritesDivergentBranch(t *testing.T) {
sourceRepo, sourceFS := newSourceRepo(t)
targetRepo, _ := newSourceRepo(t)
makeCommits(t, sourceRepo, sourceFS, 3)
// Seed: both sides at the latest commit.
if _, err := Run(context.Background(), Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
}); err != nil {
t.Fatalf("seed sync failed: %v", err)
}
head, err := sourceRepo.Reference(plumbing.NewBranchReferenceName(testBranch), true)
if err != nil {
t.Fatalf("source head: %v", err)
}
headCommit, err := sourceRepo.CommitObject(head.Hash())
if err != nil {
t.Fatalf("load head commit: %v", err)
}
if len(headCommit.ParentHashes) == 0 {
t.Fatalf("expected head to have a parent")
}
olderHash := headCommit.ParentHashes[0]
// Rewind source's branch to an earlier commit. All objects still exist on
// both sides, but the refs now disagree in a non-fast-forward way from the
// target's perspective.
branchRef := plumbing.NewBranchReferenceName(testBranch)
if err := sourceRepo.Storer.SetReference(plumbing.NewHashReference(branchRef, olderHash)); err != nil {
t.Fatalf("rewind source branch: %v", err)
}
// Baseline: plain sync refuses the non-fast-forward rewind without force.
if _, err := Run(context.Background(), Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
}); err == nil {
t.Fatal("expected plain sync to refuse non-fast-forward overwrite without force")
}
result, err := Run(context.Background(), Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
Mode: modeReplicate,
})
if err != nil {
t.Fatalf("replicate failed: %v", err)
}
if result.OperationMode != modeReplicate {
t.Fatalf("expected operation_mode=replicate, got %q", result.OperationMode)
}
if result.Pushed != 1 || result.Blocked != 0 || result.Deleted != 0 {
t.Fatalf("unexpected result counts: %+v", result)
}
if !result.Relay || result.RelayMode != "replicate" || result.RelayReason != "replicate-overwrite-relay" {
t.Fatalf("expected replicate relay execution, got %+v", result)
}
got, err := targetRepo.Reference(branchRef, true)
if err != nil {
t.Fatalf("read target branch: %v", err)
}
if got.Hash() != olderHash {
t.Fatalf("expected target branch retargeted to %s, got %s", olderHash, got.Hash())
}
}
func TestRun_IntegrationReplicateBootstrapBatchesWhenConfigured(t *testing.T) {
// Large bootstrap-relay pushes can overwhelm targets with body-size
// limits (e.g. go-git-based receive-pack on raft-backed storage).
// Replicate falls back through the bootstrap strategy for empty targets,
// and must honor TargetMaxPackBytes so callers can split a huge push
// into tractable receive-pack POSTs. Without this plumbing the replicate
// CLI flag would be silently ignored.
sourceRepo, sourceFS := newSourceRepo(t)
makeLargeCommits(t, sourceRepo, sourceFS, 100, 5_000)
targetRepo, err := git.Init(memory.NewStorage())
if err != nil {
t.Fatalf("init target repo: %v", err)
}
result, err := Run(context.Background(), Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
Mode: modeReplicate,
ProtocolMode: protocolModeAuto,
TargetMaxPackBytes: 350_000, // force > 1 batch for the generated pack
})
if err != nil {
t.Fatalf("replicate with batched bootstrap failed: %v", err)
}
if result.OperationMode != modeReplicate {
t.Fatalf("expected operation_mode=replicate, got %q", result.OperationMode)
}
if !result.Batching || result.BatchCount < 2 {
t.Fatalf("expected batched bootstrap inside replicate, got batching=%t batch_count=%d result=%+v",
result.Batching, result.BatchCount, result)
}
assertHeadsMatch(t, sourceRepo, targetRepo, testBranch)
}
func TestRun_IntegrationReplicateBootstrapsEmptyTarget(t *testing.T) {
sourceRepo, sourceFS := newSourceRepo(t)
makeCommits(t, sourceRepo, sourceFS, 2)
targetRepo, err := git.Init(memory.NewStorage())
if err != nil {
t.Fatalf("init target repo: %v", err)
}
result, err := Run(context.Background(), Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
Mode: modeReplicate,
})
if err != nil {
t.Fatalf("replicate against empty target failed: %v", err)
}
if result.OperationMode != modeReplicate {
t.Fatalf("expected operation_mode=replicate carried through bootstrap, got %q", result.OperationMode)
}
if !result.Relay {
t.Fatalf("expected bootstrap relay to run, got %+v", result)
}
if result.RelayReason != "empty-target-managed-refs" {
t.Fatalf("expected empty-target bootstrap reason, got %+v", result)
}
if result.Pushed != 1 {
t.Fatalf("expected one pushed ref, got %+v", result)
}
assertHeadsMatch(t, sourceRepo, targetRepo, testBranch)
}
func TestRun_IntegrationReplicateOverwritesDivergentTag(t *testing.T) {
sourceRepo, sourceFS := newSourceRepo(t)
targetRepo, _ := newSourceRepo(t)
makeCommits(t, sourceRepo, sourceFS, 2)
if _, err := Run(context.Background(), Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
}); err != nil {
t.Fatalf("seed sync failed: %v", err)
}
sourceHead, err := sourceRepo.Reference(plumbing.NewBranchReferenceName(testBranch), true)
if err != nil {
t.Fatalf("source head: %v", err)
}
targetHead, err := targetRepo.Reference(plumbing.NewBranchReferenceName(testBranch), true)
if err != nil {
t.Fatalf("target head: %v", err)
}
// Parent of source head — exists on both sides after the seed sync.
sourceCommit, err := sourceRepo.CommitObject(sourceHead.Hash())
if err != nil {
t.Fatalf("load source head commit: %v", err)
}
if len(sourceCommit.ParentHashes) == 0 {
t.Fatalf("expected source head to have a parent")
}
olderHash := sourceCommit.ParentHashes[0]
tagRef := plumbing.NewTagReferenceName("v1")
// Source tag points to HEAD, target tag points to HEAD's parent: divergent.
if err := sourceRepo.Storer.SetReference(plumbing.NewHashReference(tagRef, sourceHead.Hash())); err != nil {
t.Fatalf("set source tag: %v", err)
}
if err := targetRepo.Storer.SetReference(plumbing.NewHashReference(tagRef, olderHash)); err != nil {
t.Fatalf("set target tag: %v", err)
}
if targetHead.Hash() != sourceHead.Hash() {
t.Fatalf("expected seed sync to equalize branch heads")
}
result, err := Run(context.Background(), Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
Mode: modeReplicate,
IncludeTags: true,
})
if err != nil {
t.Fatalf("replicate tag overwrite failed: %v", err)
}
if result.OperationMode != modeReplicate {
t.Fatalf("expected operation_mode=replicate, got %q", result.OperationMode)
}
if result.Pushed != 1 {
t.Fatalf("expected exactly one tag overwrite push, got %+v", result)
}
if !result.Relay || result.RelayMode != "replicate" {
t.Fatalf("expected replicate relay execution, got %+v", result)
}
got, err := targetRepo.Reference(tagRef, true)
if err != nil {
t.Fatalf("read target tag: %v", err)
}
if got.Hash() != sourceHead.Hash() {
t.Fatalf("expected target tag to be retargeted to %s, got %s", sourceHead.Hash(), got.Hash())
}
}
func TestRun_IntegrationReplicatePruneDeletesOrphanedManagedRef(t *testing.T) {
sourceRepo, sourceFS := newSourceRepo(t)
targetRepo, _ := newSourceRepo(t)
makeCommits(t, sourceRepo, sourceFS, 2)
targetHead, err := targetRepo.Reference(plumbing.NewBranchReferenceName(testBranch), true)
if err != nil {
t.Fatalf("target head: %v", err)
}
orphanRef := plumbing.NewBranchReferenceName("stale")
if err := targetRepo.Storer.SetReference(plumbing.NewHashReference(orphanRef, targetHead.Hash())); err != nil {
t.Fatalf("set orphan branch: %v", err)
}
// Advance source so we have at least one real overwrite plan alongside the delete,
// keeping the replicate path out of the empty-target bootstrap shortcut.
makeCommits(t, sourceRepo, sourceFS, 1)
result, err := Run(context.Background(), Config{
Source: Endpoint{URL: sourceServer.RepoURL()},
Target: Endpoint{URL: targetServer.RepoURL()},
Mode: modeReplicate,
Prune: true,
})
if err != nil {
t.Fatalf("replicate --prune failed: %v", err)
}
if result.OperationMode != modeReplicate {
t.Fatalf("expected operation_mode=replicate, got %q", result.OperationMode)
}
if result.Deleted != 1 {
t.Fatalf("expected exactly one deleted ref, got %+v", result)
}
if result.Pushed != 1 {
t.Fatalf("expected exactly one pushed ref alongside the delete, got %+v", result)
}
if _, err := targetRepo.Reference(orphanRef, true); err != plumbing.ErrReferenceNotFound {
t.Fatalf("expected orphan branch to be pruned, got err=%v", err)
}
assertHeadsMatch(t, sourceRepo, targetRepo, testBranch)
}
func TestRun_ReplicateRejectsForceAtSessionConstruction(t *testing.T) {
// The Force-with-replicate check happens before any network I/O, so the URLs
// never get dialed. Using obviously-invalid URLs also asserts that fact.
_, err := Run(context.Background(), Config{
Source: Endpoint{URL: "http://127.0.0.1:1/source.git"},
Target: Endpoint{URL: "http://127.0.0.1:1/target.git"},
Mode: modeReplicate,
Force: true,
})
if err == nil {
t.Fatal("expected replicate+force to be rejected")
}
if !strings.Contains(err.Error(), "replicate does not support --force") ||
!strings.Contains(err.Error(), "use sync instead") {
t.Fatalf("unexpected error: %v", err)
}
}
func TestRun_IntegrationAddAnnotatedTagAfterInitialBranchSync(t *testing.T) {
sourceRepo, sourceFS := newSourceRepo(t)
makeCommits(t, sourceRepo, sourceFS, 2)
158 unmodified lines
receivePackBodyLimit int64
receivePackNoThin bool
receivePackThinCap bool
commandHook func(*packp.UpdateRequests) *packp.ReportStatus
receivePackHook func(*packp.UpdateRequests, bool) *packp.ReportStatus
uploadPackRaw func(http.ResponseWriter, *http.Request, []byte) bool
146 unmodified lines
return
}
if service == serviceReceivePack && s.receivePackNoThin {
// For no-thin, we need to decode, modify capabilities, and re-encode
ar := packp.NewAdvRefs()
if err := ar.Decode(bytes.NewReader(buf.Bytes())); err == nil {
_ = ar.Capabilities.Set(capability.Capability("no-thin"))
buf.Reset()
_ = ar.Encode(&buf)
if service == serviceReceivePack && (s.receivePackNoThin || s.receivePackThinCap) {
rewritten, err := rewriteReceivePackAdvertisement(buf.Bytes(), func(caps *capability.List) {
if s.receivePackThinCap {
caps.Delete(capability.Capability("no-thin"))
}
if s.receivePackNoThin {
_ = caps.Set(capability.Capability("no-thin"))
}
})
if err != nil {
s.tb.Fatalf("rewrite receive-pack advertisement: %v", err)
}
buf.Reset()
_, _ = buf.Write(rewritten)
}
w.Header().Set("Content-Type", fmt.Sprintf("application/x-%s-advertisement", service))
4 unmodified lines
s.recordMetric(service, metricInfoRefs, 0, int64(buf.Len()), 0, 0)
}
func (s *smartHTTPRepoServer) handleInfoRefsV2(w http.ResponseWriter, r *http.Request) {
var buf bytes.Buffer
lines := []string{
Minternal/syncer/integration_test.go+450/-56
41 unmodified lines
42
43
44
45
45
46
47
48
41 unmodified lines
Source: Endpoint{URL: "https://github.com/torvalds/linux.git"},
Target: Endpoint{URL: server.RepoURL("target.git")},
Branches: []string{"master"},
BatchMaxPackBytes: batchMaxPackBytes,
TargetMaxPackBytes: batchMaxPackBytes,
ProtocolMode: protocolModeAuto,
ShowStats: true,
MeasureMemory: true,
Minternal/syncer/live_bootstrap_test.go+1/-1
27 unmodified lines
28
29
30
31
32
33
34
1 unmodified line
36
37
38
39
40
41
42
43
22 unmodified lines
66
67
68
69
70
71
72
69
73
74
75
76
36 unmodified lines
113
114
115
116
117
118
119
13 unmodified lines
133
134
135
131
132
136
137
138
139
140
211 unmodified lines
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
16 unmodified lines
387
388
389
390
391
392
393
36 unmodified lines
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
423
446
447
448
449
2 unmodified lines
452
453
454
432
433
455
456
457
458
459
460
461
462
440
463
464
465
466
12 unmodified lines
479
480
481
459
482
483
484
485
486
487
465
488
489
490
491
1 unmodified line
493
494
495
473
496
497
498
499
5 unmodified lines
505
506
507
485
508
509
510
488
511
512
490
513
514
515
516
24 unmodified lines
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
88 unmodified lines
756
757
758
615
759
760
761
762
763
764
765
622
766
767
768
769
30 unmodified lines
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
27 unmodified lines
bstrap "github.com/soph/git-sync/internal/strategy/bootstrap"
"github.com/soph/git-sync/internal/strategy/incremental"
"github.com/soph/git-sync/internal/strategy/materialized"
repstrat "github.com/soph/git-sync/internal/strategy/replicate"
"github.com/soph/git-sync/internal/validation"
)
1 unmodified line
protocolModeAuto = validation.ProtocolAuto
protocolModeV1 = validation.ProtocolV1
protocolModeV2 = validation.ProtocolV2
modeSync = "sync"
modeReplicate = "replicate"
)
const DefaultMaterializedMaxObjects = materialized.DefaultMaxMaterializedObjects
22 unmodified lines
Verbose bool
ShowStats bool
MeasureMemory bool
Mode string
Force bool
Prune bool
MaxPackBytes int64
BatchMaxPackBytes int64
TargetMaxPackBytes int64
MaterializedMaxObjects int
ProtocolMode string
}
36 unmodified lines
Blocked int `json:"blocked"`
Deleted int `json:"deleted"`
DryRun bool `json:"dry_run"`
OperationMode string `json:"operation_mode"`
Relay bool `json:"relay"`
RelayMode string `json:"relay_mode"`
RelayReason string `json:"relay_reason"`
13 unmodified lines
lines = append(lines, planner.FormatPlanLine(plan))
}
summary := fmt.Sprintf(
"summary: pushed=%d deleted=%d skipped=%d blocked=%d protocol=%s relay=%t relay-mode=%s relay-reason=%s batching=%t batch-count=%d planned-batches=%d",
r.Pushed, r.Deleted, r.Skipped, r.Blocked, r.Protocol, r.Relay, r.RelayMode, r.RelayReason, r.Batching, r.BatchCount, r.PlannedBatchCount,
"summary: pushed=%d deleted=%d skipped=%d blocked=%d mode=%s protocol=%s relay=%t relay-mode=%s relay-reason=%s batching=%t batch-count=%d planned-batches=%d",
r.Pushed, r.Deleted, r.Skipped, r.Blocked, r.OperationMode, r.Protocol, r.Relay, r.RelayMode, r.RelayReason, r.Batching, r.BatchCount, r.PlannedBatchCount,
)
if r.DryRun {
summary += " dry-run=true"
211 unmodified lines
return nil, err
}
cfg.ProtocolMode = mode
switch cfg.Mode {
case "", modeSync:
cfg.Mode = modeSync
case modeReplicate:
default:
return nil, fmt.Errorf("unsupported operation mode %q", cfg.Mode)
}
if _, err := validation.ValidateMappings(cfg.Mappings); err != nil {
return nil, err
}
if cfg.Mode == modeReplicate && cfg.Force {
return nil, fmt.Errorf("replicate does not support --force; use sync instead")
}
s := &syncSession{
cfg: cfg,
16 unmodified lines
if err != nil {
return nil, fmt.Errorf("list source refs: %w", err)
}
sourceService.Verbose = cfg.Verbose
s.sourceService = sourceService
s.sourceRefMap = gitproto.RefHashMap(sourceRefs)
36 unmodified lines
if err != nil {
return Result{}, err
}
if s.cfg.Mode == modeReplicate {
return s.runReplicate(ctx)
}
return s.runSync(ctx)
}
func (s *syncSession) runSync(ctx context.Context) (Result, error) {
measurementDone := s.measurementDone
stats := s.stats
sourceService := s.sourceService
sourceRefMap := s.sourceRefMap
targetRefMap := s.target.refMap
desiredRefs, managedTargets, err := planner.BuildDesiredRefs(sourceRefMap, planConfig(cfg))
desiredRefs, managedTargets, err := planner.BuildDesiredRefs(sourceRefMap, planConfig(s.cfg))
if err != nil {
return Result{}, err
}
2 unmodified lines
}
// Check for bootstrap opportunity (before allocating in-memory repo)
if ok, reason := planner.CanBootstrapRelay(cfg.Force, cfg.Prune, desiredRefs, targetRefMap); ok {
if cfg.DryRun {
if ok, reason := planner.CanBootstrapRelay(s.cfg.Force, s.cfg.Prune, desiredRefs, targetRefMap); ok {
if s.cfg.DryRun {
plans, err := planner.BuildBootstrapPlans(desiredRefs, targetRefMap)
if err != nil {
return Result{}, err
}
return Result{
Plans: plans, DryRun: true, RelayReason: reason,
BootstrapSuggested: true, Stats: stats.snapshot(),
OperationMode: modeSync, BootstrapSuggested: true, Stats: stats.snapshot(),
Measurement: measurementDone(), Protocol: sourceService.Protocol,
}, nil
}
12 unmodified lines
}
}
plans, err := planner.BuildPlans(repo.Storer, desiredRefs, targetRefMap, managedTargets, planConfig(cfg))
plans, err := planner.BuildPlans(repo.Storer, desiredRefs, targetRefMap, managedTargets, planConfig(s.cfg))
if err != nil {
return Result{}, err
}
result := Result{
Plans: plans, DryRun: cfg.DryRun, Protocol: sourceService.Protocol,
Plans: plans, DryRun: s.cfg.DryRun, OperationMode: modeSync, Protocol: sourceService.Protocol,
Stats: stats.snapshot(), Measurement: measurementDone(),
}
1 unmodified line
for _, plan := range plans {
switch plan.Action {
case ActionCreate, ActionUpdate, ActionDelete:
if cfg.DryRun {
if s.cfg.DryRun {
result.Skipped++
continue
}
5 unmodified lines
}
}
if !cfg.DryRun && result.Blocked > 0 {
if !s.cfg.DryRun && result.Blocked > 0 {
return result, fmt.Errorf("blocked %d ref update(s); rerun with --force where appropriate", result.Blocked)
}
result.RelayReason = planner.RelayFallbackReason(cfg.Force, cfg.Prune, cfg.DryRun, pushPlans, s.target.policy)
result.RelayReason = planner.RelayFallbackReason(s.cfg.Force, s.cfg.Prune, s.cfg.DryRun, pushPlans, s.target.policy)
if !cfg.DryRun {
if !s.cfg.DryRun {
// Try incremental relay first
incResult, err := s.executeIncremental(ctx, desiredRefs, pushPlans)
if err != nil {
24 unmodified lines
return result, nil
}
func (s *syncSession) runReplicate(ctx context.Context) (Result, error) {
desiredRefs, managedTargets, err := planner.BuildDesiredRefs(s.sourceRefMap, planConfig(s.cfg))
if err != nil {
return Result{}, err
}
if len(desiredRefs) == 0 {
return Result{}, fmt.Errorf("no source refs matched")
}
if ok, reason := planner.SupportsReplicateRelay(s.target.policy); !ok {
return Result{OperationMode: modeReplicate}, fmt.Errorf("replicate requires relay-capable target: %s; use sync instead", reason)
}
allAbsent := s.replicateCanBootstrap(desiredRefs)
if allAbsent {
if s.cfg.DryRun {
plans, err := planner.BuildBootstrapPlans(desiredRefs, s.target.refMap)
if err != nil {
return Result{}, err
}
return Result{
Plans: plans,
DryRun: true,
OperationMode: modeReplicate,
RelayReason: "empty-target-managed-refs",
BootstrapSuggested: true,
Stats: s.stats.snapshot(),
Measurement: s.measurementDone(),
Protocol: s.sourceService.Protocol,
}, nil
}
return bootstrapWithInputs(ctx, s, desiredRefs, s.target.refMap, "empty-target-managed-refs")
}
plans, err := planner.BuildReplicationPlans(desiredRefs, s.target.refMap, managedTargets, planConfig(s.cfg))
if err != nil {
return Result{}, err
}
result := Result{
Plans: plans,
DryRun: s.cfg.DryRun,
OperationMode: modeReplicate,
Protocol: s.sourceService.Protocol,
Stats: s.stats.snapshot(),
Measurement: s.measurementDone(),
}
pushPlans := make([]BranchPlan, 0, len(plans))
relayPlans := make([]BranchPlan, 0, len(plans))
for _, plan := range plans {
switch plan.Action {
case ActionCreate, ActionUpdate, ActionDelete:
if s.cfg.DryRun {
result.Skipped++
continue
}
pushPlans = append(pushPlans, plan)
if plan.Action != ActionDelete {
relayPlans = append(relayPlans, plan)
}
case ActionSkip:
result.Skipped++
case ActionBlock:
result.Blocked++
}
}
if !s.cfg.DryRun && len(relayPlans) > 0 {
ok, reason := planner.CanReplicateRelay(relayPlans)
if !ok {
return result, fmt.Errorf("replicate requires relay-capable target: %s; use sync instead", reason)
}
repResult, err := s.executeReplicate(ctx, desiredRefs, pushPlans)
if err != nil {
return result, fmt.Errorf("replicate relay failed: %w; use sync instead", err)
}
result.Relay = repResult.Relay
result.RelayMode = repResult.RelayMode
result.RelayReason = repResult.RelayReason
}
for _, plan := range pushPlans {
switch plan.Action {
case ActionCreate, ActionUpdate:
result.Pushed++
case ActionDelete:
result.Deleted++
}
}
result.Stats = s.stats.snapshot()
result.Measurement = s.measurementDone()
return result, nil
}
func (s *syncSession) replicateCanBootstrap(desiredRefs map[plumbing.ReferenceName]planner.DesiredRef) bool {
for targetRef := range desiredRefs {
if !s.target.refMap[targetRef].IsZero() {
return false
}
}
if !s.cfg.Prune {
return true
}
for targetRef, hash := range s.target.refMap {
if hash.IsZero() {
continue
}
if _, ok := desiredRefs[targetRef]; ok {
continue
}
switch {
case targetRef.IsTag() && s.cfg.IncludeTags:
return false
case targetRef.IsBranch() && len(s.cfg.Mappings) == 0 && len(s.cfg.Branches) == 0:
return false
}
}
return true
}
// Bootstrap seeds an empty target with relay behavior.
func Bootstrap(ctx context.Context, cfg Config) (Result, error) {
if cfg.Force {
88 unmodified lines
bResult, err := bstrap.Execute(ctx, bstrap.Params{
SourceConn: s.sourceConn, SourceService: s.sourceService, TargetPusher: s.target.pusher,
DesiredRefs: desiredRefs, TargetRefs: targetRefs,
MaxPackBytes: s.cfg.MaxPackBytes, BatchMaxPack: s.cfg.BatchMaxPackBytes,
MaxPackBytes: s.cfg.MaxPackBytes, TargetMaxPack: s.cfg.TargetMaxPackBytes,
Verbose: s.cfg.Verbose, Logger: s.logger,
}, relayReason)
if err != nil {
return Result{}, err
}
return Result{
Plans: bResult.Plans, Pushed: bResult.Pushed,
Plans: bResult.Plans, Pushed: bResult.Pushed, OperationMode: s.cfg.Mode,
Relay: bResult.Relay, RelayMode: bResult.RelayMode, RelayReason: bResult.RelayReason,
Batching: bResult.Batching, BatchCount: bResult.BatchCount,
PlannedBatchCount: bResult.PlannedBatchCount, TempRefs: bResult.TempRefs,
30 unmodified lines
})
}
func (s *syncSession) executeReplicate(
ctx context.Context,
desiredRefs map[plumbing.ReferenceName]planner.DesiredRef,
pushPlans []planner.BranchPlan,
) (repstrat.Result, error) {
return repstrat.Execute(ctx, repstrat.Params{
SourceConn: s.sourceConn, SourceService: s.sourceService, TargetPusher: s.target.pusher,
DesiredRefs: desiredRefs, TargetRefs: s.target.refMap,
PushPlans: pushPlans, MaxPackBytes: s.cfg.MaxPackBytes,
})
}
func (s *syncSession) buildDesiredRefs() (map[plumbing.ReferenceName]planner.DesiredRef, error) {
desiredRefs, _, err := planner.BuildDesiredRefs(s.sourceRefMap, planConfig(s.cfg))
if err != nil {
Minternal/syncer/syncer.go+171/-15
73 unmodified lines
74
75
76
77
78
79
80
81
82
83
84
85
57 unmodified lines
143
144
145
146
147
148
149
150
151
10 unmodified lines
162
163
164
165
166
167
168
169
170
45 unmodified lines
216
217
218
219
220
221
222
1 unmodified line
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
73 unmodified lines
return internalbridge.FromSyncResult(result), nil
}
// Replicate executes source-authoritative relay-only replication between two remotes.
func (c *Client) Replicate(ctx context.Context, req SyncRequest) (SyncResult, error) {
req.Policy.Mode = ModeReplicate
return c.Sync(ctx, req)
}
func (c *Client) buildProbeConfig(ctx context.Context, req ProbeRequest) (internalbridge.Config, error) {
sourceAuth, err := c.authFor(ctx, req.Source, SourceRole)
if err != nil {
57 unmodified lines
if r.Target.URL == "" {
return fmt.Errorf("target URL is required")
}
if err := validateOperationMode(r.Policy.Mode); err != nil {
return err
}
if _, err := validation.NormalizeProtocolMode(string(r.Policy.Protocol)); err != nil {
return err
}
10 unmodified lines
if r.Target.URL == "" {
return fmt.Errorf("target URL is required")
}
if err := validateOperationMode(r.Policy.Mode); err != nil {
return err
}
if _, err := validation.NormalizeProtocolMode(string(r.Policy.Protocol)); err != nil {
return err
}
45 unmodified lines
func bridgePolicy(policy SyncPolicy) internalbridge.SyncPolicy {
return internalbridge.SyncPolicy{
Mode: internalbridge.OperationMode(policy.Mode),
IncludeTags: policy.IncludeTags,
Force: policy.Force,
Prune: policy.Prune,
1 unmodified line
}
}
func validateOperationMode(mode OperationMode) error {
switch mode {
case "", ModeSync, ModeReplicate:
return nil
default:
return fmt.Errorf("unsupported operation mode %q", mode)
}
}
func ptr[T any](v T) *T {
return &v
}
Mpkg/gitsync/client.go+22
108 unmodified lines
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
108 unmodified lines
}
}
func TestClientReplicateRejectsUnsupportedMode(t *testing.T) {
err := (SyncRequest{
Source: Endpoint{URL: "https://source.example/repo.git"},
Target: Endpoint{URL: "https://target.example/repo.git"},
Policy: SyncPolicy{Mode: "bogus"},
}).Validate()
if err == nil {
t.Fatalf("expected invalid operation mode validation error")
}
}
type smartHTTPRepoServer struct {
tb testing.TB
repo *git.Repository
Mpkg/gitsync/client_test.go+11
8 unmodified lines
9
10
11
12
13
14
15
3 unmodified lines
19
20
21
22
23
24
25
26
27
16 unmodified lines
44
45
46
47
48
49
50
24 unmodified lines
75
76
77
78
79
80
81
26 unmodified lines
108
109
110
111
112
113
114
115
116
117
118
119
120
8 unmodified lines
)
type ProtocolMode string
type OperationMode string
type Config struct {
raw syncer.Config
3 unmodified lines
const ProtocolV1 ProtocolMode = validation.ProtocolV1
const ProtocolV2 ProtocolMode = validation.ProtocolV2
const ModeSync OperationMode = "sync"
const ModeReplicate OperationMode = "replicate"
type RefMapping struct {
Source string
Target string
16 unmodified lines
}
type SyncPolicy struct {
Mode OperationMode
IncludeTags bool
Force bool
Prune bool
24 unmodified lines
IncludeTags: policy.IncludeTags,
DryRun: dryRun,
ShowStats: collectStats,
Mode: operationModeString(policy.Mode),
Force: policy.Force,
Prune: policy.Prune,
ProtocolMode: protocolString(policy.Protocol),
26 unmodified lines
return string(mode)
}
func operationModeString(mode OperationMode) string {
if mode == "" {
return string(ModeSync)
}
return string(mode)
}
func ToValidationMappings(mappings []RefMapping) []validation.RefMapping {
out := make([]validation.RefMapping, 0, len(mappings))
for _, mapping := range mappings {
Mpkg/gitsync/internalbridge/config.go+13
94 unmodified lines
95
96
97
98
99
99
100
101
102
103
40 unmodified lines
144
145
146
147
148
147
149
150
151
152
94 unmodified lines
type ExecutionSummary struct {
DryRun bool `json:"dry_run"`
Protocol string `json:"protocol"`
OperationMode string `json:"operation_mode"`
Relay bool `json:"relay"`
Mode string `json:"mode"`
TransferMode string `json:"transfer_mode"`
Reason string `json:"reason"`
BootstrapSuggested bool `json:"bootstrap_suggested"`
Batch BatchSummary `json:"batch"`
40 unmodified lines
Execution: ExecutionSummary{
DryRun: result.DryRun,
Protocol: result.Protocol,
OperationMode: result.OperationMode,
Relay: result.Relay,
Mode: result.RelayMode,
TransferMode: result.RelayMode,
Reason: result.RelayReason,
BootstrapSuggested: result.BootstrapSuggested,
Batch: BatchSummary{
Mpkg/gitsync/internalbridge/model.go+4/-2
66 unmodified lines
67
68
69
70
71
72
73
11 unmodified lines
85
86
87
87
88
89
90
91
66 unmodified lines
Blocked: 3,
Deleted: 4,
DryRun: true,
OperationMode: "replicate",
Relay: true,
RelayMode: "incremental-relay",
RelayReason: "fast-forward",
11 unmodified lines
if got.Counts.Applied != 1 || got.Counts.Skipped != 2 || got.Counts.Blocked != 3 || got.Counts.Deleted != 4 {
t.Fatalf("unexpected counts: %+v", got.Counts)
}
if !got.Execution.DryRun || !got.Execution.Relay || got.Execution.Mode != "incremental-relay" || got.Execution.Reason != "fast-forward" {
if !got.Execution.DryRun || !got.Execution.Relay || got.Execution.OperationMode != "replicate" || got.Execution.TransferMode != "incremental-relay" || got.Execution.Reason != "fast-forward" {
t.Fatalf("unexpected execution summary: %+v", got.Execution)
}
if !got.Execution.Batch.Enabled || got.Execution.Batch.Done != 5 || got.Execution.Batch.Planned != 6 {
Mpkg/gitsync/internalbridge/model_test.go+2/-1
13 unmodified lines
14
15
16
17
18
19
20
21
22
23
24
25
26
27
50 unmodified lines
78
79
80
81
82
83
84
13 unmodified lines
ProtocolV2 ProtocolMode = "v2"
)
// OperationMode controls high-level sync semantics.
type OperationMode string
const (
ModeSync OperationMode = "sync"
ModeReplicate OperationMode = "replicate"
)
// Endpoint identifies a remote Git endpoint.
type Endpoint struct {
URL string
50 unmodified lines
// SyncPolicy controls high-level sync behavior.
type SyncPolicy struct {
Mode OperationMode
IncludeTags bool
Force bool
Prune bool
Mpkg/gitsync/types.go+9
37 unmodified lines
38
39
40
41
41
42
43
44
63 unmodified lines
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
57 unmodified lines
180
181
182
183
184
185
186
187
188
189
190
19 unmodified lines
210
211
212
201
213
214
215
216
38 unmodified lines
255
256
257
258
259
260
261
262
263
264
265
266
267
37 unmodified lines
MeasureMemory bool
Verbose bool
MaxPackBytes int64
BatchMaxPackBytes int64
TargetMaxPackBytes int64
MaterializedMaxObjects int
}
63 unmodified lines
return syncer.Run(ctx, cfg)
}
func (c *Client) Replicate(ctx context.Context, req SyncRequest) (Result, error) {
req.Policy.Mode = gitsync.ModeReplicate
cfg, err := c.buildSyncConfig(ctx, req)
if err != nil {
return Result{}, err
}
return syncer.Run(ctx, cfg)
}
func (c *Client) Bootstrap(ctx context.Context, req BootstrapRequest) (Result, error) {
cfg, err := c.buildBootstrapConfig(ctx, req)
if err != nil {
57 unmodified lines
DryRun: req.DryRun,
ShowStats: req.Options.CollectStats,
MeasureMemory: req.Options.MeasureMemory,
Mode: operationModeString(req.Policy.Mode),
Force: req.Policy.Force,
Prune: req.Policy.Prune,
MaxPackBytes: req.Options.MaxPackBytes,
TargetMaxPackBytes: req.Options.TargetMaxPackBytes,
MaterializedMaxObjects: maxObjects,
ProtocolMode: protocolString(req.Policy.Protocol),
Verbose: req.Options.Verbose,
19 unmodified lines
ShowStats: req.Options.CollectStats,
MeasureMemory: req.Options.MeasureMemory,
MaxPackBytes: req.Options.MaxPackBytes,
BatchMaxPackBytes: req.Options.BatchMaxPackBytes,
TargetMaxPackBytes: req.Options.TargetMaxPackBytes,
ProtocolMode: protocolString(req.Protocol),
Verbose: req.Options.Verbose,
}, nil
38 unmodified lines
return string(mode)
}
func operationModeString(mode gitsync.OperationMode) string {
if mode == "" {
return string(gitsync.ModeSync)
}
return string(mode)
}
func syncerEndpoint(endpoint gitsync.Endpoint, auth gitsync.EndpointAuth) syncer.Endpoint {
return internalbridge.ToSyncerEndpoint(
internalbridge.Endpoint{URL: endpoint.URL},
Mpkg/gitsync/unstable/client.go+21/-2