Add command to convert a repo from SHA1 to SHA256 · Entire
Log in
Add command to convert a repo from SHA1 to SHA256
726692f→main·
nodo·1mo ago·7 files·+2,207 added/-0 removed
Sessions
Transcript data is unavailable for this checkpoint.
Changes
7
.entire
M.gitignore+1
MREADME.md+3
cmd/git-sync
Aconvert_sha256.go+106
internal/sha256convert
Asha256convert.go+1016
Asha256convert_test.go+783
Mroot.go+1
docs
Aconvert-sha256.md+297
1 unmodified line
2
3
4
5
1 unmodified line
settings.local.json
metadata/
logs/
redactors/local/
M.entire/.gitignore+1
26 unmodified lines
27
28
29
30
31
32
33
34
60 unmodified lines
95
96
97
98
99
100
101
26 unmodified lines
`sync` automatically bootstraps an empty target, so the same command covers initial seeding and ongoing sync. To preview what would happen without pushing, run `git-sync plan` — it takes the same flags as `sync`, and `--mode replicate` previews a `replicate` run.
For one-off SHA1 → SHA256 repo conversion, `git-sync convert-sha256` fetches from an HTTP source and writes a new SHA256 bare repo on disk, with optional commit-message hash rewrites, an origin-notes ref, and a sidecar mapping file. See [docs/convert-sha256.md](docs/convert-sha256.md).
For command examples, JSON output, auth, protocol flags, and advanced command notes, see [docs/usage.md](docs/usage.md).
## Library API
60 unmodified lines
- [docs/usage.md](docs/usage.md) — CLI commands, examples, sync behavior, JSON output, auth, protocol notes
- [docs/architecture.md](docs/architecture.md) — product rationale, package layout, operation modes vs transfer modes, memory model
- [docs/protocol.md](docs/protocol.md) — smart HTTP, pkt-line, capability negotiation, sideband, relay framing
- [docs/convert-sha256.md](docs/convert-sha256.md) — one-off SHA1 → SHA256 repo conversion, mapping outputs, sharp edges
- [docs/testing.md](docs/testing.md) — test suites and integration coverage
## FAQ
MREADME.md+3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
package main
import (
"errors"
"fmt"
gitsync "entire.io/entire/git-sync"
"entire.io/entire/git-sync/cmd/git-sync/internal/sha256convert"
"entire.io/entire/git-sync/internal/validation"
"github.com/spf13/cobra"
)
func newConvertSHA256Cmd() *cobra.Command {
var (
req = sha256convert.Request{}
mappings []string
branches string
jsonOutput bool
protocolVal = newProtocolFlag()
)
cmd := &cobra.Command{
Use: "convert-sha256 [flags] <source-url> <target-dir>",
Short: "One-off SHA1 → SHA256 conversion of a remote repo into a local bare repo",
Long: `convert-sha256 fetches a pack from a SHA1 HTTP source and writes a new
SHA256 bare repository on disk at <target-dir>. Every reachable object is
re-hashed under SHA256 and tree/commit/tag references are rewritten.
The conversion is destructive in two ways the caller should be aware of:
no SHA1↔SHA256 mapping is persisted, and any GPG signatures on commits or
tags are dropped (they sign over the original SHA1 content and would be
invalid post-rewrite). Submodule gitlinks that point at a commit outside
this repository cannot be embedded in a SHA256 tree; if the source repo
contains any, the command exits with an error so the caller can scope
around the offending refs.`,
Args: cobra.MaximumNArgs(2),
SilenceErrors: true,
SilenceUsage: true,
RunE: func(cmd *cobra.Command, args []string) error {
req.ProtocolMode = gitsync.ProtocolMode(protocolVal)
if req.SourceURL == "" && len(args) > 0 {
req.SourceURL = args[0]
}
if req.TargetDir == "" && len(args) > 1 {
req.TargetDir = args[1]
}
if req.SourceURL == "" || req.TargetDir == "" {
return errors.New("convert-sha256 requires a source URL and a target directory")
}
if branches != "" {
req.Branches = splitCSV(branches)
}
for _, raw := range mappings {
mapping, err := validation.ParseMapping(raw)
if err != nil {
return fmt.Errorf("parse mapping %q: %w", raw, err)
}
req.Mappings = append(req.Mappings, gitsync.RefMapping{
Source: mapping.Source,
Target: mapping.Target,
})
}
result, err := sha256convert.Run(cmd.Context(), req)
if err != nil {
return fmt.Errorf("convert-sha256: %w", err)
}
printOutput(jsonOutput, result)
return nil
},
}
cmd.Flags().StringVar(&req.SourceURL, "source-url", "", "source repository URL")
cmd.Flags().BoolVar(&req.SourceFollowInfoRefsRedirect, "source-follow-info-refs-redirect",
envBool("GITSYNC_SOURCE_FOLLOW_INFO_REFS_REDIRECT"),
"send follow-up source RPCs to the final /info/refs redirect host")
cmd.Flags().StringVar(&req.SourceAuth.Token, "source-token",
envOr("GITSYNC_SOURCE_TOKEN", ""), "source token/password")
cmd.Flags().StringVar(&req.SourceAuth.Username, "source-username",
envOr("GITSYNC_SOURCE_USERNAME", "git"), "source basic auth username")
cmd.Flags().StringVar(&req.SourceAuth.BearerToken, "source-bearer-token",
envOr("GITSYNC_SOURCE_BEARER_TOKEN", ""), "source bearer token")
cmd.Flags().BoolVar(&req.SourceAuth.SkipTLSVerify, "source-insecure-skip-tls-verify",
envBool("GITSYNC_SOURCE_INSECURE_SKIP_TLS_VERIFY"),
"skip TLS certificate verification for the source")
cmd.Flags().StringVar(&req.TargetDir, "target-dir", "", "directory to initialize as a SHA256 bare repository")
cmd.Flags().StringVar(&branches, "branch", "", "comma-separated branch list; default is all source branches")
cmd.Flags().StringArrayVar(&mappings, "map", nil, "ref mapping in src:dst form; short names map branches, full refs map exact refs")
cmd.Flags().BoolVar(&req.IncludeTags, "tags", false, "include annotated and lightweight tags")
allRefsFlag(cmd, allRefsUsageScopeOnly, &req.AllRefs)
excludeRefPrefixFlag(cmd, &req.ExcludeRefPrefixes)
addProtocolFlag(cmd, &protocolVal)
cmd.Flags().BoolVarP(&req.Verbose, "verbose", "v", false, "verbose logging")
cmd.Flags().BoolVar(&req.KeepSourceObjects, "keep-source-objects", false,
"keep the temporary SHA1 store on disk after conversion (for debugging)")
cmd.Flags().StringVar(&req.MappingFile, "write-mapping", "",
"write the full SHA1 → SHA256 mapping as a TSV to this path; useful for rewriting external references")
cmd.Flags().BoolVar(&req.SkipMessageRewrite, "no-rewrite-messages", false,
"do not rewrite SHA1 hash references found in commit and tag messages")
cmd.Flags().BoolVar(&req.SkipOriginNotes, "no-origin-notes", false,
"do not write a refs/notes/sha1-origin ref recording each commit's original SHA1")
cmd.Flags().BoolVar(&jsonOutput, "json", false, "print JSON output")
return cmd
}
Acmd/git-sync/convert_sha256.go+106
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
// Package sha256convert implements a one-off SHA1 → SHA256 conversion for a
// single repository. It fetches a pack from a remote SHA1 HTTP endpoint into
// a temporary on-disk SHA1 bare repo, then walks every reachable object and
// re-emits it under SHA256 into a new bare repo at the user-supplied path.
//
// The tool is intentionally scoped: no hash mapping is persisted, GPG
// signatures on commits and tags are dropped (they sign over the original
// SHA1 byte stream and would be invalid post-rewrite), and submodule
// gitlinks are left at their original SHA1 hash unless the referenced
// commit happens to live in the same repo. A run that encounters an
// unresolvable submodule entry fails so the caller can choose which refs
// to exclude.
package sha256convert
import (
"bufio"
"bytes"
"compress/zlib"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"time"
git "github.com/go-git/go-git/v6"
"github.com/go-git/go-git/v6/plumbing"
"github.com/go-git/go-git/v6/plumbing/filemode"
formatcfg "github.com/go-git/go-git/v6/plumbing/format/config"
"github.com/go-git/go-git/v6/plumbing/object"
"github.com/go-git/go-git/v6/plumbing/storer"
transporthttp "github.com/go-git/go-git/v6/plumbing/transport/http"
"github.com/go-git/go-git/v6/storage/filesystem"
gitsync "entire.io/entire/git-sync"
"entire.io/entire/git-sync/internal/auth"
"entire.io/entire/git-sync/internal/convert"
"entire.io/entire/git-sync/internal/gitproto"
"entire.io/entire/git-sync/internal/planner"
)
// Request describes a single SHA1 → SHA256 conversion.
type Request struct {
SourceURL string
SourceAuth gitsync.EndpointAuth
SourceFollowInfoRefsRedirect bool
TargetDir string
Branches []string
IncludeTags bool
AllRefs bool
ExcludeRefPrefixes []string
Mappings []gitsync.RefMapping
ProtocolMode gitsync.ProtocolMode
Verbose bool
KeepSourceObjects bool
// MappingFile, when non-empty, is a path to which a TSV of every
// translated object's SHA1 → SHA256 mapping is written. Useful for
// rewriting external systems that reference old commit hashes.
MappingFile string
// SkipMessageRewrite disables the inline rewrite of SHA1 hashes found
// in commit and tag messages. Off by default (rewriting is on).
SkipMessageRewrite bool
// SkipOriginNotes disables the refs/notes/sha1-origin output that
// records each translated commit's original SHA1. Off by default
// (notes are written).
SkipOriginNotes bool
// Out receives human-readable status lines. Nil means os.Stderr.
Out io.Writer
}
// Counts tallies converted objects by kind.
type Counts struct {
Blobs int `json:"blobs"`
Trees int `json:"trees"`
Commits int `json:"commits"`
Tags int `json:"tags"`
}
// Result is the conversion summary, suitable for JSON output.
type Result struct {
SourceURL string `json:"sourceUrl"`
TargetDir string `json:"targetDir"`
Protocol string `json:"protocol"`
RefsConverted int `json:"refsConverted"`
Counts Counts `json:"counts"`
SignaturesStripped int `json:"signaturesStripped"`
MessageRewrites int `json:"messageRewrites"`
OriginNotesRef string `json:"originNotesRef,omitempty"`
MappingFile string `json:"mappingFile,omitempty"`
TempDir string `json:"tempDir,omitempty"`
}
// Lines satisfies the human-readable output contract used by other git-sync subcommands.
func (r Result) Lines() []string {
lines := []string{
fmt.Sprintf("sha256 bare repo: %s", r.TargetDir),
fmt.Sprintf("source: %s (%s)", r.SourceURL, r.Protocol),
fmt.Sprintf("converted: %d blobs, %d trees, %d commits, %d tags",
r.Counts.Blobs, r.Counts.Trees, r.Counts.Commits, r.Counts.Tags),
fmt.Sprintf("refs written: %d", r.RefsConverted),
}
if r.SignaturesStripped > 0 {
lines = append(lines, fmt.Sprintf("warning: stripped %d GPG signature(s); they no longer match the rewritten object content", r.SignaturesStripped))
}
if r.MessageRewrites > 0 {
lines = append(lines, fmt.Sprintf("rewrote %d SHA1 hash reference(s) in commit/tag messages", r.MessageRewrites))
}
if r.OriginNotesRef != "" {
lines = append(lines, fmt.Sprintf("origin notes ref: %s (use `git notes --ref=%s show <sha256>` to recover old SHA1)",
r.OriginNotesRef, strings.TrimPrefix(r.OriginNotesRef, "refs/notes/")))
}
if r.MappingFile != "" {
lines = append(lines, fmt.Sprintf("mapping written to: %s", r.MappingFile))
}
if r.TempDir != "" {
lines = append(lines, fmt.Sprintf("kept source objects: %s", r.TempDir))
}
return lines
}
// Run performs the conversion described by req.
func Run(ctx context.Context, req Request) (Result, error) {
if req.SourceURL == "" {
return Result{}, errors.New("convert-sha256 requires --source-url")
}
if req.TargetDir == "" {
return Result{}, errors.New("convert-sha256 requires a target directory")
}
out := req.Out
if out == nil {
out = os.Stderr
}
if err := ensureEmptyTarget(req.TargetDir); err != nil {
return Result{}, err
}
tempDir, err := os.MkdirTemp("", "git-sync-sha256-src-")
if err != nil {
return Result{}, fmt.Errorf("create temp dir: %w", err)
}
cleanupTemp := true
defer func() {
if cleanupTemp {
_ = os.RemoveAll(tempDir)
}
}()
srcRepo, err := git.PlainInit(tempDir, true)
if err != nil {
return Result{}, fmt.Errorf("init temporary SHA1 store: %w", err)
}
dstRepo, err := git.PlainInit(req.TargetDir, true, git.WithObjectFormat(formatcfg.SHA256))
if err != nil {
return Result{}, fmt.Errorf("init SHA256 target at %s: %w", req.TargetDir, err)
}
// Source connection + ref discovery -----------------------------------
planCfg := planner.PlanConfig{
Branches: append([]string(nil), req.Branches...),
Mappings: toPlannerMappings(req.Mappings),
IncludeTags: req.IncludeTags,
AllRefs: req.AllRefs,
ExcludeRefPrefixes: append([]string(nil), req.ExcludeRefPrefixes...),
}
conn, refService, sourceRefList, err := openSource(ctx, req, planCfg)
if err != nil {
return Result{}, err
}
defer conn.Close()
refService.Verbose = req.Verbose
sourceRefs := gitproto.RefHashMap(sourceRefList)
desired, _, err := planner.BuildDesiredRefs(sourceRefs, planCfg)
if err != nil {
return Result{}, fmt.Errorf("build desired refs: %w", err)
}
if len(desired) == 0 {
return Result{}, errors.New("no source refs matched the requested scope")
}
// Fetch into temp SHA1 store ------------------------------------------
fmt.Fprintf(out, "fetching %d ref(s) from %s ...\n", len(desired), req.SourceURL)
gpDesired := convert.DesiredRefs(desired)
if err := refService.FetchToStore(ctx, srcRepo.Storer, conn, gpDesired, nil); err != nil &&
!errors.Is(err, git.NoErrAlreadyUpToDate) {
return Result{}, fmt.Errorf("fetch source pack: %w", err)
}
// Discover + translate ------------------------------------------------
tr, err := newTranslator(srcRepo.Storer, dstRepo.Storer, req.TargetDir, !req.SkipMessageRewrite)
if err != nil {
return Result{}, err
}
rootSHA1s := make([]plumbing.Hash, 0, len(desired))
for _, d := range desired {
rootSHA1s = append(rootSHA1s, d.SourceHash)
}
fmt.Fprintln(out, "discovering reachable objects ...")
if err := tr.discover(rootSHA1s); err != nil {
return Result{}, fmt.Errorf("discover reachable: %w", err)
}
fmt.Fprintln(out, "translating objects to sha256 ...")
for _, d := range desired {
if _, err := tr.translate(d.SourceHash); err != nil {
return Result{}, fmt.Errorf("translate %s: %w", d.SourceRef, err)
}
}
// Write refs ---------------------------------------------------------
refsWritten, err := writeRefs(dstRepo.Storer, desired, tr.mapping)
if err != nil {
return Result{}, fmt.Errorf("write target refs: %w", err)
}
// Point HEAD at the source's symbolic HEAD if it landed in the
// converted ref set. PlainInit defaults HEAD to refs/heads/master,
// which often doesn't exist (e.g. repos using "main" as the default).
if refService.HeadTarget != "" {
if _, ok := desired[refService.HeadTarget]; ok {
head := plumbing.NewSymbolicReference(plumbing.HEAD, refService.HeadTarget)
if err := dstRepo.Storer.SetReference(head); err != nil {
return Result{}, fmt.Errorf("set HEAD: %w", err)
}
}
}
res := Result{
SourceURL: req.SourceURL,
TargetDir: req.TargetDir,
Protocol: refService.Protocol,
RefsConverted: refsWritten,
Counts: tr.counts,
SignaturesStripped: tr.signaturesStripped,
MessageRewrites: tr.messageRewrites,
}
if !req.SkipOriginNotes && len(tr.commits) > 0 {
notesRef, err := tr.writeOriginNotes(originNotesRef)
if err != nil {
return Result{}, fmt.Errorf("write origin notes: %w", err)
}
if err := dstRepo.Storer.SetReference(plumbing.NewHashReference(plumbing.ReferenceName(notesRef), tr.lastNotesCommit)); err != nil {
return Result{}, fmt.Errorf("set %s: %w", notesRef, err)
}
res.OriginNotesRef = notesRef
}
if req.MappingFile != "" {
if err := tr.writeMappingFile(req.MappingFile); err != nil {
return Result{}, fmt.Errorf("write mapping file: %w", err)
}
res.MappingFile = req.MappingFile
}
if req.KeepSourceObjects {
cleanupTemp = false
res.TempDir = tempDir
}
return res, nil
}
const originNotesRef = "refs/notes/sha1-origin"
// ensureEmptyTarget refuses to init into a non-empty directory so the user
// doesn't quietly accumulate objects into an existing repo.
func ensureEmptyTarget(path string) error {
entries, err := os.ReadDir(path)
if err != nil {
if os.IsNotExist(err) {
if mkErr := os.MkdirAll(path, 0o755); mkErr != nil {
return fmt.Errorf("create target dir: %w", mkErr)
}
return nil
}
return fmt.Errorf("read target dir: %w", err)
}
if len(entries) > 0 {
return fmt.Errorf("target directory %s is not empty", path)
}
return nil
}
func openSource(ctx context.Context, req Request, planCfg planner.PlanConfig) (gitproto.Conn, *gitproto.RefService, []*plumbing.Reference, error) {
ep, err := url.Parse(req.SourceURL)
if err != nil {
return nil, nil, nil, fmt.Errorf("parse source URL: %w", err)
}
if ep.Scheme != "http" && ep.Scheme != "https" {
return nil, nil, nil, fmt.Errorf("convert-sha256 currently supports HTTP/HTTPS sources only; got %q", ep.Scheme)
}
authMethod, err := auth.Resolve(auth.Endpoint{
Username: req.SourceAuth.Username,
Token: req.SourceAuth.Token,
BearerToken: req.SourceAuth.BearerToken,
SkipTLSVerify: req.SourceAuth.SkipTLSVerify,
}, ep)
if err != nil {
return nil, nil, nil, fmt.Errorf("resolve source auth: %w", err)
}
httpClient := &http.Client{Transport: gitproto.NewHTTPTransport(req.SourceAuth.SkipTLSVerify)}
conn := gitproto.NewHTTPConnWithClient(ep, "source", normalizeAuth(authMethod), httpClient)
conn.FollowInfoRefsRedirect = req.SourceFollowInfoRefsRedirect
mode := string(req.ProtocolMode)
if mode == "" {
mode = string(gitsync.ProtocolAuto)
}
refs, svc, err := gitproto.ListSourceRefs(ctx, conn, mode, planner.RefPrefixes(planCfg))
if err != nil {
_ = conn.Close()
return nil, nil, nil, fmt.Errorf("list source refs: %w", err)
}
return conn, svc, refs, nil
}
func normalizeAuth(m auth.Method) gitproto.AuthMethod {
if m == nil {
return nil
}
// auth.Method and gitproto.AuthMethod share the same Authorizer signature.
// Wrap so we can pass either *transporthttp.BasicAuth or *transporthttp.TokenAuth.
if a, ok := m.(*transporthttp.BasicAuth); ok {
return a
}
if a, ok := m.(*transporthttp.TokenAuth); ok {
return a
}
return authAdapter{m: m}
}
type authAdapter struct{ m auth.Method }
func (a authAdapter) Authorizer(req *http.Request) error { return a.m.Authorizer(req) }
func toPlannerMappings(in []gitsync.RefMapping) []planner.RefMapping {
out := make([]planner.RefMapping, 0, len(in))
for _, m := range in {
out = append(out, planner.RefMapping{Source: m.Source, Target: m.Target})
}
return out
}
// translator walks the SHA1 source store, rewrites object content with
// SHA256-mapped hashes, and writes the result as loose objects under the
// target bare repo. Loose object writing is done by hand because go-git
// v6 alpha 3's objfile.Writer hardcodes SHA1 in prepareForWrite (see
// plumbing/format/objfile/writer.go:68), which would store every SHA256
// object at a SHA1-derived path.
type translator struct {
src *filesystem.Storage
dst *filesystem.Storage
objectsDir string
// reachable holds every in-scope SHA1 with its object type, built up
// front by a discovery pass that walks tree/commit/tag dependencies
// from the desired ref tips. It is the authoritative "what's in
// scope" set: abbreviated SHA1 prefixes in commit/tag messages are
// resolved against this set so a unique match is fixed before any
// encoding starts, and so message-reference edges can be added to
// the translation DFS in topological order.
reachable map[plumbing.Hash]plumbing.ObjectType
mapping map[plumbing.Hash]plumbing.Hash
// inProgress detects cycles in the translation DFS. Real Git
// histories cannot form cycles (the parent/tree/tag-target edges
// are a DAG by construction, and SHA1 message-reference cycles are
// cryptographically infeasible), but a defensive guard turns
// surprising input into a clear error instead of a stack overflow.
inProgress map[plumbing.Hash]struct{}
// commits records every translated commit's old SHA1, in DFS order,
// for use by writeOriginNotes. We track separately rather than walking
// the full mapping because notes only attach meaningfully to commits.
commits []plumbing.Hash
counts Counts
signaturesStripped int
messageRewrites int
rewriteMessages bool
lastNotesCommit plumbing.Hash
}
func newTranslator(src, dst storer.Storer, targetDir string, rewriteMessages bool) (*translator, error) {
srcFS, ok := src.(*filesystem.Storage)
if !ok {
return nil, fmt.Errorf("source storage is not filesystem-backed (%T)", src)
}
dstFS, ok := dst.(*filesystem.Storage)
if !ok {
return nil, fmt.Errorf("target storage is not filesystem-backed (%T)", dst)
}
return &translator{
src: srcFS,
dst: dstFS,
objectsDir: filepath.Join(targetDir, "objects"),
reachable: make(map[plumbing.Hash]plumbing.ObjectType),
mapping: make(map[plumbing.Hash]plumbing.Hash),
inProgress: make(map[plumbing.Hash]struct{}),
rewriteMessages: rewriteMessages,
}, nil
}
// discover walks every object reachable from roots (via tree entries,
// commit tree+parent links, and tag targets) and records each one in
// t.reachable with its object type. Submodule gitlinks are followed
// only when the referenced commit exists in the same source store, to
// stay consistent with translateTree's handling. Message-reference
// edges are not part of this pass — those are added during translation.
func (t *translator) discover(roots []plumbing.Hash) error {
for _, root := range roots {
if err := t.visit(root); err != nil {
return err
}
}
return nil
}
func (t *translator) visit(sha1 plumbing.Hash) error {
if _, seen := t.reachable[sha1]; seen {
return nil
}
obj, err := t.src.EncodedObject(plumbing.AnyObject, sha1)
if err != nil {
return fmt.Errorf("discover %s: %w", sha1, err)
}
t.reachable[sha1] = obj.Type()
switch obj.Type() {
case plumbing.BlobObject:
return nil
case plumbing.TreeObject:
tree := &object.Tree{}
if err := tree.Decode(obj); err != nil {
return fmt.Errorf("discover decode tree %s: %w", sha1, err)
}
for _, e := range tree.Entries {
if e.Mode == filemode.Submodule {
if _, err := t.src.EncodedObject(plumbing.CommitObject, e.Hash); err == nil {
if err := t.visit(e.Hash); err != nil {
return err
}
}
continue
}
if err := t.visit(e.Hash); err != nil {
return err
}
}
case plumbing.CommitObject:
c := &object.Commit{}
if err := c.Decode(obj); err != nil {
return fmt.Errorf("discover decode commit %s: %w", sha1, err)
}
if err := t.visit(c.TreeHash); err != nil {
return err
}
for _, p := range c.ParentHashes {
if err := t.visit(p); err != nil {
return err
}
}
case plumbing.TagObject:
tag := &object.Tag{}
if err := tag.Decode(obj); err != nil {
return fmt.Errorf("discover decode tag %s: %w", sha1, err)
}
if err := t.visit(tag.Target); err != nil {
return err
}
}
return nil
}
func (t *translator) translate(sha1 plumbing.Hash) (plumbing.Hash, error) {
if newH, ok := t.mapping[sha1]; ok {
return newH, nil
}
if _, busy := t.inProgress[sha1]; busy {
// Real Git histories cannot form cycles via parent, tree, or
// tag-target edges (those are a DAG by construction), and
// SHA1 message-reference cycles are cryptographically
// infeasible (each commit's hash depends on its content,
// including any hash it embeds). A trip here would mean an
// unexpected graph shape; surface it instead of overflowing
// the stack.
return plumbing.ZeroHash, fmt.Errorf("translation cycle detected at %s", sha1)
}
t.inProgress[sha1] = struct{}{}
defer delete(t.inProgress, sha1)
obj, err := t.src.EncodedObject(plumbing.AnyObject, sha1)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("lookup %s: %w", sha1, err)
}
switch obj.Type() {
case plumbing.BlobObject:
return t.translateBlob(sha1, obj)
case plumbing.TreeObject:
return t.translateTree(sha1, obj)
case plumbing.CommitObject:
return t.translateCommit(sha1, obj)
case plumbing.TagObject:
return t.translateTag(sha1, obj)
default:
return plumbing.ZeroHash, fmt.Errorf("unexpected object type %v for %s", obj.Type(), sha1)
}
}
func (t *translator) translateBlob(sha1 plumbing.Hash, src plumbing.EncodedObject) (plumbing.Hash, error) {
r, err := src.Reader()
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("blob reader: %w", err)
}
defer r.Close()
body, err := io.ReadAll(r)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("blob read: %w", err)
}
newHash, err := t.writeLoose(plumbing.BlobObject, body)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("blob store: %w", err)
}
t.mapping[sha1] = newHash
t.counts.Blobs++
return newHash, nil
}
func (t *translator) translateTree(sha1 plumbing.Hash, src plumbing.EncodedObject) (plumbing.Hash, error) {
tree := &object.Tree{}
if err := tree.Decode(src); err != nil {
return plumbing.ZeroHash, fmt.Errorf("decode tree %s: %w", sha1, err)
}
for i, entry := range tree.Entries {
if entry.Mode == filemode.Submodule {
// Submodule gitlinks reference a commit in a different repo.
// We can only translate if that commit happens to live in our
// SHA1 store too (rare, e.g. vendored). Otherwise the SHA1
// pointer can't be embedded in a SHA256 tree, so we error
// out and let the caller scope around it.
if _, ok := t.mapping[entry.Hash]; ok {
tree.Entries[i].Hash = t.mapping[entry.Hash]
continue
}
if _, err := t.src.EncodedObject(plumbing.CommitObject, entry.Hash); err == nil {
newH, err := t.translate(entry.Hash)
if err != nil {
return plumbing.ZeroHash, err
}
tree.Entries[i].Hash = newH
continue
}
return plumbing.ZeroHash, fmt.Errorf(
"tree %s contains submodule gitlink %q at %s that is not present in the source repo; "+
"exclude refs that reference it or convert the submodule repository first",
sha1, entry.Name, entry.Hash)
}
newH, err := t.translate(entry.Hash)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("tree %s entry %q: %w", sha1, entry.Name, err)
}
tree.Entries[i].Hash = newH
}
body, err := encodeBody(plumbing.TreeObject, tree.Encode)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("encode tree %s: %w", sha1, err)
}
newHash, err := t.writeLoose(plumbing.TreeObject, body)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("store tree %s: %w", sha1, err)
}
t.mapping[sha1] = newHash
t.counts.Trees++
return newHash, nil
}
func (t *translator) translateCommit(sha1 plumbing.Hash, src plumbing.EncodedObject) (plumbing.Hash, error) {
c := &object.Commit{}
if err := c.Decode(src); err != nil {
return plumbing.ZeroHash, fmt.Errorf("decode commit %s: %w", sha1, err)
}
newTree, err := t.translate(c.TreeHash)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("commit %s tree: %w", sha1, err)
}
c.TreeHash = newTree
for i, p := range c.ParentHashes {
newP, err := t.translate(p)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("commit %s parent %s: %w", sha1, p, err)
}
c.ParentHashes[i] = newP
}
if t.rewriteMessages {
// Translate every in-scope SHA1 mentioned in this commit's
// message before rewriting it. This makes the message-reference
// edge part of the translation DFS, so the mapping contains
// each referenced object by the time we substitute. Without
// it, sibling-branch references (cherry-picks, etc.) would
// only resolve when ref iteration happened to process the
// referenced commit's branch first.
for _, ref := range t.extractMessageReferences(c.Message) {
if _, err := t.translate(ref); err != nil {
return plumbing.ZeroHash, fmt.Errorf("commit %s message ref %s: %w", sha1, ref, err)
}
}
if rewritten, n := t.rewriteHashesInMessage(c.Message); n > 0 {
c.Message = rewritten
t.messageRewrites += n
}
}
if c.Signature != "" {
c.Signature = ""
t.signaturesStripped++
}
if c.SignatureSHA256 != "" {
c.SignatureSHA256 = ""
t.signaturesStripped++
}
// "mergetag" extra headers embed a copy of a signed annotated tag with
// its own signature. Drop them too — they reference the pre-rewrite
// commit/tag content and cannot be re-signed here.
if len(c.ExtraHeaders) > 0 {
filtered := c.ExtraHeaders[:0]
for _, h := range c.ExtraHeaders {
if h.Key == "mergetag" {
t.signaturesStripped++
continue
}
filtered = append(filtered, h)
}
c.ExtraHeaders = filtered
}
body, err := encodeBody(plumbing.CommitObject, c.Encode)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("encode commit %s: %w", sha1, err)
}
newHash, err := t.writeLoose(plumbing.CommitObject, body)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("store commit %s: %w", sha1, err)
}
t.mapping[sha1] = newHash
t.commits = append(t.commits, sha1)
t.counts.Commits++
return newHash, nil
}
func (t *translator) translateTag(sha1 plumbing.Hash, src plumbing.EncodedObject) (plumbing.Hash, error) {
tag := &object.Tag{}
if err := tag.Decode(src); err != nil {
return plumbing.ZeroHash, fmt.Errorf("decode tag %s: %w", sha1, err)
}
newTarget, err := t.translate(tag.Target)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("tag %s target: %w", sha1, err)
}
tag.Target = newTarget
if t.rewriteMessages {
// Same as translateCommit: translate every in-scope message
// reference before rewriting, so cross-branch references
// always resolve regardless of ref iteration order.
for _, ref := range t.extractMessageReferences(tag.Message) {
if _, err := t.translate(ref); err != nil {
return plumbing.ZeroHash, fmt.Errorf("tag %s message ref %s: %w", sha1, ref, err)
}
}
if rewritten, n := t.rewriteHashesInMessage(tag.Message); n > 0 {
tag.Message = rewritten
t.messageRewrites += n
}
}
if tag.Signature != "" {
tag.Signature = ""
t.signaturesStripped++
}
if tag.SignatureSHA256 != "" {
tag.SignatureSHA256 = ""
t.signaturesStripped++
}
body, err := encodeBody(plumbing.TagObject, tag.Encode)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("encode tag %s: %w", sha1, err)
}
newHash, err := t.writeLoose(plumbing.TagObject, body)
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("store tag %s: %w", sha1, err)
}
t.mapping[sha1] = newHash
t.counts.Tags++
return newHash, nil
}
// encodeBody runs an object's go-git Encode method into a SHA1-hasher
// MemoryObject (the hasher we use to capture bytes is irrelevant; we only
// read the body back out) and returns just the payload bytes — without the
// "<type> <size>\x00" header. writeLoose adds the SHA256-correct header.
func encodeBody(typ plumbing.ObjectType, encode func(plumbing.EncodedObject) error) ([]byte, error) {
scratch := plumbing.NewMemoryObject(plumbing.FromObjectFormat(formatcfg.SHA1))
scratch.SetType(typ)
if err := encode(scratch); err != nil {
return nil, err
}
r, err := scratch.Reader()
if err != nil {
return nil, err
}
defer r.Close()
return io.ReadAll(r)
}
// writeLoose writes a single object as a SHA256-named loose object under
// objects/<aa>/<rest>. Bypasses go-git's objfile.Writer, which would hash
// with SHA1. Atomic via tempfile+rename, idempotent on duplicate hashes.
func (t *translator) writeLoose(typ plumbing.ObjectType, body []byte) (plumbing.Hash, error) {
h := sha256.New()
header := append(typ.Bytes(), ' ')
header = strconv.AppendInt(header, int64(len(body)), 10)
header = append(header, 0)
h.Write(header)
h.Write(body)
sum := h.Sum(nil)
hexSum := hex.EncodeToString(sum)
dir := filepath.Join(t.objectsDir, hexSum[:2])
file := filepath.Join(dir, hexSum[2:])
hashID, ok := plumbing.FromBytes(sum)
if !ok {
return plumbing.ZeroHash, fmt.Errorf("internal: bad sha256 sum length %d", len(sum))
}
if _, err := os.Stat(file); err == nil {
return hashID, nil
}
if err := os.MkdirAll(dir, 0o755); err != nil {
return plumbing.ZeroHash, fmt.Errorf("mkdir %s: %w", dir, err)
}
var buf bytes.Buffer
zw := zlib.NewWriter(&buf)
if _, err := zw.Write(header); err != nil {
return plumbing.ZeroHash, fmt.Errorf("zlib write header: %w", err)
}
if _, err := zw.Write(body); err != nil {
return plumbing.ZeroHash, fmt.Errorf("zlib write body: %w", err)
}
if err := zw.Close(); err != nil {
return plumbing.ZeroHash, fmt.Errorf("zlib close: %w", err)
}
tmp, err := os.CreateTemp(dir, "tmp_obj_")
if err != nil {
return plumbing.ZeroHash, fmt.Errorf("create temp object: %w", err)
}
if _, err := tmp.Write(buf.Bytes()); err != nil {
_ = tmp.Close()
_ = os.Remove(tmp.Name())
return plumbing.ZeroHash, fmt.Errorf("write temp object: %w", err)
}
if err := tmp.Close(); err != nil {
_ = os.Remove(tmp.Name())
return plumbing.ZeroHash, fmt.Errorf("close temp object: %w", err)
}
if err := os.Rename(tmp.Name(), file); err != nil {
_ = os.Remove(tmp.Name())
return plumbing.ZeroHash, fmt.Errorf("rename %s: %w", file, err)
}
return hashID, nil
}
// hashPattern matches hex runs that could be a git object hash. Git's
// default abbreviation is 7 chars; 40 is a full SHA1. We only rewrite a
// match if the prefix uniquely identifies a commit or tag in the
// reachable set, so false positives on incidental hex strings are
// essentially impossible (a random hex would have to collide with a
// real source SHA1).
var hashPattern = regexp.MustCompile(`\b[0-9a-f]{7,40}\b`)
// rewriteHashesInMessage scans msg for short and full SHA1 hashes,
// replacing any that uniquely identify a commit or tag in t.reachable
// with the corresponding full SHA256 hex from t.mapping. Returns the
// rewritten message and the number of substitutions made.
//
// Uniqueness is decided against t.reachable rather than t.mapping so
// that abbreviated prefixes get the same verdict during translation as
// they would after every object has been translated — the answer cannot
// flip depending on what has been processed so far.
//
// Performance: the abbreviated-hash path scans the reachable set
// linearly for each match. Fine for repos up to ~100k commits; slower
// past that. If this ever matters, build a sorted-prefix index over
// reachable SHA1 hex strings once and binary-search.
func (t *translator) rewriteHashesInMessage(msg string) (string, int) {
count := 0
out := hashPattern.ReplaceAllStringFunc(msg, func(s string) string {
sha1, ok := t.resolveMessageRef(s)
if !ok {
return s
}
newHash, ok := t.mapping[sha1]
if !ok {
// The reachable set says this SHA1 is in scope, but the
// translation DFS hasn't placed it yet. Shouldn't happen
// because translateCommit/translateTag add message-reference
// edges before encoding — leave the hex untouched if it
// somehow does.
return s
}
count++
return newHash.String()
})
return out, count
}
// resolveMessageRef returns the unique commit/tag SHA1 in t.reachable
// that matches the given hex prefix. Returns (zero, false) for no
// match, an ambiguous prefix, or a match that is not a commit or tag
// (incidental hex strings that happen to collide with a blob or tree
// hash are not rewritten).
func (t *translator) resolveMessageRef(prefix string) (plumbing.Hash, bool) {
if len(prefix) == 40 {
sha1, ok := plumbing.FromHex(prefix)
if !ok {
return plumbing.ZeroHash, false
}
typ, in := t.reachable[sha1]
if !in {
return plumbing.ZeroHash, false
}
if typ != plumbing.CommitObject && typ != plumbing.TagObject {
return plumbing.ZeroHash, false
}
return sha1, true
}
var match plumbing.Hash
matches := 0
for sha1, typ := range t.reachable {
if typ != plumbing.CommitObject && typ != plumbing.TagObject {
continue
}
if strings.HasPrefix(sha1.String(), prefix) {
matches++
if matches > 1 {
return plumbing.ZeroHash, false
}
match = sha1
}
}
if matches != 1 {
return plumbing.ZeroHash, false
}
return match, true
}
// extractMessageReferences returns the unique commit/tag SHA1s mentioned
// by hex prefix in msg. Used by translateCommit/translateTag to add
// message-reference edges to the translation DFS so the mapping is
// fully populated by the time the message is rewritten.
func (t *translator) extractMessageReferences(msg string) []plumbing.Hash {
seen := make(map[plumbing.Hash]struct{})
var out []plumbing.Hash
for _, match := range hashPattern.FindAllString(msg, -1) {
sha1, ok := t.resolveMessageRef(match)
if !ok {
continue
}
if _, dup := seen[sha1]; dup {
continue
}
seen[sha1] = struct{}{}
out = append(out, sha1)
}
return out
}
// writeOriginNotes writes a `git notes` ref to dst that records each
// translated commit's original SHA1, keyed by its new SHA256. Standard
// git tooling (`git log --notes=<ref>`, `git notes --ref=<ref> show
// <commit>`) can then surface the old hash to anyone with the repo.
//
// The notes tree is flat (no fanout). Git supports either layout, and a
// flat layout keeps this code small; on repos with millions of commits
// lookups slow down to a linear tree scan, but the data is preserved.
func (t *translator) writeOriginNotes(refName string) (string, error) {
if len(t.commits) == 0 {
return "", nil
}
// Note for each commit: a blob containing the original SHA1 hex + newline.
// We collect (sha256-of-new-commit → blob hash) pairs so the tree entry
// path is the commit's new hash.
type entry struct {
key plumbing.Hash
blob plumbing.Hash
}
entries := make([]entry, 0, len(t.commits))
for _, oldSHA1 := range t.commits {
newCommit, ok := t.mapping[oldSHA1]
if !ok {
continue
}
blobHash, err := t.writeLoose(plumbing.BlobObject, []byte(oldSHA1.String()+"\n"))
if err != nil {
return "", fmt.Errorf("note blob for %s: %w", oldSHA1, err)
}
entries = append(entries, entry{key: newCommit, blob: blobHash})
}
if len(entries) == 0 {
return "", nil
}
treeEntries := make([]object.TreeEntry, 0, len(entries))
for _, e := range entries {
treeEntries = append(treeEntries, object.TreeEntry{
Name: e.key.String(),
Mode: filemode.Regular,
Hash: e.blob,
})
}
sort.Slice(treeEntries, func(i, j int) bool {
return treeEntries[i].Name < treeEntries[j].Name
})
tree := &object.Tree{Entries: treeEntries}
treeBody, err := encodeBody(plumbing.TreeObject, tree.Encode)
if err != nil {
return "", fmt.Errorf("encode notes tree: %w", err)
}
treeHash, err := t.writeLoose(plumbing.TreeObject, treeBody)
if err != nil {
return "", fmt.Errorf("store notes tree: %w", err)
}
now := time.Now().UTC()
sig := object.Signature{Name: "git-sync", Email: "noreply@entire.io", When: now}
commit := &object.Commit{
Author: sig,
Committer: sig,
Message: "git-sync convert-sha256: SHA1 origin notes\n",
TreeHash: treeHash,
}
commitBody, err := encodeBody(plumbing.CommitObject, commit.Encode)
if err != nil {
return "", fmt.Errorf("encode notes commit: %w", err)
}
commitHash, err := t.writeLoose(plumbing.CommitObject, commitBody)
if err != nil {
return "", fmt.Errorf("store notes commit: %w", err)
}
t.lastNotesCommit = commitHash
return refName, nil
}
// writeMappingFile dumps the SHA1 → SHA256 mapping as a TSV. Lines are
// sorted by SHA1 so diffs across runs are stable. Includes every
// translated object (blob/tree/commit/tag), so external tooling can use
// it for content-addressed lookups regardless of object kind.
func (t *translator) writeMappingFile(path string) error {
type pair struct{ sha1, sha256 string }
pairs := make([]pair, 0, len(t.mapping))
for old, newH := range t.mapping {
pairs = append(pairs, pair{sha1: old.String(), sha256: newH.String()})
}
sort.Slice(pairs, func(i, j int) bool { return pairs[i].sha1 < pairs[j].sha1 })
if dir := filepath.Dir(path); dir != "" {
if err := os.MkdirAll(dir, 0o755); err != nil {
return fmt.Errorf("mkdir %s: %w", dir, err)
}
}
f, err := os.Create(path)
if err != nil {
return fmt.Errorf("create %s: %w", path, err)
}
defer f.Close()
w := bufio.NewWriter(f)
if _, err := fmt.Fprintln(w, "# sha1\tsha256"); err != nil {
return err
}
for _, p := range pairs {
if _, err := fmt.Fprintf(w, "%s\t%s\n", p.sha1, p.sha256); err != nil {
return err
}
}
return w.Flush()
}
func writeRefs(
dst storer.Storer,
desired map[plumbing.ReferenceName]planner.DesiredRef,
mapping map[plumbing.Hash]plumbing.Hash,
) (int, error) {
written := 0
for _, d := range desired {
newHash, ok := mapping[d.SourceHash]
if !ok {
return written, fmt.Errorf("ref %s tip %s missing from translation map", d.TargetRef, d.SourceHash)
}
if err := dst.SetReference(plumbing.NewHashReference(d.TargetRef, newHash)); err != nil {
return written, fmt.Errorf("set ref %s: %w", d.TargetRef, err)
}
written++
}
return written, nil
}
Acmd/git-sync/internal/sha256convert/sha256convert.go+1016
package sha256convert
import ( "bytes" "compress/zlib" "context" "crypto/sha256" "encoding/hex" "fmt" "io" "net/http" "net/http/cgi" "net/http/httptest" "os" "os/exec" "path/filepath" "strings" "testing" "time"
git "github.com/go-git/go-git/v6" "github.com/go-git/go-git/v6/plumbing" "github.com/go-git/go-git/v6/plumbing/filemode" formatcfg "github.com/go-git/go-git/v6/plumbing/format/config" "github.com/go-git/go-git/v6/plumbing/object" "github.com/go-git/go-git/v6/storage/filesystem" )
// TestTranslator builds a small SHA1 source repo with blobs, trees, commits, // and an annotated tag — including signed commit/tag — then runs the // translator and asserts both the bookkeeping counts and the on-disk // invariant: every loose object's filename equals sha256(headered content). // That invariant is the one go-git v6 alpha 3 gets wrong via its // SetEncodedObject path; verifying it directly prevents regressing back // onto the broken loose-object writer. func TestTranslator(t *testing.T) { root := t.TempDir() srcDir := filepath.Join(root, "src.git") dstDir := filepath.Join(root, "dst.git")
srcRepo, err := git.PlainInit(srcDir, true) if err != nil { t.Fatalf("init SHA1 source: %v", err) } dstRepo, err := git.PlainInit(dstDir, true, git.WithObjectFormat(formatcfg.SHA256)) if err != nil { t.Fatalf("init SHA256 target: %v", err) }
blobHash := writeBlob(t, srcRepo.Storer, []byte("hello world\n")) treeHash := writeTree(t, srcRepo.Storer, []object.TreeEntry{ {Name: "README", Mode: filemode.Regular, Hash: blobHash}, })
sig := object.Signature{Name: "Test", Email: "test@example.com", When: time.Unix(1700000000, 0).UTC()} commit1 := &object.Commit{ Author: sig, Committer: sig, Message: "initial\n", TreeHash: treeHash, Signature: "-----BEGIN PGP SIGNATURE-----\nfake sig data\n-----END PGP SIGNATURE-----", } c1Hash := writeObject(t, srcRepo.Storer, commit1.Encode)
commit2 := &object.Commit{ Author: sig, Committer: sig, Message: "second\n", TreeHash: treeHash, ParentHashes: []plumbing.Hash{c1Hash}, } c2Hash := writeObject(t, srcRepo.Storer, commit2.Encode)
tag := &object.Tag{ Name: "v1", Tagger: sig, Message: "annotated tag\n", TargetType: plumbing.CommitObject, Target: c2Hash, Signature: "-----BEGIN PGP SIGNATURE-----\nfake tag sig\n-----END PGP SIGNATURE-----", } tagHash := writeObject(t, srcRepo.Storer, tag.Encode)
tr, err := newTranslator(srcRepo.Storer, dstRepo.Storer, dstDir, false) if err != nil { t.Fatalf("newTranslator: %v", err) } if err := tr.discover([]plumbing.Hash{tagHash}); err != nil { t.Fatalf("discover: %v", err) } newTagHash, err := tr.translate(tagHash) if err != nil { t.Fatalf("translate tag: %v", err) }
wantCounts := Counts{Blobs: 1, Trees: 1, Commits: 2, Tags: 1} if tr.counts != wantCounts { t.Errorf("counts: got %+v, want %+v", tr.counts, wantCounts) } if tr.signaturesStripped != 2 { t.Errorf("signatures stripped: got %d, want 2 (commit + tag)", tr.signaturesStripped) }
// Idempotency: translating the same hash again must reuse the mapping // without writing more objects or bumping counters. startBlobs := tr.counts.Blobs if _, err := tr.translate(tagHash); err != nil { t.Fatalf("re-translate tag: %v", err) } if tr.counts.Blobs != startBlobs { t.Errorf("re-translate increased blob count; memoization broken") }
// Every translated hash must point at a loose object whose filename // equals sha256(headered content). This is the precise invariant the // go-git bug violates — keep it as a test. objectsDir := filepath.Join(dstDir, "objects") verified := 0 for _, h := range tr.mapping { assertLooseObjectHashMatches(t, objectsDir, h) verified++ } if verified == 0 { t.Fatal("no objects in mapping; nothing was verified") }
// The translated tag must decode under the SHA256 target and point at // a SHA256 commit whose tree resolves to a SHA256 tree. tagObj, err := object.GetTag(dstRepo.Storer, newTagHash) if err != nil { t.Fatalf("read translated tag: %v", err) } if tagObj.Signature != "" { t.Errorf("translated tag still carries a signature: %q", tagObj.Signature) } if tagObj.Target != tr.mapping[c2Hash] { t.Errorf("translated tag target: got %s, want %s", tagObj.Target, tr.mapping[c2Hash]) }
commit, err := object.GetCommit(dstRepo.Storer, tagObj.Target) if err != nil { t.Fatalf("read translated commit: %v", err) } if commit.Signature != "" { t.Errorf("translated commit still carries a signature: %q", commit.Signature) } if len(commit.ParentHashes) != 1 || commit.ParentHashes[0] != tr.mapping[c1Hash] { t.Errorf("translated commit parents: got %v, want [%s]", commit.ParentHashes, tr.mapping[c1Hash]) } if commit.TreeHash != tr.mapping[treeHash] { t.Errorf("translated commit tree: got %s, want %s", commit.TreeHash, tr.mapping[treeHash]) } }
// TestTranslator_RewritesMessageHashes confirms that SHA1 hash references // in commit and tag messages — both full 40-char and short forms — are // rewritten to the corresponding SHA256 when those SHA1s are translated // objects in the same conversion, and that ambiguous/unknown short // prefixes are left alone. func TestTranslator_RewritesMessageHashes(t *testing.T) { root := t.TempDir() srcDir := filepath.Join(root, "src.git") dstDir := filepath.Join(root, "dst.git")
blobHash := writeBlob(t, srcRepo.Storer, []byte("x\n")) treeHash := writeTree(t, srcRepo.Storer, []object.TreeEntry{ {Name: "f", Mode: filemode.Regular, Hash: blobHash}, }) sig := object.Signature{Name: "Test", Email: "t@example.com", When: time.Unix(1700000000, 0).UTC()} parent := &object.Commit{Author: sig, Committer: sig, Message: "first\n", TreeHash: treeHash} parentSHA1 := writeObject(t, srcRepo.Storer, parent.Encode)
// Child commit's message references the parent by full hash, by 7-char // short prefix, and includes an unrelated 7-char hex string that should // not match anything in the mapping. parentHex := parentSHA1.String() childMsg := fmt.Sprintf( "reverts %s\nsee short %s for context\nunrelated hex 1234567 follows\n", parentHex, parentHex[:7]) child := &object.Commit{ Author: sig, Committer: sig, Message: childMsg, TreeHash: treeHash, ParentHashes: []plumbing.Hash{parentSHA1}, } childSHA1 := writeObject(t, srcRepo.Storer, child.Encode)
tr, err := newTranslator(srcRepo.Storer, dstRepo.Storer, dstDir, true) if err != nil { t.Fatalf("newTranslator: %v", err) } if err := tr.discover([]plumbing.Hash{childSHA1}); err != nil { t.Fatalf("discover: %v", err) } if _, err := tr.translate(childSHA1); err != nil { t.Fatalf("translate child: %v", err) }
// 2 references should have been rewritten (full + short). The unrelated // 7-char hex string is not in the mapping, so it stays. if tr.messageRewrites != 2 { t.Errorf("message rewrites: got %d, want 2", tr.messageRewrites) }
childNew := tr.mapping[childSHA1] parentNew := tr.mapping[parentSHA1] gotChild, err := object.GetCommit(dstRepo.Storer, childNew) if err != nil { t.Fatalf("read translated child: %v", err) } if !strings.Contains(gotChild.Message, parentNew.String()) { t.Errorf("child message missing full SHA256 of parent:\n%s", gotChild.Message) } if strings.Contains(gotChild.Message, parentHex) { t.Errorf("child message still contains original parent SHA1:\n%s", gotChild.Message) } if !strings.Contains(gotChild.Message, "1234567") { t.Errorf("unrelated short hex was wrongly substituted:\n%s", gotChild.Message) } }
// TestTranslator_RewritesCrossBranchReferences is the test that proves the // discovery-plus-topological-DFS design fixes the cross-branch limitation // the older inline-only rewriter had. Two unrelated branches share no // ancestry. Branch A has a single commit cA. Branch B has commit cB whose // message references cA by both full and abbreviated SHA1. We translate B // first, then A — the order under which the older code would have left // cB's message un-rewritten because cA was not yet in the mapping when cB // was encoded. With message-reference edges in the DFS, translating cB // pulls cA in via t.translate, so the mapping is populated and the // rewrite succeeds. func TestTranslator_RewritesCrossBranchReferences(t *testing.T) { root := t.TempDir() srcDir := filepath.Join(root, "src.git") dstDir := filepath.Join(root, "dst.git") srcRepo, _ := git.PlainInit(srcDir, true) dstRepo, _ := git.PlainInit(dstDir, true, git.WithObjectFormat(formatcfg.SHA256))
blobA := writeBlob(t, srcRepo.Storer, []byte("a\n")) treeA := writeTree(t, srcRepo.Storer, []object.TreeEntry{ {Name: "a", Mode: filemode.Regular, Hash: blobA}, }) blobB := writeBlob(t, srcRepo.Storer, []byte("b\n")) treeB := writeTree(t, srcRepo.Storer, []object.TreeEntry{ {Name: "b", Mode: filemode.Regular, Hash: blobB}, })
sig := object.Signature{Name: "Test", Email: "t@example.com", When: time.Unix(1700000000, 0).UTC()} cA := writeObject(t, srcRepo.Storer, (&object.Commit{ Author: sig, Committer: sig, Message: "branch A tip\n", TreeHash: treeA, }).Encode) // cB has no parent in common with cA — they are siblings under // no ancestor, exactly the case where ancestor-only inline // rewriting would have failed. cAHex := cA.String() cB := writeObject(t, srcRepo.Storer, (&object.Commit{ Author: sig, Committer: sig, Message: fmt.Sprintf("branch B tip\n\nCherry-picked from %s\nsee short %s\n", cAHex, cAHex[:8]), TreeHash: treeB, }).Encode)
tr, _ := newTranslator(srcRepo.Storer, dstRepo.Storer, dstDir, true) // Discovery must see both branches so the reachable set covers cA // before cB is encoded. if err := tr.discover([]plumbing.Hash{cB, cA}); err != nil { t.Fatalf("discover: %v", err) } // Translate B first — the order that would have left the rewrite // stranded under the old design. if _, err := tr.translate(cB); err != nil { t.Fatalf("translate cB: %v", err) } if _, err := tr.translate(cA); err != nil { t.Fatalf("translate cA: %v", err) }
if tr.messageRewrites != 2 { t.Errorf("expected 2 rewrites (full + short SHA1 of cA), got %d", tr.messageRewrites) } cBNew := tr.mapping[cB] cANew := tr.mapping[cA] if cBNew.IsZero() || cANew.IsZero() { t.Fatalf("missing mapping entries: cB=%s cA=%s", cBNew, cANew) } gotB, err := object.GetCommit(dstRepo.Storer, cBNew) if err != nil { t.Fatalf("read cB: %v", err) } if !strings.Contains(gotB.Message, cANew.String()) { t.Errorf("cB's message missing cA's SHA256:\n%s", gotB.Message) } if strings.Contains(gotB.Message, cAHex) { t.Errorf("cB's message still contains cA's original SHA1:\n%s", gotB.Message) } }
// TestTranslator_SkipMessageRewrite confirms that with rewriteMessages // false, the translator leaves message content (including SHA1 hashes) // untouched. func TestTranslator_SkipMessageRewrite(t *testing.T) { root := t.TempDir() srcRepo, _ := git.PlainInit(filepath.Join(root, "src.git"), true) dstRepo, _ := git.PlainInit(filepath.Join(root, "dst.git"), true, git.WithObjectFormat(formatcfg.SHA256))
blob := writeBlob(t, srcRepo.Storer, []byte("x\n")) tree := writeTree(t, srcRepo.Storer, []object.TreeEntry{{Name: "f", Mode: filemode.Regular, Hash: blob}}) sig := object.Signature{Name: "Test", Email: "t@example.com", When: time.Unix(1, 0).UTC()} parent := writeObject(t, srcRepo.Storer, (&object.Commit{Author: sig, Committer: sig, Message: "p\n", TreeHash: tree}).Encode) parentHex := parent.String()
child := &object.Commit{ Author: sig, Committer: sig, TreeHash: tree, ParentHashes: []plumbing.Hash{parent}, Message: "reverts " + parentHex + "\n", } childSHA1 := writeObject(t, srcRepo.Storer, child.Encode)
tr, _ := newTranslator(srcRepo.Storer, dstRepo.Storer, filepath.Join(root, "dst.git"), false) if err := tr.discover([]plumbing.Hash{childSHA1}); err != nil { t.Fatalf("discover: %v", err) } if _, err := tr.translate(childSHA1); err != nil { t.Fatalf("translate: %v", err) } if tr.messageRewrites != 0 { t.Errorf("expected no rewrites when disabled; got %d", tr.messageRewrites) } got, _ := object.GetCommit(dstRepo.Storer, tr.mapping[childSHA1]) if !strings.Contains(got.Message, parentHex) { t.Errorf("rewrite-disabled run still mutated the message: %q", got.Message) } }
// TestTranslator_WriteOriginNotes builds a small history and verifies that // the notes tree contains one entry per translated commit and that each // entry resolves to a blob whose content is the commit's original SHA1. func TestTranslator_WriteOriginNotes(t *testing.T) { root := t.TempDir() srcRepo, _ := git.PlainInit(filepath.Join(root, "src.git"), true) dstRepo, _ := git.PlainInit(filepath.Join(root, "dst.git"), true, git.WithObjectFormat(formatcfg.SHA256))
blob := writeBlob(t, srcRepo.Storer, []byte("hi\n")) tree := writeTree(t, srcRepo.Storer, []object.TreeEntry{{Name: "f", Mode: filemode.Regular, Hash: blob}}) sig := object.Signature{Name: "Test", Email: "t@example.com", When: time.Unix(1700000000, 0).UTC()} c1 := writeObject(t, srcRepo.Storer, (&object.Commit{Author: sig, Committer: sig, Message: "c1\n", TreeHash: tree}).Encode) c2 := writeObject(t, srcRepo.Storer, (&object.Commit{Author: sig, Committer: sig, Message: "c2\n", TreeHash: tree, ParentHashes: []plumbing.Hash{c1}}).Encode)
tr, _ := newTranslator(srcRepo.Storer, dstRepo.Storer, filepath.Join(root, "dst.git"), false) if err := tr.discover([]plumbing.Hash{c2}); err != nil { t.Fatalf("discover: %v", err) } if _, err := tr.translate(c2); err != nil { t.Fatalf("translate: %v", err) }
refName, err := tr.writeOriginNotes(originNotesRef) if err != nil { t.Fatalf("writeOriginNotes: %v", err) } if refName != originNotesRef { t.Errorf("ref name: got %q, want %q", refName, originNotesRef) } notesCommit, err := object.GetCommit(dstRepo.Storer, tr.lastNotesCommit) if err != nil { t.Fatalf("read notes commit: %v", err) } notesTree, err := notesCommit.Tree() if err != nil { t.Fatalf("read notes tree: %v", err) } if len(notesTree.Entries) != 2 { t.Fatalf("notes entries: got %d, want 2", len(notesTree.Entries)) } for _, mapped := range []plumbing.Hash{tr.mapping[c1], tr.mapping[c2]} { entry, err := notesTree.FindEntry(mapped.String()) if err != nil { t.Fatalf("no notes entry for %s: %v", mapped, err) } blob, err := object.GetBlob(dstRepo.Storer, entry.Hash) if err != nil { t.Fatalf("read note blob: %v", err) } reader, _ := blob.Reader() buf, _ := io.ReadAll(reader) _ = reader.Close() got := strings.TrimSpace(string(buf)) var origSHA1 plumbing.Hash for s, n := range tr.mapping { if n == mapped { origSHA1 = s break } } if got != origSHA1.String() { t.Errorf("note for %s: got %q, want %q", mapped, got, origSHA1.String()) } } }
// TestTranslator_WriteMappingFile checks the sidecar TSV format: header // line, sorted by SHA1, one entry per translated object. func TestTranslator_WriteMappingFile(t *testing.T) { root := t.TempDir() srcRepo, _ := git.PlainInit(filepath.Join(root, "src.git"), true) dstRepo, _ := git.PlainInit(filepath.Join(root, "dst.git"), true, git.WithObjectFormat(formatcfg.SHA256))
blob := writeBlob(t, srcRepo.Storer, []byte("hi\n")) tree := writeTree(t, srcRepo.Storer, []object.TreeEntry{{Name: "f", Mode: filemode.Regular, Hash: blob}}) sig := object.Signature{Name: "Test", Email: "t@example.com", When: time.Unix(1700000000, 0).UTC()} commit := writeObject(t, srcRepo.Storer, (&object.Commit{Author: sig, Committer: sig, Message: "c\n", TreeHash: tree}).Encode)
tr, _ := newTranslator(srcRepo.Storer, dstRepo.Storer, filepath.Join(root, "dst.git"), false) if err := tr.discover([]plumbing.Hash{commit}); err != nil { t.Fatalf("discover: %v", err) } if _, err := tr.translate(commit); err != nil { t.Fatalf("translate: %v", err) }
path := filepath.Join(root, "mapping.tsv") if err := tr.writeMappingFile(path); err != nil { t.Fatalf("writeMappingFile: %v", err) } raw, err := os.ReadFile(path) if err != nil { t.Fatalf("read mapping: %v", err) } lines := strings.Split(strings.TrimRight(string(raw), "\n"), "\n") if !strings.HasPrefix(lines[0], "#") { t.Errorf("first line should be a header comment, got %q", lines[0]) } data := lines[1:] if len(data) != len(tr.mapping) { t.Errorf("mapping line count: got %d, want %d", len(data), len(tr.mapping)) } // Sorted by SHA1. for i := 1; i < len(data); i++ { prev := strings.Split(data[i-1], "\t")[0] cur := strings.Split(data[i], "\t")[0] if prev >= cur { t.Errorf("mapping not sorted: %q >= %q", prev, cur) } } // Every translated hash present. mapped := map[string]string{} for _, line := range data { parts := strings.Split(line, "\t") if len(parts) != 2 { t.Errorf("malformed line %q", line) continue } mapped[parts[0]] = parts[1] } for old, newH := range tr.mapping { if mapped[old.String()] != newH.String() { t.Errorf("missing or wrong mapping for %s: got %q, want %s", old, mapped[old.String()], newH) } } }
// TestTranslator_UnresolvableSubmodule confirms that a tree entry with // Submodule mode pointing at a commit not in the source repo causes a // clear error rather than silently producing a malformed SHA256 tree. func TestTranslator_UnresolvableSubmodule(t *testing.T) { root := t.TempDir() srcDir := filepath.Join(root, "src.git") dstDir := filepath.Join(root, "dst.git")
blobHash := writeBlob(t, srcRepo.Storer, []byte("contents\n")) // External-looking SHA1 — not in source. external := plumbing.NewHash("0123456789abcdef0123456789abcdef01234567") treeHash := writeTree(t, srcRepo.Storer, []object.TreeEntry{ {Name: "file", Mode: filemode.Regular, Hash: blobHash}, {Name: "sub", Mode: filemode.Submodule, Hash: external}, })
tr, err := newTranslator(srcRepo.Storer, dstRepo.Storer, dstDir, false) if err != nil { t.Fatalf("newTranslator: %v", err) } if err := tr.discover([]plumbing.Hash{treeHash}); err != nil { t.Fatalf("discover: %v", err) } _, err = tr.translate(treeHash) if err == nil { t.Fatal("expected error for unresolvable submodule, got nil") } if !strings.Contains(err.Error(), "submodule") { t.Errorf("error should mention submodule; got: %v", err) } }
// --- helpers ---
func writeBlob(t *testing.T, storer interface { NewEncodedObject() plumbing.EncodedObject SetEncodedObject(plumbing.EncodedObject) (plumbing.Hash, error) }, content []byte) plumbing.Hash { t.Helper() obj := storer.NewEncodedObject() obj.SetType(plumbing.BlobObject) obj.SetSize(int64(len(content))) w, err := obj.Writer() if err != nil { t.Fatalf("blob writer: %v", err) } if _, err := w.Write(content); err != nil { t.Fatalf("blob write: %v", err) } if err := w.Close(); err != nil { t.Fatalf("blob close: %v", err) } h, err := storer.SetEncodedObject(obj) if err != nil { t.Fatalf("blob store: %v", err) } return h }
func writeTree(t *testing.T, storer interface { NewEncodedObject() plumbing.EncodedObject SetEncodedObject(plumbing.EncodedObject) (plumbing.Hash, error) }, entries []object.TreeEntry) plumbing.Hash { t.Helper() tree := &object.Tree{Entries: entries} // object.Tree.Encode requires the slice to be sorted by name; tests // pre-sort their entries, but be safe. return writeObject(t, storer, tree.Encode) }
func writeObject(t *testing.T, storer interface { NewEncodedObject() plumbing.EncodedObject SetEncodedObject(plumbing.EncodedObject) (plumbing.Hash, error) }, encode func(plumbing.EncodedObject) error) plumbing.Hash { t.Helper() obj := storer.NewEncodedObject() if err := encode(obj); err != nil { t.Fatalf("encode: %v", err) } h, err := storer.SetEncodedObject(obj) if err != nil { t.Fatalf("store: %v", err) } return h }
// assertLooseObjectHashMatches reads the on-disk loose object for h, zlib-
// decompresses it, and confirms sha256(decompressed bytes) == h. The
// decompressed bytes include the "
func makeHex(b []byte) string { return hex.EncodeToString(b) }
// --- Integration test (gated) ---
const gitHTTPBackendEnv = "GITSYNC_E2E_SHA256_HTTP_BACKEND"
// TestRun_GitHTTPBackend exercises the full convert-sha256 pipeline against // a local git http-backend serving a real SHA1 source repo. Gated like the // other end-to-end git-http-backend tests to keep the default test runs // hermetic (no external binaries required). func TestRun_GitHTTPBackend(t *testing.T) { if os.Getenv(gitHTTPBackendEnv) == "" { t.Skipf("set %s=1 to run the convert-sha256 git-http-backend integration test", gitHTTPBackendEnv) } gitBin, err := exec.LookPath("git") if err != nil { t.Skipf("git binary not available: %v", err) }
root := t.TempDir() srcBare := filepath.Join(root, "source.git") worktree := filepath.Join(root, "work") dstDir := filepath.Join(root, "target.git")
mustGit(t, root, "init", "--bare", srcBare) mustGit(t, root, "init", "-b", "main", worktree) mustGit(t, worktree, "config", "user.name", "convert-sha256 test") mustGit(t, worktree, "config", "user.email", "test@example.com") mustWrite(t, filepath.Join(worktree, "README"), "hello\n") mustGit(t, worktree, "add", "README") mustGit(t, worktree, "commit", "-m", "initial") // Capture the first commit's SHA1 so the second commit's message can // reference it (both full and abbreviated). The conversion should // rewrite both to the new SHA256 hash. firstSHA1 := strings.TrimSpace(mustGitOutput(t, worktree, "rev-parse", "HEAD")) mustWrite(t, filepath.Join(worktree, "second.txt"), "world\n") mustGit(t, worktree, "add", "second.txt") mustGit(t, worktree, "commit", "-m", fmt.Sprintf("second\n\nreverts %s\nsee short %s", firstSHA1, firstSHA1[:7])) mustGit(t, worktree, "tag", "-a", "v1", "-m", "first tag") mustGit(t, worktree, "remote", "add", "origin", srcBare) mustGit(t, worktree, "push", "origin", "HEAD:refs/heads/main") mustGit(t, worktree, "push", "origin", "v1")
srv := newCGIBackend(t, gitBin, root) defer srv.Close()
mappingPath := filepath.Join(root, "mapping.tsv") res, err := Run(context.Background(), Request{ SourceURL: srv.URL + "/source.git", TargetDir: dstDir, IncludeTags: true, MappingFile: mappingPath, Out: io.Discard, }) if err != nil { t.Fatalf("convert-sha256 run: %v", err) } if res.Counts.Commits < 2 { t.Errorf("expected at least 2 commits converted, got %+v", res.Counts) } if res.Counts.Tags != 1 { t.Errorf("expected 1 tag converted, got %d", res.Counts.Tags) } if res.RefsConverted < 2 { t.Errorf("expected at least 2 refs (main + v1), got %d", res.RefsConverted) }
// The converted repo must be self-consistent under SHA256. fsckOut, err := exec.Command(gitBin, "-C", dstDir, "fsck", "--full").CombinedOutput() if err != nil { t.Fatalf("git fsck failed: %v\n%s", err, fsckOut) } if strings.Contains(string(fsckOut), "error") || strings.Contains(string(fsckOut), "bad sha") { t.Fatalf("git fsck reported errors:\n%s", fsckOut) }
// Sanity: extensions.objectformat is set, and git can walk the history. format := mustGitOutput(t, dstDir, "config", "extensions.objectformat") if strings.TrimSpace(format) != "sha256" { t.Errorf("extensions.objectformat: got %q, want %q", strings.TrimSpace(format), "sha256") } log := mustGitOutput(t, dstDir, "log", "--oneline", "refs/heads/main") if !strings.Contains(log, "initial") || !strings.Contains(log, "second") { t.Errorf("git log missing expected commit subjects:\n%s", log) } tagShow := mustGitOutput(t, dstDir, "cat-file", "-p", "refs/tags/v1") if !strings.Contains(tagShow, "first tag") { t.Errorf("annotated tag did not round-trip:\n%s", tagShow) }
// Message rewriting: the second commit's body referenced firstSHA1 // twice (full + 7-char short). Both should now be SHA256 hashes. if res.MessageRewrites != 2 { t.Errorf("message rewrites: got %d, want 2", res.MessageRewrites) } secondMsg := mustGitOutput(t, dstDir, "log", "-1", "--format=%B", "refs/heads/main") if strings.Contains(secondMsg, firstSHA1) { t.Errorf("second commit message still contains the original SHA1:\n%s", secondMsg) }
// Origin notes: the ref exists, and the head commit's note resolves // to the original SHA1 it was rewritten from. if res.OriginNotesRef != "refs/notes/sha1-origin" { t.Errorf("OriginNotesRef: got %q, want refs/notes/sha1-origin", res.OriginNotesRef) } headSHA256 := strings.TrimSpace(mustGitOutput(t, dstDir, "rev-parse", "refs/heads/main")) note := strings.TrimSpace(mustGitOutput(t, dstDir, "notes", "--ref=sha1-origin", "show", headSHA256)) // The note for the second (head) commit holds its pre-conversion SHA1. headSHA1 := strings.TrimSpace(mustGitOutput(t, srcBare, "rev-parse", "refs/heads/main")) if note != headSHA1 { t.Errorf("origin note for head: got %q, want %q", note, headSHA1) }
// Mapping file: present, sorted, has at least one entry per // translated commit/tree/blob/tag. if res.MappingFile != mappingPath { t.Errorf("MappingFile: got %q, want %q", res.MappingFile, mappingPath) } mapping, err := os.ReadFile(mappingPath) if err != nil { t.Fatalf("read mapping file: %v", err) } if !strings.Contains(string(mapping), headSHA1) { t.Errorf("mapping file missing head SHA1 %s:\n%s", headSHA1, mapping) } }
func mustGit(t *testing.T, dir string, args ...string) { t.Helper() cmd := exec.Command("git", args...) cmd.Dir = dir cmd.Env = append(os.Environ(), "GIT_TERMINAL_PROMPT=0") if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out) } }
func mustGitOutput(t *testing.T, dir string, args ...string) string { t.Helper() cmd := exec.Command("git", args...) cmd.Dir = dir cmd.Env = append(os.Environ(), "GIT_TERMINAL_PROMPT=0") out, err := cmd.CombinedOutput() if err != nil { t.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out) } return string(out) }
func mustWrite(t *testing.T, path, content string) { t.Helper() if err := os.WriteFile(path, []byte(content), 0o644); err != nil { t.Fatalf("write %s: %v", path, err) } }
type cgiBackend struct { *httptest.Server }
func newCGIBackend(t *testing.T, gitBin, root string) *cgiBackend { t.Helper() handler := &cgi.Handler{ Path: gitBin, Args: []string{"http-backend"}, Env: []string{ "GIT_PROJECT_ROOT=" + root, "GIT_HTTP_EXPORT_ALL=1", }, } srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { handler.ServeHTTP(w, r) })) return &cgiBackend{Server: srv} }
// Compile-time sanity: confirm the storers the translator expects are still // the filesystem-backed type that PlainInit returns. If a future go-git // release changes the concrete storer, the type assertion in newTranslator // will start failing in this package's tests rather than only at runtime // against a real repo. var _ = (*filesystem.Storage)(nil)
Acmd/git-sync/internal/sha256convert/sha256convert\_test.go+783
35 unmodified lines
36 37 38 39 40 41 42
35 unmodified lines
cmd.AddCommand(newBootstrapCmd()) cmd.AddCommand(newProbeCmd()) cmd.AddCommand(newFetchCmd()) cmd.AddCommand(newConvertSHA256Cmd()) cmd.AddCommand(newVersionCmd())
return cmd
Mcmd/git-sync/root.go+1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297
SHA1 → SHA256 Conversion
git-sync convert-sha256 is a one-off migration command that fetches a pack
from a SHA1 HTTP source and writes a new SHA256 bare repository on disk.
Every reachable object is re-hashed under SHA256 and tree, commit, and tag
references are rewritten accordingly.
The command is intentionally narrow: it does not push to a remote, it does not modify the source, and it is meant to be run once per repo. Resulting SHA256 hashes have no relation to the original SHA1 hashes beyond a mapping that the command can optionally emit.
Quick Start
git-sync convert-sha256 --tags \
https://github.com/source-org/source-repo.git \
/path/to/out.git
The target directory must not exist or must be empty. The result is a bare
repository with extensions.objectformat = sha256 and a refs/notes/sha1-origin
ref recording each commit's pre-conversion SHA1.
For a private source, pass the token via the environment so it isn't
exposed in ps:
GITSYNC_SOURCE_TOKEN=ghp_xxx git-sync convert-sha256 --tags \
https://github.com/source-org/private-repo.git \
/path/to/out.git
What It Does
- Probes the source via smart HTTP and discovers refs matching the
requested scope (
--branch,--tags,--all-refs,--map,--exclude-ref-prefix). - Fetches a single self-contained pack via
upload-packand lands it in a temporary on-disk SHA1 bare repo. The temp directory is cleaned up on exit unless--keep-source-objectsis passed. - Initializes the target as a bare SHA256 repository
(
git init --object-format=sha256equivalent). - Runs a discovery pass that walks every reachable object from each desired ref tip and records its SHA1 and object type. This gives the rewriter an authoritative "what's in scope" set so abbreviated message references can be resolved consistently and message-reference edges can be added to the translation graph.
- Translates every reachable object in topological order via a
memoized DFS:
- Blobs: re-hashed under SHA256; content unchanged.
- Trees: each entry's hash translated via the in-memory mapping; submodule gitlinks left as-is when the referenced commit is in this repo, otherwise the run errors.
- Commits:
treeandparenthashes translated; GPG signatures dropped;mergetagextra headers dropped; in-scope SHA1 references in the message are translated first (so their SHA256s are known) and then substituted into the message. - Tags: target hash translated; signatures dropped; tag message hashes rewritten with the same edge mechanism as commits.
- Writes refs in the SHA256 target at the translated tip hashes. HEAD is repointed at the source's symbolic HEAD when that ref made it into the conversion.
- Optionally writes the SHA1 → SHA256 mapping as a TSV sidecar
(
--write-mapping <path>).
The temp SHA1 store is on disk, not in memory, so peak RAM is bounded by the in-memory mapping plus a small fixed delta-resolution cache. Large repos still work; expect runtime dominated by the network fetch and the loose-object write throughput.
Handling External SHA1 References
A SHA1 → SHA256 cutover is destructive for external systems that reference commits by hash: PR descriptions, issue trackers, deploy logs, container labels, doc links, and so on all stop resolving. The command offers three on-ramps for migrating those out of band.
1. Inline message rewriting (default on)
Commit and tag messages are scanned for 7-to-40-character hex runs. When a run uniquely matches a commit or tag SHA1 in the conversion's reachable set, it is replaced with the full SHA256 hex. Examples that get rewritten:
Reverts: a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0 → full SHA256
Cherry-picked from a1b2c3d → full SHA256
Two design notes:
- Uniqueness is decided against the reachable set, not the
in-flight mapping. The discovery pass enumerates every reachable
SHA1 before any encoding starts, so abbreviated prefixes get the
same verdict regardless of how far the translation has progressed.
If
a1b2c3dmatches two different commits in scope, it is treated as ambiguous and left alone — never rewritten on the basis of which one happened to be translated first. - Cross-branch references work the same as ancestor references. Each in-scope SHA1 mentioned in a commit's message is added as a dependency edge in the translation DFS — that commit is translated before the referencing commit is encoded. So a cherry-pick from a sibling branch resolves just as reliably as a revert of an ancestor. (Cycles in this graph are cryptographically impossible: for both A's message to contain B's SHA1 and vice versa, you would need to know each hash before computing it.)
False positives are essentially impossible because a run is only
substituted if its prefix uniquely matches a real SHA1 of a commit or
tag in scope. Blob and tree hashes are excluded from the match set
so incidental hex strings that collide with content hashes are not
rewritten. Disable with --no-rewrite-messages if you prefer
untouched messages.
2. Origin notes ref (default on)
refs/notes/sha1-origin is written after translation and holds, for
each translated commit, the pre-conversion SHA1 hex keyed by the new
SHA256 hash. Standard git tooling can read it:
git -C /path/to/out.git notes --ref=sha1-origin show <sha256>
# prints the original SHA1
git -C /path/to/out.git log --notes=sha1-origin
# shows the original SHA1 below each commit's body
Notes attach meaningfully only to commits, so blobs, trees, and tags
are not represented in this ref. Disable with --no-origin-notes.
3. Sidecar mapping file (opt in)
--write-mapping <path> emits a TSV with one line per translated
object, sorted by SHA1:
# sha1 sha256
00027b675386b21c4ca05316145671fb7034d251 d80415fa21bebb...
000bb155604d06f1c48fc7feb4b025d991ef3366 a23cf98db5abfa...
...
Useful for bulk rewriting external systems: feed the file to a script that walks Jira tickets, PR bodies, deploy manifests, or any other system that holds frozen SHA1 references.
Flags
--source-url source repository URL
--source-token source password/token (prefer env)
--source-username source basic auth username (default git)
--source-bearer-token source bearer token
--source-insecure-skip-tls-verify skip TLS verification (testing only)
--source-follow-info-refs-redirect follow /info/refs cross-host redirects
--target-dir SHA256 bare repo directory (must be empty)
--branch comma-separated branch list
--tags include annotated and lightweight tags
--all-refs include every refs/* on the source
--exclude-ref-prefix subtract refs by prefix; repeatable
--map ref mapping in src:dst form; repeatable
--protocol protocol mode (auto, v1, v2)
--write-mapping write SHA1 → SHA256 TSV to this path
--no-rewrite-messages skip inline hash rewrites in messages
--no-origin-notes skip refs/notes/sha1-origin
--keep-source-objects leave the temp SHA1 store on disk
--json machine-readable output
--verbose, -v verbose logging
Environment fallbacks: GITSYNC_SOURCE_TOKEN, GITSYNC_SOURCE_USERNAME,
GITSYNC_SOURCE_BEARER_TOKEN, GITSYNC_SOURCE_INSECURE_SKIP_TLS_VERIFY,
GITSYNC_SOURCE_FOLLOW_INFO_REFS_REDIRECT, GITSYNC_PROTOCOL.
Sharp Edges
GPG signatures are stripped. A signature is bytes signed over the
commit's pre-conversion content (including the SHA1 tree and
parent lines). After rewriting, the bytes no longer match the
signature, so verification would always fail. Rather than persist
invalid signatures, the command drops them and prints a warning count.
This matches upstream git's own SHA256 conversion behavior. Tags
with embedded signatures and mergetag extra headers are handled the
same way.
Submodule gitlinks must resolve in-repo. Tree entries with mode
160000 reference a commit in another repository, but a SHA1 hash
cannot be embedded in a SHA256 tree. The command translates the
pointer if the referenced commit happens to live in the same store
(rare; sometimes seen in vendored modules), and otherwise exits with
an error naming the offending tree, entry, and hash. Scope around
those refs with --exclude-ref-prefix or --branch, or convert the
submodule repository first.
External SHA1 references break silently. See the section above for mitigations. References inside the repo (commit and tag messages) are rewritten when they uniquely identify a commit or tag in scope. Anything outside the repo — PR descriptions, issue trackers, deploy manifests, container labels — is not the converter's job; use the mapping file to drive those rewrites.
Replace refs and notes refs become detached. refs/replace/<sha1>
encodes a SHA1 in the ref name itself, so the name doesn't match
under SHA256 and the replacement never triggers. refs/notes/* paths
encode the target object's hash as a tree path, so existing notes
copied under --all-refs survive as data but no longer attach to
their original commits. Neither is a correctness issue, just lost
behavior.
HEAD can dangle. When the source's symbolic HEAD branch is not in
the desired ref set, the target's HEAD is left at refs/heads/master
(go-git's PlainInit default) and resolves to nothing. Either include
the HEAD branch in scope or set it manually after conversion with
git -C <target> symbolic-ref HEAD refs/heads/<branch>.
Storage is all loose objects. The command writes one file per
object. Correct, but on filesystems that dislike millions of small
files this is slow. Run git -C <target> gc --aggressive afterwards
to pack the converted repo down to a single packfile.
Verifying the Output
Standard git tooling works against the converted repo without
additional flags — the extensions.objectformat setting in the local
config is enough for git to switch hashing:
git -C /path/to/out.git fsck --full # zero errors expected
git -C /path/to/out.git config extensions.objectformat # prints sha256
git -C /path/to/out.git log --oneline -5 # SHA256 hashes
git -C /path/to/out.git log --notes=sha1-origin -5 # with original SHA1
To use the result as a working repo:
git clone /path/to/out.git /path/to/checkout
To serve it from a host that accepts SHA256:
git -C /path/to/out.git push --mirror <new-remote-url>
Implementation Notes
The translator works in four phases:
Pack fetch. A single self-contained pack is streamed into a filesystem-backed SHA1 storer via go-git's pack parser, so deltas are resolved up front and the SHA1 source is randomly addressable for the rest of the run.
Discovery. A non-encoding DFS walks every object reachable from each desired ref tip via tree entries, commit tree+parent links, and tag targets. Each visited SHA1 is recorded in a
reachable map[Hash]ObjectType. This set is the authoritative "what is in scope" answer used by both submodule resolution and message-reference rewriting — uniqueness of abbreviated SHA1 prefixes is decided against this set once, never against the in-flight mapping.Translation. Memoized recursive DFS from each desired ref tip. Blobs are copied as-is and re-hashed; trees, commits, and tags are decoded, their embedded hashes rewritten via the SHA1 → SHA256 mapping, signatures stripped, and messages rewritten. The DFS recursion includes message-reference edges: for each commit or tag whose message mentions a SHA1 of a commit or tag in the reachable set, that referenced object is translated first. This guarantees the mapping is populated before the substitution happens, so cross-branch references resolve as reliably as ancestor references. Each translated object is written as a loose object under
objects/<aa>/<rest>in the target.Refs and side outputs. Refs and HEAD are written at the translated tip hashes; the origin notes commit (if enabled) is built and stored under
refs/notes/sha1-origin; the mapping file (if requested) is written.
A defensive inProgress set guards against cycles during phase 3.
Real Git histories cannot form cycles (parent, tree, and tag-target
edges are a DAG by construction, and SHA1 message-reference cycles
are cryptographically infeasible), so a trip into this branch is a
hard error rather than a silent skip.
Note: loose object writing is done by hand rather than via go-git's
SetEncodedObject. The underlying plumbing/format/objfile.Writer
in go-git/v6@v6.0.0-alpha.3 hardcodes SHA1 in its hasher, which
would put every translated object at a SHA1-derived path even though
the content references SHA256. This is verified by a unit test that
recomputes sha256 of every loose object's decompressed content and
compares against the filename.
Adocs/convert-sha256.md+297