Stream-parse the pack as it uploads to expose objects-sent counter

242ad00→main·

Soph·2mo ago·4 files·+339 added/-29 removed

Replaces packReadCounter with packStreamObserver, a wrapping ReadCloser that adds two new instruments to the byte counter:

• ObjectsSent() — objects whose zlib stream has been fully walked by us (and therefore fully handed off to the HTTP layer)
• TotalObjects() — pack-header object count, populated as soon as the first 12 bytes flow through

The implementation tees the pack via io.Pipe to a goroutine running go-git's packfile.Scanner, which sequentially emits HeaderSection, one ObjectSection per object, and FooterSection. Each object increments the atomic; each header populates TotalObjects and closes HeaderReady so callers can wait without polling.

The current bootstrap loop only logs the new counters (objects_sent, total_objects_in_pack) on push failures so verbose runs can sanity-check the parse — no behaviour change yet. The next commit uses these counters to abort uploads early once we project we'll exceed the budget.

Cost note: Scanner does full zlib decompression to find object boundaries (compressed size isn't recorded in the pack format), plus a per-object SHA-1 over the inflated bytes. On modern hardware this runs at 200–500 MB/s per core, well above typical upload speeds, so the observer adds CPU but should not bottleneck the upload. If profiling later shows otherwise, the escape hatch is a custom format walker that drops Scanner's hashing — but the zlib walk itself is unavoidable.

Sessions

a66652708dc7View transcript

Changes

4

• internal/strategy/bootstrap
  • Mbootstrap.go+8/-23
  • Mbootstrap_test.go+12/-6
  • Apack_observer.go+153
  • Apack_observer_test.go+166

400 unmodified lines

// bytesPerObject lets the caller use a per-run calibrated value instead of the static estimatedBytesPerObject default. Calibrating after each rejection (using the bytes that flowed through packReadCounter) catches blob-heavy repos where the static 750-byte average is 10–20× too low — without calibration the pre-flight would let oversized sub-packs through and the loop would only learn after another wasted ~limit-sized upload. The subdivide callback