Use observed pack size to size post-rejection subdivisions · Entire

Use observed pack size to size post-rejection subdivisions

d239127→main· Soph·2mo ago·2 files·+431 added/-21 removed

Bootstrap's batched-mode subdivide path was always halving the remaining checkpoint count when a target rejected a pack with HTTP 413. For a repo whose real per-object byte size is far above the per-object heuristic (estimatedBytesPerObject = 750), this forces a long dance: 1 → 2 → 4 → 8 → 16 → 32 sub-packs across five rejected pushes, each preceded by a fresh source fetch. For a ~960 MiB pack against Cloudflare's ~500 MiB body cap, that's five wasted ~500 MiB uploads.

Wrap the pack reader handed to PushPack in a packReadCounter so the loop learns how many bytes actually went up before the cutoff and sizes the next attempt accordingly. Three cooperating pieces:

  1. observedSubdivisionFactor uses bytes_sent ÷ limit (with safety multiplier) to choose how many sub-packs to split into. When sent_bytes ≈ limit (server cut us off mid-stream — the common reverse-proxy 413 case), the true pack size is unknown but likely much larger than the cap, so the multiplier escalates to 4× to converge in a single round. Otherwise 2× is enough.

  2. subdivideToFactor halves at least once per call and keeps going while count < target. The unconditional first round matters when factor ≤ remaining: each surviving range may still over-shoot so always making forward progress prevents a hard failure on repeated 413s with sent_bytes ≈ limit.

  3. checkPackSizeAndSubdivide now takes a calibrated bytesPerObject instead of using the static 750-byte default. After each 413, calibrateBytesPerObject derives 2 × sent_bytes ÷ pack_objects as a pessimistic upper bound and ratchets the running estimate up. Subsequent sub-packs are pre-emptively split when the calibrated estimate exceeds the limit — saving an entire ~limit-sized wasted upload on blob-heavy repos where the default ratio is 10–20× too low.

Falls back to the legacy halving behaviour when no signal is available (sentBytes ≤ 0 or limit ≤ 0). Surfaces sent_bytes, limit_bytes, factor, and calibrated_bytes_per_object in the slog output so verbose runs show the math.

Sessions

6f5d837779f7View transcript

Changes

2

223 unmodified lines

// master and nocache-cleanup share ~99% of history).
completedRefs := planner.CopyRefHashMap(p.TargetRefs)

// calibratedBytesPerObject tracks the per-object byte estimate
// updated from observed rejected pushes. Starts at the static
// default (which under-counts blob-heavy repos) and ratchets up as
// 413s reveal that the real bytes/object are much higher than 750.
// Used in checkPackSizeAndSubdivide so subsequent sub-pack fetches
// are pre-emptively split when the calibrated estimate exceeds
// p.TargetMaxPack — saving an entire ~limit-sized wasted upload.
calibratedBytesPerObject := int64(estimatedBytesPerObject)

for _, batch := range batches {
    if batch.subsumed {
        cmds := []gitproto.PushCommand{{
            // code omitted
        }
    }
}