# Document batched planning backend test

`23cfb4a`→[main](/content/gh/entireio/git-sync/commits/main/index.html)·

Soph·3mo ago·4 files·+8 added/-135 removed

## Sessions

d9d1a8661781View transcript

## Changes

4

- Dautoresearch.jsonl-4
- Dautoresearch.md-38
- Dautoresearch.sh-93
- docs

- Mtesting.md+8

```
1
2
3
4

{"type":"config","name":"Optimize syncer benchmark execution overhead","metricName":"total_ms","metricUnit":"ms","bestDirection":"lower"}
{"run":1,"commit":"ecd3908","metric":139.968,"metrics":{},"status":"checks_failed","description":"Baseline autoresearch setup run for syncer benchmark suite; benchmark succeeded but correctness checks timed out unexpectedly, so baseline cannot be kept until checks are made reliable.","timestamp":1775996642163,"segment":0,"confidence":null,"iterationTokens":85}
{"run":2,"commit":"ecd3908","metric":145.605,"metrics":{},"status":"checks_failed","description":"Re-ran baseline after making checks script exec-friendly; benchmark remained healthy but the autoresearch checks runner still timed out despite the same script finishing quickly outside run_experiment, so checks integration appears unreliable for this repo.","timestamp":1775996755881,"segment":0,"confidence":null,"iterationTokens":1370}
{"run":3,"commit":"5463a6d","metric":134.965,"metrics":{},"status":"keep","description":"Established autoresearch baseline on the rewrite branch using the median of five syncer benchmark runs; removed the flaky checks hook and documented manual test verification instead so the loop can proceed with valid benchmark data.","timestamp":1775996800100,"segment":0,"confidence":1,"iterationTokens":1214}
```

# Autoresearch: syncer benchmark execution overhead

## Objective
Optimize the rewrite branch's end-to-end sync execution overhead as measured by the in-process `internal/syncer` benchmarks. The target workload is the trio of execution-path benchmarks that exercise bootstrap relay, incremental relay, and materialized fallback. We care about lowering aggregate benchmark time without breaking behavior or widening the product surface.

## Metrics
- **Primary**: `total_ms` (ms, lower is better) — median of repeated benchmark runs, summing the three `internal/syncer` benchmark `ns/op` values and converting to milliseconds
- **Secondary**: `bootstrap_ms`, `incremental_ms`, `materialized_ms`, `bootstrap_alloc_b`, `incremental_alloc_b`, `materialized_alloc_b`, `go_test_elapsed_s` — to detect path-specific regressions and allocation tradeoffs

## How to Run
`./autoresearch.sh` — outputs structured `METRIC name=value` lines.

## Files in Scope
- `internal/syncer/` — top-level orchestration on the hot benchmarked path
- `internal/strategy/bootstrap/` — bootstrap relay path used by benchmark coverage
- `internal/strategy/incremental/` — incremental relay path used by benchmark coverage
- `internal/strategy/materialized/` — fallback path if shared orchestration changes affect it
- `internal/gitproto/` — fetch/push helpers and pack wrappers used by the strategies
- `internal/planner/` — planning helpers if they materially affect sync benchmarks
- `internal/convert/` — conversion helpers on the execution path
- `cmd/git-sync-bench/` — instrumentation only if needed for diagnosis

## Off Limits
- Public CLI behavior and JSON output contracts unless strictly performance-neutral
- New third-party dependencies
- Unrelated rewrite cleanup not justified by benchmark wins
- Default branch comparison work; this session targets the rewrite branch only

## Constraints
- Keep user-visible behavior intact
- Prefer simpler code when benchmark results are equal
- Run `go test ./...` manually before keeping non-trivial changes; the autoresearch checks hook appears unreliable in this repo and was timing out despite fast local completion
- Benchmark noise is expected; confirm marginal wins before keeping them

## What's Been Tried
- Two initial baseline attempts produced valid benchmark metrics but `autoresearch.checks.sh` timed out under `run_experiment` even though `go test ./...` completed quickly when run directly. Treat the hook as unreliable here and run tests manually before keeping meaningful code changes.
- Initial code reading suggests avoidable map/slice conversion churn on the relay paths (`planner.DesiredSubset` -> `convert.DesiredRefs` -> `gitproto.ToPushCommands`) may be one of the few low-risk hot-path opportunities worth testing first.
- The benchmarked execution paths are already fairly small, so broad architectural rewrites are less likely to pay off than removing repeated tiny allocations or duplicated work on the strategy path.

```bash
# Autoresearch script for running benchmarks

#!/bin/bash
set -euo pipefail

# Fast pre-check: compile the benchmarked package before spending time running benches.
go test -run '^$' ./internal/syncer >/dev/null

runs=5
pattern='BenchmarkRun(BootstrapEmptyTarget|IncrementalRelay|MaterializedFallback)$'

parse_metrics() {
  awk '
    function sort_num(a, n,    i, j, tmp) {
      for (i = 1; i <= n; i++) {
        for (j = i + 1; j <= n; j++) {
          if (a[j] < a[i]) {
            tmp = a[i]
            a[i] = a[j]
            a[j] = tmp
          }
        }
      }
    }
    function median(a, n,    mid) {
      sort_num(a, n)
      mid = int((n + 1) / 2)
      if (n % 2 == 1) return a[mid]
      return (a[mid] + a[mid + 1]) / 2
    }
    /BenchmarkRunBootstrapEmptyTarget-[0-9]+/ {
      bootstrap_ns = $3 + 0
      bootstrap_alloc_b = $5 + 0
    }
    /BenchmarkRunIncrementalRelay-[0-9]+/ {
      incremental_ns = $3 + 0
      incremental_alloc_b = $5 + 0
    }
    /BenchmarkRunMaterializedFallback-[0-9]+/ {
      materialized_ns = $3 + 0
      materialized_alloc_b = $5 + 0
    }
    /^ok[[:space:]]+github.com\/soph\/git-sync\/internal\/syncer[[:space:]]+[0-9.]+s$/ {
      if (bootstrap_ns > 0 && incremental_ns > 0 && materialized_ns > 0) {
        run_count++
        total_ns[run_count] = bootstrap_ns + incremental_ns + materialized_ns
        wall_s[run_count] = substr($3, 1, length($3) - 1) + 0
        run_bootstrap_ns[run_count] = bootstrap_ns
        run_incremental_ns[run_count] = incremental_ns
        run_materialized_ns[run_count] = materialized_ns
        run_bootstrap_alloc_b[run_count] = bootstrap_alloc_b
        run_incremental_alloc_b[run_count] = incremental_alloc_b
        run_materialized_alloc_b[run_count] = materialized_alloc_b
      }
      bootstrap_ns = incremental_ns = materialized_ns = 0
      bootstrap_alloc_b = incremental_alloc_b = materialized_alloc_b = 0
    }
    END {
      if (run_count == 0) {
        print "failed_to_parse=1" > "/dev/stderr"
        exit 2
      }
      median_total_ns = median(total_ns, run_count)
      for (i = 1; i <= run_count; i++) {
        if (total_ns[i] == median_total_ns) {
          median_idx = i
          break
        }
      }
      if (median_idx == 0) median_idx = int((run_count + 1) / 2)
      printf "total_ms=%.3f\n", median_total_ns / 1000000
      printf "bootstrap_ms=%.3f\n", run_bootstrap_ns[median_idx] / 1000000
      printf "incremental_ms=%.3f\n", run_incremental_ns[median_idx] / 1000000
      printf "materialized_ms=%.3f\n", run_materialized_ns[median_idx] / 1000000
      printf "bootstrap_alloc_b=%d\n", run_bootstrap_alloc_b[median_idx]
      printf "incremental_alloc_b=%d\n", run_incremental_alloc_b[median_idx]
      printf "materialized_alloc_b=%d\n", run_materialized_alloc_b[median_idx]
      printf "go_test_elapsed_s=%.3f\n", median(wall_s, run_count)
    }
  ' "$1"
}

tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT

for _ in $(seq 1 "$runs"); do
  go test -run '^$' -bench "$pattern" -benchmem ./internal/syncer >>"$tmp"
done

metrics=$(parse_metrics "$tmp")
while IFS= read -r line; do
  key=${line%%=*}
  value=${line#*=}
  printf 'METRIC %s=%s\n' "$key" "$value"
done <<< "$metrics"
```

Batch-planning sensitivity experiment for `#14`:

```bash
env GOCACHE=/tmp/go-build GITSYNC_E2E_GIT_HTTP_BACKEND=1 go test ./internal/syncer -run TestBootstrap_GitHTTPBackendBatchedPlanningTracksBatchLimit -v
```

That test uses a real `git-http-backend` source/target pair and checks that a smaller `--batch-max-pack-bytes` planning limit produces at least as many planned checkpoints as a larger one, while still planning to the branch tip.

## Live Linux Smokes

Optional live Linux bootstrap smoke:
```