e2e: rerun failed tests once to absorb transient agent flakiness · Entire

e2e: rerun failed tests once to absorb transient agent flakiness

c7e3126→main·

toothbrush·2w ago·1 file·+11 added/-3 removed

Real-agent E2E ran single-shot (-count=1, no retries), so one non-deterministic agent response turned the whole agent job red. Add gotestsum --rerun-fails=1 so only the failed tests retry once; a test that passes on retry reports green. --rerun-fails-max-failures=10 default leaves broadly-broken runs untouched so real regressions still surface. Canary/roger-roger deterministic tasks are left strict.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

Sessions

e11d68fd1291View transcript

[?
Make E2E Tests Reliable on MainClaude Code·Opus 4.8[1m]·2 steps](/content/gh/entireio/cli/session/cb256a5c-45a5-4513-be01-ea2f88739aec#timeline-e11d68fd1291/index.html)

Changes

1

30 unmodified lines

31
32
33
34
34
35
36
37
38
39
40
41
42
43
44
36
37
45
46
47
48

30 unmodified lines

# triggers a Go toolchain panic on Windows ARM64 in modload/search.go
# (slice bounds out of range) and a GetFileInformationByHandleEx error
# with relative paths.
gotestsum --format dots --jsonfile "$E2E_ARTIFACT_DIR/test-events.json" -- 
#
# --rerun-fails retries only the tests that failed (one extra attempt) to
# absorb transient real-agent non-determinism. It requires --packages (so the
# package list can no longer be a trailing positional arg) and -count=1. The
# default --rerun-fails-max-failures=10 means a broadly-broken run (>10
# failures) is reported as-is rather than retried, so a genuine regression
# still surfaces. Do NOT add reruns to the deterministic canary/roger-roger
# tasks — there a failure is always a real bug and must not be masked.
gotestsum --format dots --rerun-fails=1 --packages=./e2e/tests \
  --jsonfile "$E2E_ARTIFACT_DIR/test-events.json" -- \
  -tags=e2e -count=1 -timeout=30m \
  ${filter:+-run "$filter"} \
  ./e2e/tests || rc=$?
  ${filter:+-run "$filter"} || rc=$?

go run ./e2e/cmd/testreport -color -o "$E2E_ARTIFACT_DIR/report.txt" "$E2E_ARTIFACT_DIR/test-events.json"
echo ""

Mmise-tasks/test/e2e/_default+11/-3