checkpoint: drop stale compact on OPF regeneration failure; document 50MB cap · Entire
checkpoint: drop stale compact on OPF regeneration failure; document 50MB cap
05e1570→main·
Soph·2w ago·4 files·+85 added/-15 removed
Address PR review (Copilot #3) and the trail finding.
OPF stale compact (#3): in replaceTranscript, when compact regeneration yields nothing (failure, empty, or oversized), drop the prior transcript.jsonl and clear CompactTranscriptStart instead of leaving a stale, less-redacted compact on the branch with a marker pointing at content that no longer matches the re-redacted full.jsonl. setCompactTranscriptStart now takes *int so it can clear the marker. Regression test added.
Oversized-drop (#1 / trail finding): document the known limitation — the compact transcript is not chunked, so output exceeding the 50MB blob cap is skipped and a very long session may lack transcript.jsonl on some checkpoints; full.jsonl stays authoritative and the compact is regenerable. Updated the architecture doc (also correcting the now-stale "finalization keeps the previous transcript.jsonl" clause) and CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Sessions
bd7061c21a95View transcript
[?
Compact Transcript Storage and Boundary MarkersClaude Code·Opus 4.8[1m]·3 steps](/content/gh/entireio/cli/session/78b0fe12-1b0c-46e0-ae16-2534880690c7#timeline-bd7061c21a95/index.html)
Changes
4
MCLAUDE.md+1/-1
cmd/entire/cli/checkpoint
Mpersistent.go+21/-12
Mpersistent_compact_transcript_test.go+56
docs/architecture
Msessions-and-checkpoints.md+7/-2
496 unmodified lines
497
498
499
500
500
501
502
503
496 unmodified lines
- **Worktree-specific branches** - each git worktree gets its own shadow branch namespace, preventing conflicts
- **Supports multiple concurrent sessions** - checkpoints from different sessions in the same directory interleave on the same shadow branch
- Condenses session logs to permanent `entire/checkpoints/v1` branch on user commits
- Each committed session stores the raw transcript (`full.jsonl`, read by CLI rewind/resume/explain) plus a best-effort compact transcript (`transcript.jsonl`, generated via `transcript/compact`). Like `full.jsonl`, `transcript.jsonl` stores the **full compacted session** on every checkpoint (via `compact.FullWithBoundary`), so each checkpoint is self-contained and the session survives a mid-history checkpoint being lost/reverted/rebased. This checkpoint's slice begins at the session metadata's `compact_transcript_start` (a line offset in compact-output coordinates, distinct from `checkpoint_transcript_start` which indexes raw `full.jsonl` lines); a nil/absent marker means a legacy delta-only `transcript.jsonl` (read from line 0). The marker rounds toward inclusion when a streaming message straddles the boundary, so the slice never drops this checkpoint's content but may repeat ≤1 merged line at its head. Both files are pushed with the v1 branch. The root `metadata.json` `sessions[].transcript` pointer keeps targeting `full.jsonl`; when the compact transcript was generated the session entry also carries a `compact_transcript` path pointing at `transcript.jsonl` (omitted otherwise) so external readers can locate it next to `full.jsonl`.
- Each committed session stores the raw transcript (`full.jsonl`, read by CLI rewind/resume/explain) plus a best-effort compact transcript (`transcript.jsonl`, generated via `transcript/compact`). Like `full.jsonl`, `transcript.jsonl` stores the **full compacted session** on every checkpoint (via `compact.FullWithBoundary`), so each checkpoint is self-contained and the session survives a mid-history checkpoint being lost/reverted/rebased. This checkpoint's slice begins at the session metadata's `compact_transcript_start` (a line offset in compact-output coordinates, distinct from `checkpoint_transcript_start` which indexes raw `full.jsonl` lines); a nil/absent marker means a legacy delta-only `transcript.jsonl` (read from line 0). The marker rounds toward inclusion when a streaming message straddles the boundary, so the slice never drops this checkpoint's content but may repeat ≤1 merged line at its head. Compact generation is best-effort and is skipped when the compacted output exceeds the 50MB blob cap (unlike `full.jsonl`, `transcript.jsonl` is not chunked — `full.jsonl` stays authoritative and the compact is regenerable); in the OPF finalize rewrite a failed/skipped regeneration drops the prior `transcript.jsonl` and clears the marker rather than shipping a stale, less-redacted compact. Both files are pushed with the v1 branch. The root `metadata.json` `sessions[].transcript` pointer keeps targeting `full.jsonl`; when the compact transcript was generated the session entry also carries a `compact_transcript` path pointing at `transcript.jsonl` (omitted otherwise) so external readers can locate it next to `full.jsonl`.
- Uses the `post-rewrite` Git hook to keep local session linkage aligned after amend/rebase rewrites
- Builds git trees in-memory using go-git plumbing APIs
- Rewind restores files from shadow branch commit tree (does not use `git reset`)
MCLAUDE.md+1/-1
1816 unmodified lines
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1834
1835
1836
1842
1843
1844
1845
1846
1838
1847
1848
1849
1850
1816 unmodified lines
}
compactStart := s.writeCompactTranscript(ctx, agentType, startLine, compactBytes, sessionPath, entries)
// Keep the session metadata's compact-transcript marker consistent with the
// regenerated transcript.jsonl. The OPF re-redaction can write a compact
// transcript the initial write lacked, or (in principle) shift line counts,
// so re-record the boundary whenever one was produced.
if compactStart != nil {
if err := s.setCompactTranscriptStart(sessionPath, *compactStart, entries); err != nil {
return fmt.Errorf("failed to update compact transcript start: %w", err)
}
// If regeneration produced no compact transcript (failure, empty, or
// oversized), drop any stale transcript.jsonl carried over from the prior
// write rather than shipping it. In the OPF rewrite path the stale file is a
// less-redacted compact (it predates the 8th-layer re-redaction), and its
// CompactTranscriptStart would point at content that no longer matches the
// re-redacted full transcript. The caller re-derives the root summary's
// compact_transcript pointer from the (now absent) tree entry.
if compactStart == nil {
delete(entries, sessionPath+paths.CompactTranscriptFileName)
}
// Keep the session metadata's marker consistent with the regenerated
// transcript.jsonl: record the new boundary when one was produced, or clear
// it (nil) when the compact transcript was dropped above.
if err := s.setCompactTranscriptStart(sessionPath, compactStart, entries); err != nil {
return fmt.Errorf("failed to update compact transcript start: %w", err)
}
return nil
}
// setCompactTranscriptStart records CompactTranscriptStart in the session
// metadata. Used by the OPF rewrite path so the finalized session metadata
// reflects the regenerated compact transcript.
func (s *GitStore) setCompactTranscriptStart(sessionPath string, start int, entries map[string]object.TreeEntry) error {
// metadata, or clears it when start is nil (no compact transcript present).
// Used by the OPF rewrite path so the finalized session metadata reflects the
// regenerated compact transcript.
func (s *GitStore) setCompactTranscriptStart(sessionPath string, start *int, entries map[string]object.TreeEntry) error {
return s.updateSessionMetadata(sessionPath, entries, func(metadata *Metadata) {
metadata.CompactTranscriptStart = &start
metadata.CompactTranscriptStart = start
})
}
Mcmd/entire/cli/checkpoint/persistent.go+21/-12
353 unmodified lines
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
353 unmodified lines
}
}
// TestUpdateCommitted_DropsStaleCompactWhenRegenerationProducesNone guards the
// OPF/finalize rewrite: if the re-redacted transcript no longer yields a compact
// transcript, the stale transcript.jsonl from the initial write must be removed
// (not shipped as a less-redacted artifact) and its marker cleared, rather than
// left pointing at content that no longer matches the re-redacted full.jsonl.
func TestUpdateCommitted_DropsStaleCompactWhenRegenerationProducesNone(t *testing.T) {
t.Parallel()
repo, _ := setupTestRepo(t)
store := NewGitStore(repo, DefaultV1Refs())
cpID := id.MustCheckpointID("a7b8c9d0e1f2")
// Initial write: compactable transcript → transcript.jsonl + marker present.
if err := store.Write(context.Background(), Session{
CheckpointID: cpID,
SessionID: "session-001",
Strategy: "manual-commit",
Transcript: redact.AlreadyRedacted(claudeStyleTranscript()),
Agent: agent.AgentTypeClaudeCode,
CheckpointTranscriptStart: 2,
AuthorName: "Test",
AuthorEmail: "test@test.com",
}); err != nil {
t.Fatalf("WriteCommitted() error = %v", err)
}
sessionPath := cpID.Path() + "/0/"
if _, ok := readBranchFile(t, store, sessionPath+paths.CompactTranscriptFileName); !ok {
t.Fatal("precondition: transcript.jsonl missing after initial write")
}
if _, ok := readSessionMetadata(t, repo, cpID).GetCompactTranscriptStart(); !ok {
t.Fatal("precondition: compact_transcript_start not recorded after initial write")
}
// Finalize with a non-compactable transcript: regeneration yields nothing.
if err := store.Write(context.Background(), SessionTranscript{
CheckpointID: cpID,
SessionID: "session-001",
Transcript: redact.AlreadyRedacted([]byte("not json at all\nstill not json\n")),
Agent: agent.AgentTypeClaudeCode,
}); err != nil {
t.Fatalf("UpdateCommitted() error = %v", err)
}
// Stale compact transcript dropped from the tree.
if _, ok := readBranchFile(t, store, sessionPath+paths.CompactTranscriptFileName); ok {
t.Error("stale transcript.jsonl left in tree after regeneration produced none")
}
// Root summary pointer cleared.
if got := readSummaryFromBranch(t, repo, cpID).Sessions[0].CompactTranscript; got != "" {
t.Errorf("sessions[0].compact_transcript = %q, want empty", got)
}
// Session metadata marker cleared.
if offset, ok := readSessionMetadata(t, repo, cpID).GetCompactTranscriptStart(); ok {
t.Errorf("compact_transcript_start still set (%d) after stale compact dropped", offset)
}
}
func TestUpdateCommitted_RegeneratesCompactTranscript(t *testing.T) {
t.Parallel()
repo, _ := setupTestRepo(t)