checkpoint: pin straddling-boundary rounding and refresh compact-transcript docs · Entire
checkpoint: pin straddling-boundary rounding and refresh compact-transcript docs
110390a→main·
Soph·2w ago·5 files·+81 added/-14 removed
P2: when StartLine falls between same-ID streaming assistant fragments, compaction merges them into one line that no integer offset can split. Document that the boundary rounds toward inclusion — the merged line stays in the slice, so fullCompactLines[boundary:] never drops this checkpoint's content but its head may repeat up to one merged line from the previous checkpoint. Add a regression test pinning this deterministic behavior, and note the bounded head overlap on CompactTranscriptStart so downstream segmenters tolerate it.
P3: refresh stale docs that still described transcript.jsonl as checkpoint- scoped/pre-sliced (metadata.go field + directory-layout comments, the sessions-and-checkpoints architecture doc, and CLAUDE.md). They now state that transcript.jsonl holds the full compacted session and that consumers slice at compact_transcript_start (compact-output coordinates, distinct from the raw checkpoint_transcript_start), with nil meaning a legacy delta-only file.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Sessions
a23237182e15View transcript
Changes
5
MCLAUDE.md+1/-1
api/checkpoint
Mmetadata.go+9/-3
cmd/entire/cli/transcript/compact
Mcompact.go+11/-4
Mcompactfull_test.go+42
docs/architecture
- Msessions-and-checkpoints.md+18/-6
496 unmodified lines
497
498
499
500
500
501
502
503
496 unmodified lines
- **Worktree-specific branches** - each git worktree gets its own shadow branch namespace, preventing conflicts
- **Supports multiple concurrent sessions** - checkpoints from different sessions in the same directory interleave on the same shadow branch
- Condenses session logs to permanent `entire/checkpoints/v1` branch on user commits
- Each committed session stores the raw transcript (`full.jsonl`, read by CLI rewind/resume/explain) plus a best-effort compact transcript (`transcript.jsonl`, generated via `transcript/compact` and pre-sliced to the checkpoint's `checkpoint_transcript_start`). Both are pushed with the v1 branch. The root `metadata.json` `sessions[].transcript` pointer keeps targeting `full.jsonl`; when the compact transcript was generated the session entry also carries a `compact_transcript` path pointing at `transcript.jsonl` (omitted otherwise) so external readers can locate it next to `full.jsonl`.
- Each committed session stores the raw transcript (`full.jsonl`, read by CLI rewind/resume/explain) plus a best-effort compact transcript (`transcript.jsonl`, generated via `transcript/compact`). Like `full.jsonl`, `transcript.jsonl` stores the **full compacted session** on every checkpoint (via `compact.FullWithBoundary`), so each checkpoint is self-contained and the session survives a mid-history checkpoint being lost/reverted/rebased. This checkpoint's slice begins at the session metadata's `compact_transcript_start` (a line offset in compact-output coordinates, distinct from `checkpoint_transcript_start` which indexes raw `full.jsonl` lines); a nil/absent marker means a legacy delta-only `transcript.jsonl` (read from line 0). The marker rounds toward inclusion when a streaming message straddles the boundary, so the slice never drops this checkpoint's content but may repeat ≤1 merged line at its head. Both files are pushed with the v1 branch. The root `metadata.json` `sessions[].transcript` pointer keeps targeting `full.jsonl`; when the compact transcript was generated the session entry also carries a `compact_transcript` path pointing at `transcript.jsonl` (omitted otherwise) so external readers can locate it next to `full.jsonl`.
- Uses the `post-rewrite` Git hook to keep local session linkage aligned after amend/rebase rewrites
- Builds git trees in-memory using go-git plumbing APIs
- Rewind restores files from shadow branch commit tree (does not use `git reset`)
MCLAUDE.md+1/-1
328 unmodified lines
329
330
331
332
332
333
334
335
336
337
338
339
76 unmodified lines
416
417
418
415
419
420
421
422
423
424
11 unmodified lines
436
437
438
433
439
440
441
442
328 unmodified lines
// CompactTranscriptStart is the line offset in the compact transcript.jsonl
// at which this checkpoint's data begins. transcript.jsonl stores the full
// compacted session (each checkpoint is self-contained), so readers segment
// this checkpoint's slice as compactLines[CompactTranscriptStart:].
// this checkpoint's slice as compactLines[CompactTranscriptStart:]. The slice
// never drops this checkpoint's content, but its first line may repeat up to
// one compact line that began in the previous checkpoint (when a streaming
// message straddles the boundary and compaction merges it into one line), so
// segmenters must tolerate a bounded head overlap.
//
// A nil pointer marks a legacy checkpoint whose transcript.jsonl holds only
// this checkpoint's delta (CLI versions before the full-compact-transcript
76 unmodified lines
Transcript string `json:"transcript,omitempty"`
// CompactTranscript points at the compact transcript.jsonl when one was
// generated alongside full.jsonl. Omitted otherwise (non-compactable,
// empty, or oversized transcripts, and older CLI versions).
// empty, or oversized transcripts, and older CLI versions). transcript.jsonl
// holds the full compacted session; this checkpoint's slice begins at the
// session metadata's compact_transcript_start (see Metadata.CompactTranscriptStart).
CompactTranscript string `json:"compact_transcript,omitempty"`
ContentHash string `json:"content_hash,omitempty"`
Prompt string `json:"prompt"`
11 unmodified lines
// ├── 1/ # First session
// │ ├── metadata.json # Session-specific Metadata
// │ ├── full.jsonl # Raw agent transcript
// │ ├── transcript.jsonl # Compact transcript scoped to this checkpoint
// │ ├── transcript.jsonl # Full compacted session (slice at compact_transcript_start)
// │ ├── prompt.txt
// │ └── content_hash.txt
// ├── 2/ # Second session
Mapi/checkpoint/metadata.go+9/-3
134 unmodified lines
135
136
137
138
139
140
141
138
139
140
141
142
143
144
145
146
147
148
149
150
151
134 unmodified lines
// StartLine). This reuses each format's existing, independently-tested slicing
// behavior — line offsets for JSONL/Copilot/Droid/pi, message-index for
// OpenCode/Gemini, response-item index for Codex — rather than threading source
// indices through every emitter. The only imprecision is an off-by-one when a
// single logical message (a streaming assistant message split across the
// boundary, or a tool_result that also carries text) straddles the exact
// StartLine; the result is deterministic and harmless for segmentation.
// indices through every emitter.
//
// A single compact line can carry content from both sides of StartLine when its
// source straddles the boundary — most notably same-ID streaming assistant
// fragments, which compaction merges into one line. No integer line offset can
// split within a line, so the boundary rounds toward inclusion: such a merged
// line is counted as part of this checkpoint's slice. fullCompactLines[boundary:]
// therefore never drops this checkpoint's content, but its first line may repeat
// up to one merged line that began in the previous checkpoint. Downstream
// segmenters must tolerate this bounded head overlap. See the straddle case in
// TestFullWithBoundary_StraddlingAssistantFragments_RoundsToInclusion.
func FullWithBoundary(redacted redact.RedactedBytes, opts MetadataFields) (full []byte, boundary int, err error) {
fullOpts := opts
fullOpts.StartLine = 0
Mcmd/entire/cli/transcript/compact/compact.go+11/-4
97 unmodified lines
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
97 unmodified lines
}
}
// TestFullWithBoundary_StraddlingAssistantFragments_RoundsToInclusion pins the
// documented behavior when StartLine falls between two same-ID streaming
// assistant fragments: compaction merges them into one line, which no integer
// boundary can split. The boundary rounds to inclusion (0 here), so the merged
// line — carrying both the pre-start and post-start fragment — stays in the
// slice. This never drops this checkpoint's content (FRAG_B), at the cost of
// the slice head repeating one merged line from the previous checkpoint (FRAG_A).
func TestFullWithBoundary_StraddlingAssistantFragments_RoundsToInclusion(t *testing.T) {
t.Parallel()
input := redact.AlreadyRedacted([]byte(
`{"type":"assistant","timestamp":"t0","message":{"id":"msg_1","content":[{"type":"text","text":"FRAG_A"}]}}
{"type":"assistant","timestamp":"t1","message":{"id":"msg_1","content":[{"type":"text","text":"FRAG_B"}]}}
`))
// StartLine=1 lands between the two fragments of the same streaming message.
opts := MetadataFields{Agent: "claude-code", CLIVersion: "0.5.1", StartLine: 1}
full, boundary, err := FullWithBoundary(input, opts)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
// The two fragments merge into a single compact line.
fullLines := nonEmptyLines(full)
if len(fullLines) != 1 {
t.Fatalf("expected 1 merged compact line, got %d:\n%s", len(fullLines), full)
}
// Rounds to inclusion: the merged line stays in the slice.
if boundary != 0 {
t.Fatalf("boundary: got %d, want 0 (merged straddling line included)", boundary)
}
// The slice retains this checkpoint's content (FRAG_B) and, unavoidably, the
// pre-start fragment (FRAG_A) merged into the same line.
slice := strings.Join(fullLines[boundary:], "\n")
if !strings.Contains(slice, "FRAG_B") {
t.Errorf("slice dropped this checkpoint's content FRAG_B:\n%s", slice)
}
if !strings.Contains(slice, "FRAG_A") {
t.Errorf("expected merged line to retain FRAG_A (inclusive rounding):\n%s", slice)
}
}
func TestCompactFull_GeminiIndexFormat_Boundary(t *testing.T) {
t.Parallel()
Mcmd/entire/cli/transcript/compact/compactfull_test.go+42
195 unmodified lines
196
197
198
199
199
200
201
202
5 unmodified lines
208
209
210
211
212
213
214
215
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
195 unmodified lines
├── 0/ # First session (0-based indexing)
│ ├── metadata.json # Session-specific Metadata
│ ├── full.jsonl # Raw agent transcript (CLI rewind/resume/explain)
│ ├── transcript.jsonl # Compact transcript, scoped to this checkpoint
│ ├── transcript.jsonl # Full compacted session (slice at compact_transcript_start)
│ ├── prompt.txt # Checkpoint-scoped user prompts
│ └── content_hash.txt # sha256 of full.jsonl (dedup short-circuit)
├── 1/ # Second session
5 unmodified lines
**Compact transcript (`transcript.jsonl`):** generated best-effort from
`full.jsonl` via `transcript/compact` on every committed write and on
transcript replacement during finalization. Unlike `full.jsonl` (the
cumulative session transcript, scoped at read time via
`checkpoint_transcript_start`), `transcript.jsonl` is pre-sliced to the
checkpoint's own portion (`compact.Compact` is called with
`StartLine = checkpoint_transcript_start`), so it needs no offset to consume.
transcript replacement during finalization. Like `full.jsonl`, it stores the
**full compacted session** on every checkpoint (via `compact.FullWithBoundary`),
so each checkpoint is self-contained — the session is reconstructable from any
single surviving checkpoint, robust to a mid-history checkpoint being lost,
reverted, or dropped during a rebase. This checkpoint's slice begins at the
session metadata's `compact_transcript_start` (a line offset into
`transcript.jsonl`, in compact-output coordinates — distinct from
`checkpoint_transcript_start`, which indexes raw `full.jsonl` lines).
Consumers segment this checkpoint's content as `compactLines[compact_transcript_start:]`.
The marker rounds toward inclusion when a streaming message straddles the
boundary (compaction merges same-ID fragments into one line that cannot be
split), so the slice never drops this checkpoint's content but its head may
repeat at most one merged line from the previous checkpoint — segmenters must
tolerate that bounded overlap. A nil/absent `compact_transcript_start` marks a
legacy checkpoint whose `transcript.jsonl` holds only its own delta (pre-change
CLI versions); read it as-is from line 0.
It is written into the checkpoint tree and pushed alongside `full.jsonl`. The
root `metadata.json` `sessions[].transcript` pointer keeps targeting
`full.jsonl`; when a compact transcript was generated the session entry also