Compact Transcript Storage and Boundary Markers · Entire

Based on my investigation of the codebase, here are the precise findings about how session logs/transcripts are stored in the entire/checkpoints/v1 branch:

Answer: NEITHER format stores ONLY deltas — BOTH store the FULL transcript with per-checkpoint start markers

Data Model

Metadata Structure (api/checkpoint/metadata.go lines 290-396):

• CheckpointSummary (root-level metadata.json) contains sessions: []SessionFilePaths

• Each SessionFilePaths has three transcript pointers:

  • Transcript → full.jsonl (raw agent transcript, stored once per session)
  • CompactTranscript → transcript.jsonl (optional, derived/indexed)
  • Metadata → session metadata.json (per-session, line 388-395)

• Session-level Metadata includes:

  • CheckpointTranscriptStart: int (line 324) — line offset where THIS checkpoint's data began
  • TranscriptLinesAtStart: int (line 327) — deprecated alias for backward compat

Directory Structure (line 403-414):

<checkpoint-id[:2]>/<checkpoint-id[2:]>/
├── metadata.json         # CheckpointSummary (root aggregate)
├── 1/                    # First session
│   ├── metadata.json     # Session Metadata (includes CheckpointTranscriptStart)
│   ├── full.jsonl        # Raw agent transcript (FULL session)
│   ├── transcript.jsonl  # Compact transcript (SCOPED to checkpoint)
│   └── prompt.txt
├── 2/                    # Second session...

FORMAT 1: full.jsonl — FULL TRANSCRIPT + START OFFSET

File Storage (cmd/entire/cli/checkpoint/persistent.go lines 744-844):

• Contains: The COMPLETE session transcript from beginning to end

• Chunked: If >MaxChunkSize, splits into full.jsonl.000, full.jsonl.001, etc. (line 781, 793-806)

• Written: Same for every checkpoint that includes this session (line 749-771)

• Start Position: Metadata stores CheckpointTranscriptStart separately — a LINE INDEX (0-indexed)

• Line 324: CheckpointTranscriptStart int

  • Line 100 (WriteOptions): "Transcript line offset at start of this checkpoint's data"

Example (from manual_commit_condensation.go lines 282, 456-457):

1
2
3
4
5
6
7
8
9
10
11
12

writeOpts := cpkg.WriteOptions{
    CheckpointTranscriptStart: state.CheckpointTranscriptStart,  // Line 282
    Transcript: redactedTranscript,  // FULL transcript
    ...
}

// Session-level metadata writes:
sessionMetadata := Metadata{
    CheckpointTranscriptStart: opts.CheckpointTranscriptStart,  // Line 456
    TranscriptLinesAtStart: opts.CheckpointTranscriptStart,     // Deprecated, line 457
    ...
}

Key Code (persistent.go lines 867-871):

1
2
3
4
5
6
7
8

compacted, err := transcriptcompact.Compact(
    redact.AlreadyRedacted(transcriptBytes),  // FULL transcript bytes
    transcriptcompact.MetadataFields{
        Agent: compactAgentName(agentType),
        CLIVersion: versioninfo.Version,
        StartLine: startLine,  // THIS is where checkpoint starts
    },
)

FORMAT 2: transcript.jsonl — SCOPED DELTA (start-to-end of THIS checkpoint)

File Storage (cmd/entire/cli/checkpoint/persistent.go lines 856-908):

• Contains: Compact format, sliced to only the lines from startLine onwards
• Purpose: "derived data" (line 859) for efficient reading
• Optional: Best-effort; skipped if generation fails, result is empty, or exceeds MaxChunkSize (lines 872-892)

How Scoping Works (cmd/entire/cli/transcript/compact/compact.go lines 71-120):

1
2
3
4
5
6
7
8

func Compact(redacted redact.RedactedBytes, opts MetadataFields) ([]byte, error) {
    // ...
    truncated := transcript.SliceFromLine(content, opts.StartLine)  // Line 106
    if truncated == nil {
        truncated = []byte{}
    }
    // ... format-specific compact functions ...
}

SliceFromLine Implementation (transcript/parse.go lines 110-143):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// Returns the content starting from line number `startLine` (0-indexed).
// This is used to extract only the checkpoint-specific portion of a cumulative transcript.
func SliceFromLine(content []byte, startLine int) []byte {
    if len(content) == 0 || startLine <= 0 {
        return content  // startLine=0 → return full
    }
    // Find byte offset of startLine's beginning
    lineCount := 0
    offset := 0
    for i, b := range content {
        if b == '\n' {
            lineCount++
            if lineCount == startLine {
                offset = i + 1
                break
            }
        }
    }
    if lineCount < startLine {
        return nil
    }
    return content[offset:]  // Return from startLine to EOF
}

What CheckpointTranscriptStart Means

Definition (cmd/entire/cli/session/state.go lines 188-192):

1
2
3
4
5

// CheckpointTranscriptStart is the transcript line offset where the current
// checkpoint cycle began. Set to 0 at session start, updated to current
// transcript length after each condensation. Used to scope the transcript
// for checkpoint condensation: "everything since last checkpoint".
CheckpointTranscriptStart int `json:"checkpoint_transcript_start,omitempty"`

Lifecycle (from code exploration):

1. Initialized to 0 at session start
2. After each checkpoint: Set to total transcript lines at condensation time (manual_commit_hooks.go line 1411)
3. Used for scoping: When writing compact transcript, SliceFromLine(fullTranscript, CheckpointTranscriptStart) extracts only new lines since the last checkpoint
4. Example from manual_commit_condensation.go line 516:

1
2
3
4

scopedTranscript = transcript.SliceFromLine(
    transcriptBytes,
    state.CheckpointTranscriptStart  // Skip everything before this line
)

Checkpoint Sharing vs. Duplication

Per Checkpoint, Per Session:

• Each <checkpoint_id>/<session_num>/ subdirectory gets its OWN full.jsonl
• Multiple sessions in one checkpoint → multiple full.jsonl files (one per session)
• Duplication is INTENTIONAL: Each checkpoint is self-contained (readable without other checkpoints)

Content Deduplication:

• Git object store deduplicates identical blobs via content hashing
• content_hash.txt stores sha256 of full transcript (line 813: fmt.Sprintf("sha256:%x", sha256.Sum256(transcriptBytes)))
• If two checkpoints have identical transcript ranges, git packs them once

Metadata Pointers

From SessionFilePaths (api/checkpoint/metadata.go lines 385-396):

1
2
3
4
5
6
7

type SessionFilePaths struct {
    Metadata          string `json:"metadata"`              // metadata.json
    Transcript        string `json:"transcript,omitempty"`  // full.jsonl
    CompactTranscript string `json:"compact_transcript,omitempty"` // transcript.jsonl (omitted if not generated)
    ContentHash       string `json:"content_hash,omitempty"`
    Prompt            string `json:"prompt"`
}

Note on line 387-393: The Transcript pointer intentionally stays on full.jsonl (not compact), because CLI read paths (rewind/resume/explain) resolve by filename:

1
2
3
4
5
6

// Transcript points at the raw full.jsonl, which CLI read paths
// (rewind/resume/explain) resolve by filename.
Transcript string `json:"transcript,omitempty"`
// CompactTranscript points at the compact transcript.jsonl when one was
// generated alongside full.jsonl. Omitted otherwise.
CompactTranscript string `json:"compact_transcript,omitempty"`

Final Answer

Both formats use the SAME underlying data — full.jsonl stores it completely, transcript.jsonl stores a pre-sliced, pre-compacted view. Neither is a delta; both are complete within their scope (full or scoped). The CheckpointTranscriptStart line offset is the pivot point for scoping. <subagent_tokens>79649</subagent_tokens><tool_uses>25</tool_uses><duration_ms>57782</duration_ms>

2w ago·26.1s

Here's the summary, based on tracing the condensation and storage code.

Short answer

Neither format stores only deltas in the git sense — but they differ in scope. Both are written as complete files per checkpoint; the difference is how much of the session each one contains:

• full.jsonl = the entire session transcript, from line 0 to the end. The checkpoint boundary isn't baked into the file — it's recorded separately as a line offset (checkpoint_transcript_start) in that session's metadata.json.
• transcript.jsonl (the new compact format) = the transcript pre-sliced to checkpoint_transcript_start → EOF, then run through transcript/compact. So it physically contains only the slice belonging to that checkpoint cycle — effectively a per-checkpoint delta, already trimmed and normalized.

So the answer to your question: the full log carries the whole thing + a start position; the compact log carries only the slice (delta) and doesn't need a start marker because the slicing already happened when it was written.

How it's laid out

Each committed checkpoint gets its own directory on entire/checkpoints/v1, with one sub-dir per session:

<cp-id[:2]>/<cp-id[2:]>/
├── metadata.json          # CheckpointSummary, lists sessions[]
└── 1/                     # per-session
    ├── metadata.json      # has checkpoint_transcript_start (line offset)
    ├── full.jsonl         # FULL session transcript (chunked if large)
    ├── transcript.jsonl   # compact, SLICED to [start..EOF] — optional
    └── prompt.txt

metadata.json``sessions[] carries two pointers:

• transcript → always full.jsonl (CLI rewind/resume/explain resolve by this filename)
• compact_transcript → transcript.jsonl, omitted when the compact one wasn't generated

The pivot: checkpoint_transcript_start

It's a 0-indexed line offset (not bytes, not message index), tracked in SessionState:

• starts at 0, and after each condensation is bumped to the transcript's current line count ("everything since the last checkpoint").
• When writing the compact file, transcript.SliceFromLine(fullBytes, CheckpointTranscriptStart) chops off everything before it, then Compact() normalizes the remainder.

So e.g. if a checkpoint's start = 750 on a 1000-line transcript: full.jsonl stores lines 0–999 (with start=750 in metadata), while transcript.jsonl stores a compacted lines-750–999 only.

Two things worth knowing

1. transcript.jsonl is best-effort. It's skipped if compaction fails, produces empty output, or exceeds the max chunk size — so a checkpoint may have full.jsonl but no compact_transcript. full.jsonl is the source of truth.

2. Per-checkpoint copies, deduped by git. Each checkpoint dir gets its own full copy (intentionally self-contained), but full.jsonl is content-hashed (content_hash.txt = sha256) and git's blob store collapses identical ranges, so unchanged history isn't physically duplicated on disk despite appearing in every checkpoint's tree.

Net: the system is full-log + offset as the canonical store, with the compact format being a pre-sliced, pre-compacted convenience view of just that checkpoint's slice.

if we would want to store the transcript.jsonl as full we need a second position marker, right?

2w ago·37.5s·1 call

Confirmed. Yes — you'd need a second marker, and the important part is it can't be the same number you already have.

Why

checkpoint_transcript_start is a line offset into the rawfull.jsonl, and today the slice happens before compaction (compact.go:106, SliceFromLine(content, opts.StartLine) then compact). If you instead stored the full compacted transcript, you'd need a marker pointing at the checkpoint boundary inside the compact file — and that's a different coordinate space, because compaction is not line-preserving. From this same code:

• streaming assistant fragments with the same message ID get merged into one line (N raw → 1 compact),
• tool_result lines get inlined into the preceding assistant's tool_use block (a line disappears),
• whole types are dropped (droppedTypes: progress, file-history-snapshot, queue-operation, system).

So raw line 750 might be compact line ~300, with no fixed ratio. The existing offset is useless for indexing into the compact file.

And it's worse than "just a second int"

For several agents StartLine isn't even a line offset in the first place — it's a message/item index interpreted per-format:

• OpenCode / Gemini (single-object formats): message-index offset
• Codex: response_item index
• pi: handled internally by compactPi

So a "full compact transcript + boundary marker" design would need that marker computed in compact-output coordinates per format, produced by the compaction pass itself (e.g. emit the boundary line index while writing, or count compacted lines for the pre-start portion). You can't derive it after the fact from checkpoint_transcript_start.

The tradeoff

That's essentially why the current design slices-then-compacts instead: storing the delta sidesteps the coordinate-mapping problem entirely — no second marker needed, and each checkpoint's compact file is self-contained. Going to full-compact buys you a single deduplicated compact transcript per session, but costs you (a) a new per-format boundary marker emitted during compaction, and (b) all read paths learning to seek to it. If the goal is dedup, note git already collapses identical blob ranges across checkpoints, so the on-disk win may be smaller than it looks.

Yeah but the issue is as soon as a checkpoint / commit in the middle get lost / reverted or during rebase the whole session is not recoverable. Also the UI needs to read either all (to piece them together) or could in theory just read the last checkpoint in a session and just the positions from the others

2w ago·2m·4 calls

Verified the crux: in manual_commit_condensation.go:220,270, the Transcript written to each checkpoint is sessionData.Transcript — the whole transcript to date, not a slice. Only the compact transcript.jsonl gets sliced via StartLine. That changes your premise a bit, so let me separate what's actually true today from what your argument is really about.

Today, the two formats have different loss profiles

• full.jsonl is already full-per-checkpoint. Every checkpoint re-stores the entire transcript-to-date (redacted). So losing a middle checkpoint loses nothing raw — the latest surviving checkpoint is a superset. The only raw data you can lose is the tail: if the most recent checkpoint(s) get dropped, you fall back to the newest survivor and lose everything after it.
• transcript.jsonl (compact) is the fragile one. It's per-checkpoint slices, so reconstructing the full compact view means concatenating all slices in order. Lose a middle one → a gap. (Though you can always regenerate it from full.jsonl, since that's intact.)

So your recoverability worry is real, but it's specifically a property of the compact/delta format, not the session as a whole. The raw session survives a mid-history loss today.

On rebase specifically

Worth being precise: checkpoints live on the separate permanent entire/checkpoints/v1 branch, not on your working branch. A working-branch rebase doesn't delete v1 commits — what breaks is the linkage (which commit ↔ which checkpoint), which the post-rewrite hook tries to realign. So "the whole session becomes unrecoverable on rebase" overstates it for the current design; the content is still on v1, it's the attachment that's at risk. Revert is similar — the v1 data stays.

Your real proposal, and why it's sound

What you're describing — make transcript.jsonl full-per-checkpoint (with a boundary marker per checkpoint instead of a slice) — is just making the compact format adopt full.jsonl's existing model. That's consistent with your own design constraint from the carry-forward work: "duplicate data on the checkpoints branch is acceptable." Benefits line up exactly with what you said:

• Every checkpoint self-contained → robust to any middle loss, no piecing-together, regeneration unnecessary.
• UI read model → your "read the last checkpoint + just positions from the others" works if the last is identifiable and survived. But the strictly-more-robust version of that same idea is "read any surviving checkpoint for content, use the markers to segment" — which is what self-containment buys you, and it doesn't depend on the tail being intact.

The two costs to weigh

1. The marker must be in compact coordinates (the per-format problem from before): you can't reuse checkpoint_transcript_start (raw line offset) — compaction merges/inlines/drops lines, and several agents treat StartLine as a message/item index. So the boundary has to be emitted during compaction, per format. That's the real implementation cost of "full + position marker."

2. Storage isn't O(n) — but packing saves you. Full-per-checkpoint means a growing prefix stored N times → O(n²) in loose objects. The earlier "git dedups identical ranges" framing I gave is wrong: git dedups whole identical objects, and a growing transcript is a different object each time. What actually rescues it is packfile delta compression (git gc) — checkpoint N+1's full.jsonl packs as a small delta against N's. So the cost is real for loose objects and largely reclaimed once packed. You already pay this for full.jsonl; mirroring it on the (smaller) compact format is the same shape, smaller constant.

Net: your instinct is right for the compact format — full-per-checkpoint + a compact-space boundary marker is more robust and gives the cleaner UI read than delta slices. The thing to budget for is computing that marker inside each format's compaction path, not the storage.

can we start working on this?

2w ago·14m·4 calls

What would you like to clarify about the storage model? Happy to walk through either option in more detail, or you may have constraints/ideas I haven't accounted for — let me know what's on your mind.

how could 2 even work, we don't write the checkpoints at the same time, so we don't know what is the last

2w ago·1h 45m·75 calls

Done. The work is implemented, tested, lint-clean, and committed in three logical layers on a new branch.

Summary

Branch:soph/full-compact-transcript-per-checkpoint (off main)

What changed:transcript.jsonl now stores the full compacted session on every checkpoint (mirroring how full.jsonl already works), with a compact_transcript_start marker in session metadata recording where each checkpoint's slice begins. This makes every checkpoint self-contained — a session is reconstructable from any single surviving checkpoint, robust to a mid-history checkpoint being lost, reverted, or dropped in a rebase — and lets the UI read one file and segment by the marker instead of stitching deltas.

The three commits (each builds on the last):

1. compact: add FullWithBoundary — compacts the whole transcript + returns the boundary as lineCount(full) − lineCount(delta), reusing every format's existing slicing logic (no rewrite of the 7 emitters). 4 new tests, incl. an index-format (Gemini) case.
2. api/checkpoint: add CompactTranscriptStart marker — *int so nil (legacy delta file) is distinguishable from 0 (full file, first checkpoint), plus GetCompactTranscriptStart(). Round-trip + getter tests.
3. checkpoint: store full compact transcript per checkpoint — wires the boundary through both the normal write and the pre-push OPF rewrite path; updates the two old delta-semantics tests to assert the new full-content + marker contract.

Verification:mise run fmt + mise run lint clean (0 issues); checkpoint, compact, api/checkpoint, fsstore, and strategy test suites all pass.

Known tradeoff (documented in code): a deterministic off-by-one when a single logical message (a streaming assistant message, or a tool_result carrying text) straddles the exact StartLine boundary — harmless for segmentation.

Two notes for when you pick this up:

• In-repo readers only write/point attranscript.jsonl; the actual consumer that segments by the marker is downstream (entire.io), which will need to read compact_transcript_start and slice — and treat its absence as "legacy, read from line 0."
• I haven't pushed or run the full test:e2e:canary — say the word and I'll run the canary and/or push.

f1a2b7acompact: add FullWithBoundary for full transcript + scope marker\ \ FullWithBoundary compacts the entire transcript (ignoring StartLine for\ truncation) and returns the full compact bytes plus the line index at which\ this checkpoint's slice begins in the compact output. The boundary is derived\ as lineCount(full) - lineCount(delta), reusing each format's existing,\ independently-tested slicing behavior rather than threading source indices\ through all seven emitters. The only imprecision is a deterministic off-by-one\ when a single logical message straddles the exact boundary.\ \ This is the foundation for storing a full, self-contained compact transcript on\ every checkpoint (so a session is reconstructable from any single checkpoint)\ while still letting readers segment a checkpoint's slice via the marker.\ \ Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com\ Entire-Checkpoint: 9bf575d87dc5+183

ed36714api/checkpoint: add CompactTranscriptStart marker to session metadata\ \ Add CompactTranscriptStart *int (compact_transcript_start) plus a\ GetCompactTranscriptStart accessor. transcript.jsonl now stores the full\ compacted session, so this marks where a checkpoint's slice begins:\ compactLines[CompactTranscriptStart:].\ \ A pointer is used so "absent" (a legacy checkpoint whose transcript.jsonl holds\ only its delta, read from line 0) is distinguishable from 0 (a full file whose\ first checkpoint starts at the beginning).\ \ Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com\ Entire-Checkpoint: da4c0cb275ab+81/-2

7e9c63fcheckpoint: store full compact transcript per checkpoint with boundary marker\ \ writeCompactTranscript now uses compact.FullWithBoundary to write the entire\ compacted session into transcript.jsonl on every checkpoint, and returns the\ checkpoint's boundary so it can be recorded in session metadata as\ CompactTranscriptStart. Both write paths are covered:\ - normal write: writeTranscript returns the boundary, threaded into the\ per-session metadata.json.\ - pre-push OPF rewrite: replaceTranscript re-records the boundary via\ setCompactTranscriptStart after regenerating the compact transcript.\ \ This makes each checkpoint self-contained: the session is reconstructable from\ any single surviving checkpoint, robust to a mid-history checkpoint being lost,\ reverted, or dropped during a rebase. Readers segment a checkpoint's slice as\ fullCompactLines[CompactTranscriptStart:].\ \ fsstore writes no compact transcript, so its metadata leaves the marker nil.\ \ Update the existing scoped-transcript tests to assert the new full-content +\ marker contract instead of the old delta semantics.\ \ Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com\ Entire-Checkpoint: 27a4b7b0bdeb+157/-46

/simplify → 4 cleanup agents in parallel → apply the fixes

You are improving the quality of the changed code, not hunting for bugs. Review it for reuse, simplification, efficiency, and altitude issues, then fix what you find. Do not look for correctness bugs — that is what /code-review is for.

Phase 0 — Gather the diff

Run git diff @{upstream}...HEAD (or git diff main...HEAD / git diff HEAD~1 if there's no upstream) to get the unified diff under review. If there are uncommitted changes, or the range diff is empty, also run git diff HEAD and include the working-tree changes in scope — the review often runs before the commit. If a PR number, branch name, or file path was passed as an argument, review that target instead. Treat this diff as the review scope.

Phase 1 — Review (4 cleanup agents in parallel)

Launch 4 independent review agents via the Agent tool, all in a single message so they run concurrently. Pass each agent the diff and one of the four angles below. Each returns its findings with file, line, a one-line summary, and the concrete cost (what is duplicated, wasted, or harder to maintain).

Reuse

Flag new code that re-implements something the codebase already has — Grep shared/utility modules and files adjacent to the change, and name the existing helper to call instead.

Simplification

Flag unnecessary complexity the diff adds: redundant or derivable state, copy-paste with slight variation, deep nesting, dead code left behind. Name the simpler form that does the same job.

Efficiency

Flag wasted work the diff introduces: redundant computation or repeated I/O, independent operations run sequentially, blocking work added to startup or hot paths. Also flag long-lived objects built from closures or captured environments — they keep the entire enclosing scope alive for the object's lifetime (a memory leak when that scope holds large values); prefer a class/struct that copies only the fields it needs. Name the cheaper alternative.

Altitude

Check that each change is implemented at the right depth, not as a fragile bandaid. Special cases layered on shared infrastructure are a sign the fix isn't deep enough — prefer generalizing the underlying mechanism over adding special cases.

Phase 2 — Apply the fixes

Wait for all four agents to complete, dedup findings that point at the same line or mechanism, and fix each remaining one directly. Skip any finding whose fix would change intended behavior, require changes well outside the reviewed diff, or that you judge to be a false positive — note the skip rather than arguing with it. Finish with a brief summary of what was fixed and what was skipped (or confirm the code was already clean).

2w ago·2m·7 calls

I'll wait for the remaining three agents to report.

1
2
3
4
5
6
7

// Change signature from:
func (s *GitStore) setCompactTranscriptStart(sessionPath string, start int, entries map[string]object.TreeEntry) error

// To:
func (s *GitStore) setCompactTranscriptStart(sessionPath string, start int, metadata *Metadata, entries map[string]object.TreeEntry) error
  // caller: metadata.CompactTranscriptStart = &start
  // then re-marshal once, no need to fetch

2w ago

1

func (s *GitStore) updateSessionMetadata(sessionPath string, mutate func(*Metadata) error, entries map[string]object.TreeEntry) error

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

// updateSessionMetadata reads session metadata, applies a mutation, and rewrites it to the entries map.
func (s *GitStore) updateSessionMetadata(sessionPath string, mutate func(*Metadata) error, entries map[string]object.TreeEntry) error {
    metadataPath := sessionPath + paths.MetadataFileName
    entry, exists := entries[metadataPath]
    if !exists {
        return fmt.Errorf("session metadata not found at %s", metadataPath)
    }
    metadata, err := s.readMetadataFromBlob(entry.Hash)
    if err != nil {
        return fmt.Errorf("read session metadata: %w", err)
    }
    if err := mutate(metadata); err != nil {
        return err
    }
    metadataJSON, err := jsonutil.MarshalIndentWithNewline(metadata, "", "  ")
    if err != nil {
        return fmt.Errorf("marshal session metadata: %w", err)
    }
    metadataHash, err := CreateBlobFromContent(s.repo, metadataJSON)
    if err != nil {
        return err
    }
    entries[metadataPath] = object.TreeEntry{
        Name: metadataPath,
        Mode: filemode.Regular,
        Hash: metadataHash,
    }
    return nil
}