Merge branch 'main' into fix/1743-defer-checkpoint-push-empty-remote

4b5b46e→main·

karthik-rameshkumar·3d ago·8 files·+728 added/-54 removed

Changes

8

• MCLAUDE.md+1/-1

• cmd/entire/cli

  • integration_test

    • Asupabase_secret_redaction_test.go+117

• Msearch_cmd.go+130/-14

• Msearch_cmd_test.go+133/-4

• docs

  • Msecurity-and-privacy.md+19/-18

• redact

  • Aproviders.go+75

  • Mredact.go+25/-17

  • Mredact_test.go+228

`` 582 unmodified lines

582 unmodified lines

• Shadow branch migration - if user does stash/pull/rebase (HEAD changes without commit), shadow branch is automatically moved to new base commit
• Orphaned branch cleanup - if a shadow branch exists without a corresponding session state file, it is automatically reset when a new session starts
• PrePush hook can push entire/checkpoints/v1 branch alongside user pushes
• OPF (OpenAI Privacy Filter) runs at pre-push, not post-commit: when redaction.openai_privacy_filter.enabled is true, the PrePush hook re-redacts unpushed entire/checkpoints/v1 commits with the OPF 8th layer, builds new commits carrying an Entire-OPF-Applied: true trailer, and atomically updates the local v1 ref before pushing. Per-commit condensation stays on the fast 7-layer pipeline. See strategy/manual_commit_opf_rewrite.go and docs/security-and-privacy.md for the full flow, including divergence detection, bootstrap caps, and CAS-on-conflict semantics.
• OPF (OpenAI Privacy Filter) runs at pre-push, not post-commit: when redaction.openai_privacy_filter.enabled is true, the PrePush hook re-redacts unpushed entire/checkpoints/v1 commits with the OPF 9th layer, builds new commits carrying an Entire-OPF-Applied: true trailer, and atomically updates the local v1 ref before pushing. Per-commit condensation stays on the fast 8-layer pipeline. See strategy/manual_commit_opf_rewrite.go and docs/security-and-privacy.md for the full flow, including divergence detection, bootstrap caps, and CAS-on-conflict semantics.
• Safe to use on main/master since it never modifies commit history

Key Files

MCLAUDE.md+1/-1

What Entire redacts automatically

Entire automatically scans transcript and metadata content before writing it to the entire/checkpoints/v1 branch. Five always-on secret detection methods run during condensation, plus a conditional sixth pass for user-defined secret rules (see Customizing redaction below), an opt-in seventh pass for PII (see Optional PII redaction below), and an opt-in eighth pass that shells out to the OpenAI Privacy Filter model (see Optional OpenAI Privacy Filter below):
Entire automatically scans transcript and metadata content before writing it to the entire/checkpoints/v1 branch. Six always-on secret detection methods run during condensation, plus a conditional seventh pass for user-defined secret rules (see Customizing redaction below), an opt-in eighth pass for PII (see Optional PII redaction below), and an opt-in ninth pass that shells out to the OpenAI Privacy Filter model (see Optional OpenAI Privacy Filter below):

1. Entropy scoring — Identifies high-entropy strings (Shannon entropy > 4.5) that look like randomly generated secrets, even if they don't match a known pattern.
2. Pattern matching — Uses Betterleaks built-in rules to detect known secret formats.
3. Credentialed URI detection — Redacts URLs with embedded passwords, such as scheme://user:password@host.
4. Database connection-string detection — Redacts JDBC, Postgres keyword DSN, SQL Server, and ODBC-style connection strings containing passwords.
5. Bounded credential value detection — Redacts password-like config values such as DB_PASSWORD=... and PGPASSWORD=... while preserving the surrounding key.
6. Provider token prefixes — Deterministically redacts known secret-key prefixes (e.g. Supabase sb_secret_, sbp_) regardless of entropy or surrounding context.
7. Credentialed URI detection — Redacts URLs with embedded passwords, such as scheme://user:password@host.
8. Database connection-string detection — Redacts JDBC, Postgres keyword DSN, SQL Server, and ODBC-style connection strings containing passwords.
9. Bounded credential value detection — Redacts password-like config values such as DB_PASSWORD=... and PGPASSWORD=... while preserving the surrounding key.

Detected secrets are replaced with REDACTED before the data is ever written to a git object. The five secret-detection passes above are always on and cannot be disabled. User-defined rules (inline custom_redactions and rule packs) add a sixth secret-detection pass that only runs when configured.

Optional PII redaction

Optional OpenAI Privacy Filter (opf)

A separate, opt-in layer that shells out to the OpenAI Privacy Filter (opf) — a 1.5B-parameter token-classification model that finds names, emails, phone numbers, addresses, dates, URLs, account numbers, and secrets that pure regex can miss. Disabled by default. Runs in addition to the seven built-in layers, only at push time — never per-turn and never at commit time. Local commits stay on the fast 7-layer pipeline so per-commit latency is unchanged; OPF only re-redacts checkpoints right before they leave the machine via git push.

• Yes runs OPF for this push only.
• No skips OPF for this push only; the 7-layer-redacted content reaches the remote.
• No skips OPF for this push only; the 8-layer-redacted content reaches the remote.
• Always runs OPF this push AND writes prompt_default: "always" to .entire/settings.local.json so future pushes don't ask.
• Ctrl-C aborts the push entirely — git push exits non-zero, no refs go to the remote.

CI consideration: if you've enabled OPF locally and your CI runs git push (e.g. an agent-driven workflow), the CI push will attempt to run OPF too. If the opf binary isn't installed in CI, the push will abort with OPFRuntimeFailedError rather than silently shipping under-redacted content — by design, since "I enabled OPF" should mean "no content leaves my machines without OPF."

OPF failures at push time are fail-closed: if OPF is not on PATH, fails to start, or times out during the pre-push rewrite, the per-process circuit breaker trips and the rewrite aborts the push with OPF runtime failed; aborting push. Nothing reaches the remote. The intent is that "the user enabled OPF" means "I do not want unredacted content leaving this machine" — falling back to 7-layer silently on the push path would violate that contract.