can you put it in meaningful commits? · Entire

Overwriting Sync Design Proposal

Assumptions

• Target is read-only, allowing overwrites.
• Bandwidth efficiency is not a priority.

Current Design Overview

The existing synchronization is designed for safe incremental mirroring, ensuring no data loss or conflicts by checking:

• Fast-forward checks: Verify that the updates can safely push to the target without losing data.
• Blocked updates: Avoid any operations that would disrupt ongoing updates.
• Prune semantics: Manage the state effectively to minimize the impact of deletions.

Key Components

1. Fast-forward/Conflict Planning in PlanRef
2. Relay Eligibility Rules in CanIncrementalRelay
3. Materialized Fallback Path in materialized.Execute

Proposed Simplifications

1. Adopt a Forceful Replication Model:
   • Treat sync as direct replication, which eliminates complexities tied to incremental checks.
2. Update Logic:
   • Always list source and target references.
   • For each desired reference:
     • If source hash matches target hash, skip.
     • Otherwise, send an update command with target's current hash as old and the source hash as new.
3. Fetch Logic:
   • Fetch from the source without haves to receive a complete data pack instead of a minimal delta.
4. Direct Target Push:
   • Push updated data directly to the target.
5. Simplify Other Components:
   • Remove complexities related to incremental logic, including bootstrap special-casing, incremental relay eligibility, and materialized object counts.

Caveats

• Target reference discovery is still required for effective updates since receive-pack commands need the current old hash.
• Considerations for request/body size limits at the target could still warrant bootstrap batching.

Additional Considerations

In practice, this structure allows:

• Source discovery and validation to ensure updates are effective.
• Streamlined planning for creating, updating, deleting based purely on hash comparisons.
• Focused, efficient data transfer without needless overhead.

Implementation Steps

1. Define New Operations:
   • Introduce clear operational definitions within the software architecture.
2. Create Testing Framework:
   • Ensure that execution and behavior align with the single-command approach.
3. Validate Performance:
   • Measure efficiency gains in both computation and data transfer with the new approach.
4. Analyze User Impact:
   • Gather feedback based on user interactions and adaptation of the new operational model.

Conclusion

Through these modifications, the entire sync process can shift from a complex, safety-focused model to a more streamlined and direct replication approach, yielding faster operations with less overhead.