Skip to content

Record why persistence is SQLite, and the trigger that would change it — prerequisite-gated #526

Description

@eaitbrahim

Raised in docs/presentations/se.md §9.6. Filed prerequisite-gated, in the style of #333/#335/#341: not work to start, but a named trigger so the question is answered before it is forced.

The current shape, and why it is right today

Persistence is SQLite in WAL mode via a hand-rolled Repository — no ORM, plain SQL, hand-written mapping. That is a good fit for what keel is:

  • One writer. One account, one venue, one cycle per interval, one process. The agent is synchronous by design (see se.md §10 — concurrency buys nothing here and costs testability).
  • The audit trail is the schema. Every order, veto and state transition is a row the engine reads back to make its next decision. Hand-written SQL keeps that inspectable.
  • The database is a file. It can be copied, diffed, attached to a bug report, and backed up before a migration — which keel update does automatically (*.db.bak-before-<version>-<stamp>).

None of that is accidental, and this issue is not a proposal to change it.

What would actually strain it

Recording the shapes, so the trigger is concrete rather than a vague "when we scale":

  1. Concurrent writers. Today's deployment runs three profiles — live, paper-daily, paper-hourly — but each has its own database file, so there is still one writer per file. A design that had two processes writing one file (e.g. a supervisor alongside the agent, or a web layer that writes) is the first genuine break. WAL permits concurrent readers with one writer; it does not make two writers safe.
  2. Cycle frequency. The live wrapper trades once per UTC day; the hourly evidence profile cycles hourly. Neither is close to write-bound. A sub-minute cycle with per-tick persistence would be a different question.
  3. Candle volume. The deployment's keel.db already holds ~129MB, mostly candles, and grows with every fetched product. Read performance on the median-volume statistic and backtests is the thing to watch, not write contention.
  4. Long-running transactions during a migration. keel migrate runs per database on update; a schema change over a large candle table is the one operation with a real stall risk.

The trigger — revisit when any of these fires

  • A second process needs to write the same database file.
  • A deployment profile moves to a cycle faster than hourly.
  • A migration on the candle tables takes long enough to threaten a cycle window.
  • Read latency on the liquidity/backtest paths becomes a measured complaint, not a suspicion.

Until one fires, the answer is "SQLite, and here is why" — and that answer should be written down so it stops being re-litigated by each new reader.

Acceptance criteria

  • The reasoning above lives somewhere durable (an ADR, or the architecture section of the docs) rather than only in an issue.
  • The trigger conditions are stated where someone changing the process model would see them.
  • If a trigger fires, the question is answered with a measurement — not by reaching for Postgres because it feels more serious.

Notes

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions