Feat/embedding api spec

.openspec/adr/0034-embedding-api-owns-the-connection-driver-out-of-tree.md

@@ -0,0 +1,103 @@
+# 0034 — The embedding API owns the connection; the `sqlx` driver stays out of tree
+
+**Status:** Proposed · **Date:** 2026-08-28
+
+## Context
+
+`examples/README.md`: "this crate exposes its parser/codegen/VM pipeline
+directly rather than an ergonomic `Connection`/`prepare`/`bind` wrapper, so each
+example wires those pieces together the same way the `sqlite-rs` CLI binary
+does."
+
+The pieces are built. `Vm::bind_params` and the `Variable` opcode (spec 009,
+ADR-0015), `compile_statement`, the autocommit state `execute_transaction_step`
+threads, and `Pager`'s file locks (spec 007). Creating a database shows the
+shape of what is left: the primitive is public
+(`DatabaseHeader::new_empty_page1`) and the CLI calls it, but no API offers it.
+Missing, then: a facade, a `Send + Sync` boundary, and one counter.
+
+The counter is the only genuine capability gap. Nothing in `src/vdbe/` reports
+rows changed, so a caller cannot tell a conditional `UPDATE` that matched from
+one that did not, and optimistic concurrency is built on exactly that. The
+driving consumer (SQE, which stores Iceberg catalog pointers in SQLite) is
+working around it with SELECT-then-UPDATE in a transaction, sound only while a
+single writer is guaranteed.
+
+Spec 013 defines the surface and makes the counter its Requirement 1. This ADR
+records what that closes.
+
+## Decision
+
+**A native `Connection`/`Statement`/`Transaction` API in `src/api.rs`** with a
+rows-affected count, open-or-create modes, positional `?` binding, explicit
+transactions, a stated durability contract, and a retryable busy error distinct
+from fatal ones. Named parameters are rejected at prepare time rather than
+reaching execution as ADR-0015's always-NULL stub, which a public facade would
+otherwise make reachable and unattributable.
+
+**A `Send + Sync` handle over a connection-owned worker thread.** `Rc` is not
+`Send` and ownership does not change that, so making the connection type itself
+`Send` requires an `Arc` refactor of `Pager`/`PageSource`. The connection
+instead creates its `Rc` graph on a thread it owns and never lets it leave.
+`sqlx`'s own SQLite driver does this for a C `sqlite3*`
+(`sqlx-sqlite-0.9.0/src/connection/worker.rs`), which is evidence the shape is
+the standard answer rather than a concession.
+
+**The `sqlx` driver lives in a separate crate** (`sqlx-sqlite-rs`) and is a
+`SHOULD`, not a `MUST`. This crate's `[dependencies]` stays empty. SQE showed
+the driver is not the only path by implementing its catalog trait directly over
+the public internals; the driver remains right for consumers who will not write
+one.
+
+## Alternatives rejected
+
+**A C ABI shim** exporting `sqlite3_open`/`sqlite3_prepare_v2`/`sqlite3_step`
+so `libsqlite3-sys` links here. Every existing consumer in every language would
+work unchanged. Rejected: it reintroduces the `unsafe` boundary the crate exists
+to remove, needs a carve-out larger than `src/sys/`, returns the error surface
+to null-pointer semantics, and forces the C threading contract on a design that
+gets to choose one. Defensible later, on top of spec 013; a bad substitute for
+it.
+
+**Cloning rusqlite's API.** Rejected: `.openspec/README.md` already commits to
+"inspired by rusqlite but not a wrapper", and rusqlite's shape is dictated by C
+ownership (`Connection` is `!Sync` because `sqlite3*` is, statements borrow
+through a `RefCell` cache, `close` hands the connection back on failure).
+Copying it imports constraints this crate does not have.
+
+**Making `Pager` `Arc`/`Mutex`.** Rejected on the grounds ADR-0013 and ADR-0017
+already established: one `Vm` shares a page source across N cursors via cheap
+`Rc` clones, so `Arc` taxes the Tier 0 read path to serve a requirement that
+lives at the connection boundary.
+
+**Putting the `sqlx` driver in this repository**, feature-gated. Rejected: an
+empty `[dependencies]` table is one of this crate's two headline properties, a
+feature-gated dependency still appears in `Cargo.lock`, the SBOM and
+`cargo deny`, and it would couple this crate's cadence to `sqlx`'s. A separate
+crate can track `sqlx` 0.9, 0.10 and 1.0 while this API stays still.
+
+**Async connections.** Rejected for now, not on principle: `sqlx` drivers run
+blocking work on their own executor, so async buys the driving consumer nothing,
+and an async pager is a storage decision spec 013 must not pre-empt.
+
+## Consequences
+
+- Spec 013 needs its own value block. It sits outside the V1--V12 ladder (those
+  deliver SQL surface, it delivers consumability) and is a prerequisite for V7's
+  stated demo, so it belongs before V8. One minor per phase (ADR-0006) implies
+  its own minor.
+- `foreign_keys = ON` will be accepted before it is enforced (V8), because
+  `sqlx` issues it unconditionally. That divergence needs its own ADR under
+  ADR-0004, not a silent no-op.
+- The rows-affected count is the one item a consumer cannot work around
+  cleanly. Shipping the facade without it leaves every compare-and-swap consumer
+  reinventing a single-writer workaround, and the ones who get it wrong lose
+  writes silently.
+- Two SQL-semantics prerequisites surface alongside this work and belong to
+  V3/V7: `CREATE TABLE IF NOT EXISTS` is ignored after parsing, and a composite
+  `PRIMARY KEY`/`UNIQUE` table constraint is neither enforced nor backed by an
+  auto-created `sqlite_autoindex_*`. The second also affects this crate's own
+  byte-compatibility claim, which makes it the highest-value item in the set.
+- Consumer statement sets become fixture families under spec 004's harness
+  (spec 013/Req-6), so "an application can use this" is measured rather than
+  asserted.

.openspec/adr/index.md

@@ -37,3 +37,4 @@ Specs record what the system must do; ADRs record **why it is shaped this way**
 | [0031](0031-vendor-nix-subset.md) | Vendor a `nix` subset: reintroduce a single, narrow `unsafe` boundary | 2026-08-26 |
 | [0032](0032-hash-group-by-second-strategy.md) | Hash `GROUP BY` is a second strategy with its own opcode family, and still emits groups in key order | 2026-08-27 |
 | [0033](0033-constant-propagation-and-or-to-in-extend-fast-paths-in-place.md) | Constant propagation and OR-to-IN extend existing equality fast paths in place; only genuine range seeks wait for a new opcode | 2026-08-28 |
+| [0034](0034-embedding-api-owns-the-connection-driver-out-of-tree.md) | The embedding API owns the connection; the `sqlx` driver stays out of tree | 2026-08-28 |

.openspec/specs/010-vdbe-write-opcodes/spec.md

@@ -323,6 +323,103 @@ before the row's own `Insert`, dispatching `ON CONFLICT`
 **Tests:**
 `tests/corpus/unique_constraint_test.rs::insert_or_replace_displaces_the_conflicting_row`
 
+### Requirement 8: A Write MUST NOT Ignore an Index It Cannot Read [MUST]
+
+A write to a table MUST maintain every index present for that table in
+`sqlite_master`, or MUST refuse the write. It MUST NOT succeed while leaving an
+index on disk unmaintained.
+
+Requirement 7's `emit_unique_check` iterates `schema.indexes.iter().filter(|i|
+i.unique)` and `IdxInsert` (Requirement 5) maintains what codegen emits for, so
+both are bounded by what the schema reader returned. `src/schema/ddl_reader.rs`
+deliberately returns less than the file contains: its own test,
+`auto_index_with_null_sql_is_omitted`, records that an index whose
+`sqlite_master.sql` is NULL is "gracefully skipped rather than erroring",
+because the naive reader cannot recover a column list from a NULL. Every
+`sqlite_autoindex_*` stock SQLite creates for a `PRIMARY KEY` or `UNIQUE` table
+constraint has exactly that shape.
+
+Graceful degradation is correct for a read. For a write it is data loss, and it
+is silent: the row lands in the table b-tree, the index keeps its old contents,
+and the write returns success.
+
+Measured 2026-08-28, this crate's CLI at 0.18.5 against stock `sqlite3` 3.51.0:
+
+```sql
+-- stock sqlite3 creates the table; sqlite_autoindex_t_1 exists
+CREATE TABLE t (a TEXT NOT NULL, b TEXT NOT NULL, c TEXT NOT NULL, v TEXT,
+                PRIMARY KEY (a, b, c));
+INSERT INTO t VALUES ('c','ns','t1','v1');
+```
+
+```console
+$ sqlite-rs exec t.db "INSERT INTO t VALUES ('c','ns','t2','v2')"   # rc=0
+$ sqlite-rs exec t.db "INSERT INTO t VALUES ('c','ns','t1','dup')"  # rc=0, duplicate accepted
+$ sqlite3 t.db "SELECT count(*) FROM t; PRAGMA integrity_check;"
+1
+wrong # of entries in index sqlite_autoindex_t_1
+row 2 missing from index sqlite_autoindex_t_1
+row 3 missing from index sqlite_autoindex_t_1
+```
+
+`count(*)` answers from the stale index and undercounts. A control table with no
+primary key writes and verifies clean, which places the fault at index
+maintenance rather than the file format, page 1 or the schema table.
+
+The precedent for the shape of the fix is spec 007 Requirement 1: a hot journal
+is detected and refused rather than silently ignored, because serving
+pre-rollback pages is worse than failing to open. Same argument, write side.
+
+Two ways to satisfy this, and either is acceptable:
+
+- **Read the autoindex.** Its column list is recoverable without SQL text, from
+  the table's declared constraint in its own `CREATE TABLE`. Then the existing
+  `NoConflict` and `IdxInsert` paths cover it with no new opcode.
+- **Refuse the write.** A table carrying an index the reader could not parse is
+  read-only until it can. Narrower, and it converts silent corruption into an
+  error a caller can act on.
+
+Creating `sqlite_autoindex_*` for a declared constraint is a separate `CREATE
+TABLE`-side gap (`src/codegen/stmt/insert.rs` records it, V3/V7 owns it), and it
+has its own file-level consequence: a table this crate creates with a declared
+composite `PRIMARY KEY` has no autoindex, and stock `sqlite3` then answers any
+write or `integrity_check` on it with "database disk image is malformed (11)".
+Closing this requirement without closing that one leaves creation broken; closing
+that one without this leaves adoption of a foreign file broken.
+
+**Implementation:** `src/schema/ddl_reader.rs::index_schema` (planned), consumed
+by `src/codegen/stmt/insert.rs::emit_unique_check`
+
+**Tests:** `tests/corpus/autoindex_maintenance_test.rs` (planned)
+
+#### Scenario: A write to a stock-created composite-PK table keeps the index consistent
+
+- GIVEN a table created by the pinned oracle with a composite `PRIMARY KEY`, so
+  `sqlite_autoindex_*` exists, holding one row
+- WHEN this crate inserts a second row
+- THEN the oracle's `PRAGMA integrity_check` reports ok and `count(*)` returns 2,
+  or the insert was refused and the file is byte-identical to before
+
+**Tests:** `tests/corpus/autoindex_maintenance_test.rs::stock_composite_pk_stays_consistent` (planned)
+
+#### Scenario: A duplicate against an autoindex-backed constraint is refused
+
+- GIVEN the same table holding `('c','ns','t1')`
+- WHEN this crate inserts `('c','ns','t1')` again
+- THEN it fails with a uniqueness error, or the write is refused for the reason
+  above, and in neither case is a duplicate row persisted
+
+**Tests:** `tests/corpus/autoindex_maintenance_test.rs::autoindex_duplicate_is_refused` (planned)
+
+#### Scenario: A named index is unaffected
+
+- GIVEN a table with no declared primary key and a named `CREATE UNIQUE INDEX`
+- WHEN this crate inserts a new key and then a duplicate
+- THEN the new key is written, the duplicate is refused, and the oracle's
+  `integrity_check` reports ok
+
+**Tests:** `tests/corpus/autoindex_maintenance_test.rs::named_index_round_trips` (planned)
+
 ## Related regimes
 
 - Tier suite: `tests/tiers/tier2.rs::t2_crud_round_trips_on_rowid_tables`