(Depends on #1597) Declare indexed properties per store, close the filter union, and route undeclared predicates to the segment store (speedkick)

[edwinyyyu]

Purpose of the change

Implements the records-and-queries side of the vector store redesign (design/vector_store_handoff.md on the design/tenant-lifecycle branch): what a record carries, which keys are indexed, what a filter may say, which nodes each backend evaluates during a search, how datetimes are stored, and what a client needs. Everything is written against VectorStoreCollection as it is today; the collections-and-keys side (UUID keys, registries, strict create, logical delete, stateless handles, shared tables) lands with the lifecycle work and replaces only how a handle is obtained.

Stacked on #1597 (its commit shows in this diff until it merges). #1602 is deferred to the lifecycle work; the typed, indexed columns added here are the foundation its selectivity probe needs.

What changed

• Declared, not dynamic, indexes. A store declares one indexed_properties schema for every collection it holds, from deployment configuration (indexed_properties on QdrantConf, MilvusConf, SQLiteVectorStoreConf, SQLiteVecVectorStoreConf) merged with the system keys of the one service that uses it, at the point the store is built for that service: DatabaseManager.get_vector_store(name, indexed_properties=...). indexed_properties_schema leaves VectorStoreCollectionConfig. EventMemory keeps expected_vector_store_collection_schema as its declaration and raises InvalidCollectionSchemaError when the collection's store does not declare its four keys. A second service whose keys the built store does not declare is refused (VectorStoreConfigurationError): services do not share a vector store.
• Undeclared keys are rejected. upsert raises UndeclaredPropertyKeyError before anything is sent; query raises it for a filter naming an undeclared key and UnsupportedFilterError for a node outside supported_filter_nodes. An undeclared key never exists in the vector store.
• The filter language. common/filter/filter_expression.py is the closed union from default (822ccb6): Equals, NotEquals, Ordering, In, IsMissing, n-ary And/Or, Not; every compiler is an exhaustive match. The string parser stays as the server's translation into the union. split_declared and the tree helpers the routing needs live beside it.
• Semantics. A predicate matches only a record holding a value of the compared type; NotEquals keeps records holding a differing value; Not(Equals) also keeps records holding none. The SQL column compiler renders a leaf of another type as FALSE and Not as NOT COALESCE(x, FALSE); Qdrant compiles NotEquals as "not the value and not empty"; Milvus pushes Not to the leaves by De Morgan (its own not excludes entities lacking the field, measured on Milvus Lite).
• Datetimes. Where a backend has no datetime type, a datetime is stored as microseconds since the epoch; both SQLite stores keep one typed, indexed, nullable column per declared key on the records table (sql_columns.py).
• Clients. request_timeout is required on QdrantConf and MilvusConf and passed to the client. Qdrant's hnsw_config, optimizers_config and quantization_config come over from default (a0753d3) with the m = 0 rule.
• EventMemory. One plan in query: the vector store gets the system predicates and the declared part of the caller's filter; the segment store applies the whole filter to seeds and windows afterward; with an undeclared part the vector limit widens by four up to limit * FilterOptions.max_overfetch_factor and returns what survived at the cap; with none, the first query is the last. Vector records carry only declared caller keys. An empty id or kind list admits nothing without a query.

Where this departs from the handoff, and why

• sqlite-vec evaluates every node, not four. The handoff planned vec0 metadata columns (comparisons joined by AND). sqlite-vec 0.1.9 rejects NULL in a metadata column:

  Expected text for TEXT metadata column name, received NULL
  

  and a declared key is optional per record (memmachine_event_session on its own). The store keeps the declared columns on the records table and hands the KNN rowid IN (SELECT rowid ... WHERE <filter>), which vec0 takes into the search (verified: k=1 with the nearest rows excluded returns the admitted row). Cost on 50k vectors, k=10: 0.5 ms unfiltered, 3.9 ms with a 1-in-7 allowlist, 7.9 ms with a 5-in-7 allowlist.
• Two errors the doc does not name, both forced by typed columns: PropertyTypeMismatchError on upsert of a declared key with a value of another type (a column would otherwise coerce it, and Qdrant's typed index would not index it), and IndexedPropertiesMismatchError when a SQLite store built with another schema opens a collection (nothing here migrates a column). Flagging both as additions.
• Milvus's properties JSON field is dropped (nothing read it since Answer with cosine scores and uuids, not vectors and stale properties (speedkick) #1598). Declared keys stay in the _p_ dynamic fields.
• Two semantic-storage tests that ordered string values are removed: ordering strings is not expressible in the union, by 822ccb6's decision.
• Native Qdrant and Milvus collection names still hash the collection config, which no longer carries a schema, so existing native collections are not reused. Nothing here migrates data.

Relation to the store-as-collection decisions

Decided in parallel with this PR: one store is one collection (a cell of the purpose x embedder matrix), handles are partitions, the collection name is a constructor parameter and the discriminator between stores sharing an engine or client, and namespace and container go. That is the handle side of the seam, so this PR keeps today's create_collection(namespace, name) shape and vocabulary rather than renaming half of it; the reshape lands with the lifecycle work and replaces how a handle is obtained. What it will find here:

• The store-level indexed_properties schema is already one schema per store, which under the decision is one schema per collection. The composition root builds the store for its one service and merges that service's system keys in, which is where a cell's store gets built. The "second service refused" guard in get_vector_store becomes unreachable once a store is a cell by construction, and can go with the reshape.
• The SQLite registry row records the schema the collection was created under (IndexedPropertiesMismatchError); with a registry table named by the collection, that record becomes per store.
• The typed, indexed columns in sql_columns.py are layout-agnostic: they attach to whatever records table a store has, per-collection now, one per store later.
• Native Qdrant and Milvus collection names still hash (namespace, config) here; the constructor-supplied collection name replaces that.

Tests

• Per backend, mixed into each store's test module (declared_schema_contract.py): undeclared key on upsert and in a filter raises; a value of another type raises; every node in supported_filter_nodes evaluates during a search; NotEquals excludes absence and Not(Equals) includes it; IsMissing matches absence; a predicate of another type matches nothing; datetime bounds at microsecond precision and in another zone; a filtered search finds an admitted record behind nearer excluded ones. These exercise API this change introduces (supported_filter_nodes, the declared schema, the new errors), so they cannot run against the unfixed stores; the semantics they pin were checked by hand against the old compilers (Qdrant must_not and Milvus not both admitted missing keys, SQL NOT excluded them).
• SQLite stores: a store built with another schema cannot open the collection. Qdrant: collection options coerce and forward, m is pinned; every declared key gets a payload index (integration).
• EventMemory: an undeclared predicate never reaches the vector store and is never stored there; a fully declared filter issues one query; widening makes up for dropped seeds and stops at the cap returning what survived; every count is a maximum; an empty id list issues no query.
• uv run pytest packages/server/server_tests: 2027 passed, 2 skipped. ruff check, ruff format --check clean. ty check packages reports only the diagnostics already on the base (spacy, sqlalchemy_segment_store.py:1045/1105, three test-side argument types).

🤖 Generated with Claude Code

https://claude.ai/code/session_01DtbsPQafKpBUn7ksnDU2UY