Skip to content

fix(quantizers): use the field names the server actually reads - #612

Open
dudanogueira wants to merge 1 commit into
mainfrom
fix/quantizer-field-names
Open

fix(quantizers): use the field names the server actually reads#612
dudanogueira wants to merge 1 commit into
mainfrom
fix/quantizer-field-names

Conversation

@dudanogueira

Copy link
Copy Markdown

Motivation

Weaviate parses quantizer settings out of the config map by exact key, with no case folding and no snake_case fallback:

// entities/vectorindex/hnsw/rq_config.go
if err := common.OptionalIntFromMap(rqConfigMap, "rescoreLimit", func(v int) {

and returns them under the same names (RescoreLimit int \json:"rescoreLimit"``). The client used snake_case for half of them, so those settings were silently dropped on write and null on read, on hnsw and flat alike. Setting a rescore limit did nothing and reported nothing.

Found while investigating #606, which is unfixable in practice without this: locating a nested rq under a dynamic index still yields a null rescoreLimit.

Approach

Was Now Records
rescore_limit rescoreLimit BQ, SQ, RQ
training_limit trainingLimit PQ, SQ
bit_compression bitCompression PQ
encoder_type + encoder_distribution nested encoder: {type, distribution} PQ

PQ's encoder needed a shape change rather than a rename — the server models it as a nested object (pq_config.go: Encoder PQEncoder \json:"encoder"`, read via encoderFromMap/encoderDistributionFromMap), while the client had two flat components. The record now carries an Encodercomponent;PQ.encoderType()andPQ.encoderDistribution()` are kept as derived accessors and the builder API is untouched, so the flat, convenient way of setting them still works.

No alternate names, deliberately: unlike #607, the snake_case spellings were never valid on the wire, so there is no stored config spelled that way to stay compatible with.

enabled, bits, segments, centroids and cache already matched and are unchanged.

Key areas for review

  • PQ.java — the Encoder record and the builder→record mapping, which collapses to null when neither encoder field was set so an empty encoder: {} is never emitted
  • Whether keeping encoderType()/encoderDistribution() as derived accessors is the right call, or whether callers should be pushed to encoder()

Testing

  • JSONTest — the PQ/SQ/RQ/BQ rows now expect the server's spelling. Each row asserts both directions, so they cover read as well as write.
  • CollectionsITest.test_quantizerSettingsRoundTrip — creates a collection with rq.rescoreLimit(42).bits(8), reads the config back from a real server and asserts both values survive. This is the assertion that was missing: the existing quantizer IT only checked _kind(), which matched even while every setting was being discarded.

Verified the new test fails without the fix: expected:<42> but was:<null>.

Locally green: 382 unit tests, and CollectionsITest against a 1.39.0 container (16 run, 0 failures).

Breaking changes

PQ's canonical constructor changes — encoderType, encoderDistribution become a single encoder component. Accessors and the builder are unchanged, so only code calling new PQ(...) positionally or destructuring it in a record pattern is affected.

Behaviourally, collections created by older clients have no quantizer settings stored at all (they were dropped), so there is nothing to migrate — this only starts sending values that previously went nowhere. Worth knowing that a collection created after this change will actually apply a rescore limit where before it silently used the server default.

Closes #611

🤖 Generated with Claude Code

https://claude.ai/code/session_01WmY5dAGWCccWDoqkKNC2JU

Weaviate parses quantizer settings out of the config map by exact key,
and returns them under the same names. The client used snake_case for
half of them, so rescoreLimit, trainingLimit and bitCompression were
dropped on write and came back null on read -- on every index type. A
user who set a rescore limit never set one.

  rescore_limit    -> rescoreLimit    (BQ, SQ, RQ)
  training_limit   -> trainingLimit   (PQ, SQ)
  bit_compression  -> bitCompression  (PQ)

PQ's encoder needed a shape change rather than a rename: the server
nests it as encoder: {type, distribution} while the client had two flat
components. PQ.encoderType() and PQ.encoderDistribution() are kept as
derived accessors and the builder is unchanged, so only the canonical
constructor differs.

No alternate names: the snake_case spellings were never valid on the
wire, so no stored config uses them.

Closes #611

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WmY5dAGWCccWDoqkKNC2JU

@orca-security-eu orca-security-eu Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v6: quantizer settings use snake_case names the server never reads

1 participant