Skip to content

feat(gfql): verified column-stat facts in the index registry (typed-ontology phase 1) - #1854

Merged
lmeyerov merged 6 commits into
masterfrom
perf/gfql-colstats-facts
Aug 7, 2026
Merged

feat(gfql): verified column-stat facts in the index registry (typed-ontology phase 1)#1854
lmeyerov merged 6 commits into
masterfrom
perf/gfql-colstats-facts

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

First fact kind from the typed-ontology design (phase 1 needs no policy answers — it mirrors the existing explicit gfql_index_* surface and the benchmark I-lanes already measure indexed arms honestly, with build cost in disclosed bind_ms).

What: g.gfql_index_col_stats() (and folded into gfql_index_all()) computes per-column min/max/null-count facts for the bound node id and edge endpoint columns, stored in the index registry under the SAME identity+fingerprint validity contract as the physical indexes — a frame rebind is a safe miss, never a stale answer.

Consumption (conservative by construction): full-frame bounds contain every row subset's bounds, and zero nulls on the frame means zero nulls on any subset — so the dense two-hop count(*) kernel skips its O(E) endpoint-bounds scan when valid facts prove containment, and falls back to the scan (never declines) when they cannot. This targets the remaining q8@100k proof cost alongside the landed round-4 projection.

Robustness: registry copy methods move from positional construction to dataclasses.replace, so an added field can never again be silently dropped by a copy method.

Tests (both sides at every boundary): fact build + values per engine (pandas/polars/cudf); identity invalidation (equal-valued copy of the frame must MISS); the fact gate both ways (facts ⇒ no scan; no facts ⇒ scan; same count); conservative-miss (an out-of-domain endpoint on a row the rel filter drops makes facts insufficient — must scan AND serve); gfql_index_all inclusion. Container matrix run reporting below.

For owner review — not in the self-merge authorization.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi

lmeyerov and others added 2 commits August 6, 2026 09:04
…ntology phase 1)

gfql_index_col_stats() / gfql_index_all(): per-column min/max/null-count
facts for the bound node id and edge endpoint columns, same
identity+fingerprint validity as physical indexes. The dense two-hop
count kernel consumes them conservatively: valid facts proving
containment skip the O(E) endpoint-bounds scan; insufficient facts fall
back to the scan, never decline. Registry copies via dataclasses.replace
so added fields can't be dropped silently. Both-sides pins: skip/scan
gate, identity invalidation, conservative-miss fallback, index_all
inclusion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
Nodes-role facts also carry n_unique, so an UNFILTERED equal-domain
count (domain == the bound node frame) elides the interval scan when
the fact proves density; filtered domains always scan (a full-frame
fact cannot prove a subset dense). Pinned both ways with count parity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov

lmeyerov commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

DGX container matrix on the current head (3eec16fbe, includes the n_unique/interval-hint extension): 8,382 passed, 53 skipped, 19 xfailed across the full GFQL tree with real cudf/polars-gpu arms.

🤖 Generated with Claude Code

…eted API (owner review)

Point-by-point on #1854: (2) build_col_stats_fact declines ONLY by
explicit precondition (absent col, non-integer, empty); reductions run
bare so real errors reraise; null-bearing int cols get null_count with
min/max omitted. (3) eager targeting: node_columns/edge_columns params,
explicit unfactable requests raise by name, binding defaults skip;
laziness deferred to ontology Q7. (4) approximation DIRECTION stated on
_facts_prove_bounds + interval hint (under-approximate provability).
(5) boundary matrix: precondition declines, propagating reduction
errors, null-gate both sides, exact bounds edges, explicit-request
raises, gapped-id hint refusal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov

lmeyerov commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Review rework pushed — point by point:

  1. CI red — the red runs were GitHub-infra flakes (Service Unavailable resolving actions in the setup job, also hitting CodeQL); reruns issued and the rework head has fresh runs. I'll report the verified green (job-count checked) when it lands.
  2. Blanket except — gone. build_col_stats_fact now declines ONLY by explicit precondition (column absent, non-integer dtype, empty frame); the reductions run bare so a real backend error PROPAGATES — pinned by a test that monkeypatches an exploding Series.min and asserts the RuntimeError surfaces. A null-bearing integer column no longer risks int(NA): it gets a fact with null_count recorded and min/max omitted, and a both-sides pin shows consumers route it to the scan.
  3. Targeting/laziness — explicit and EAGER: defaults fact only the plan-relevant minimum (node id + edge src/dst bindings, what the count paths consult); new node_columns=/edge_columns= params fact more, and an explicitly named unfactable column raises by name (you asked for it) while binding defaults skip silently. gfql_index_all stays the catchall convenience. Plan-driven LAZY fact building is deliberately out of scope — that's the ontology Q7 re-verification question plus the GFQL: two-hop count returns a STALE answer after in-place frame mutation (degree-count memo setattrs onto the caller's Plottable, keyed by id() — the BLOCKER-1 pattern the same file forbids) #1825 implicit-state hazard; eager build keeps fact cost a declared setup step, matching how the harness discloses index builds in bind_ms.
  4. Approximation direction — now stated as a contract: _facts_prove_bounds is an UNDER-approximation of provability (True is always sound; False may be over-cautious and costs at most the O(E) scan — never an answer, never a decline). The interval hint documents the same direction, and notes its unfiltered-domain case is exact, not approximate.
  5. Thin error-path tests — added the boundary matrix: precondition declines (absent/non-integer/empty), propagating reduction errors, the null gate both sides, _facts_prove_bounds exact edges (min==lo / max==hi prove; ±1, nulls, missing bounds, non-integer refuse), explicit-request raises vs default skips, and gapped-id hint refusal (n_unique mismatch ⇒ interval scan runs ⇒ correct fallback answer). 17 green locally; container matrix queued.

🤖 Generated with Claude Code

@lmeyerov

lmeyerov commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Container matrix on the rework head (3018a48a): 8,387 passed, 53 skipped, 19 xfailed — full GFQL tree, real cudf/polars-gpu arms, including the new error-path boundary matrix.

🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov

lmeyerov commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Lock-lane receipts are in, and they refute the benefit on the board shape — reporting per the receipts-only rule.

Candidate build master+this (6398632f4), VALID lanes both scales. Unindexed cells: unchanged vs base (no harm). Indexed arm, q8@100k polars, per the harness's disclosed split: query_median_ms 13.88 → 13.79 (≈0.1ms, not the projected ~1.5ms), bind_median_ms +4.4ms (fact building).

Root cause is structural: this graph is multi-typed, so the full edge frame's endpoint bounds include city/country ids outside the Person interval — the conservative full-frame fact can never prove containment for a rel-filtered subset and always falls back to the scan. Sound by design, but the flagship count shape gets nothing. Facts would need to be per-relationship-type (typed-ontology phase 2) to earn their bind cost here.

Per the stated merge condition ('helping & not hurting, locked in pyg-bench'), this does not qualify for auto-merge: not hurting ✓, helping ✗. Options: (a) hold until per-type facts exist and re-lane (my recommendation — this PR then becomes their foundation), (b) merge as infrastructure accepting the +4.4ms indexed-bind cost, (c) close and fold into phase 2. Owner's call.

🤖 Generated with Claude Code

@lmeyerov

lmeyerov commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Motivation found and measured (owner's question: any real scenario?). On a Pokec-shaped graph — ONE node type, ONE relation, dense ids, 100k nodes / 2.4M edges, the exact shape of the mgBench suite Memgraph markets on — the facts fire as designed and save 2.48ms = 19.5% of the whole two-hop count (12.74 → 10.25ms, value-identical, exploratory micro-benchmark on the dev box, engine=polars). The board suite's multi-type graph was the one shape that can't benefit; homogeneous graphs — the classic social-network benchmark class — get the full win. Proper receipted Pokec lane available on request before merge.

🤖 Generated with Claude Code

lmeyerov and others added 2 commits August 6, 2026 23:27
…branch

CHANGELOG keeps both entries; gfql_fast_paths.py auto-merged (the projection
work and the fact consult touch different regions of the count path).

Verified locally: the affected suites pass apart from the 69 cudf cases that
fail IDENTICALLY on origin/master here (libnvrtc.so.12 absent on this box --
#1855's territory), baselined before attributing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
CHANGELOG keeps all entries; code and test files auto-merged. Verified locally:
only the 3 cudf cases that fail identically on origin/master here (libnvrtc
absent) fail; zero non-cudf failures across the lowering and grouped-aggregate
suites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
lmeyerov added a commit that referenced this pull request Aug 7, 2026
Registry keyed (role, column, type_column, type_value); build/consult/
tests/benchmark-motivation to follow. Parked per owner sequencing:
benefit-lock and merge #1854/#1856 first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov
lmeyerov merged commit 3539ba4 into master Aug 7, 2026
69 checks passed
lmeyerov added a commit that referenced this pull request Aug 7, 2026
Registry keyed (role, column, type_column, type_value); build/consult/
tests/benchmark-motivation to follow. Parked per owner sequencing:
benefit-lock and merge #1854/#1856 first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant