Skip to content

feat(analytics): add Metric View metadata pipeline - #512

Merged
atilafassina merged 8 commits into
mainfrom
mv-metadata-pipeline
Aug 5, 2026
Merged

feat(analytics): add Metric View metadata pipeline#512
atilafassina merged 8 commits into
mainfrom
mv-metadata-pipeline

Conversation

@atilafassina

Copy link
Copy Markdown
Contributor

TL;DR

Carries Unity Catalog Metric View display metadata from type generation into Analytics SSE results, giving clients a typed, cache-safe metadata source without runtime DESCRIBE calls or duplicated labels and formats.

Delivered

  • Generate metric-views.ts with MetricRegistry augmentation and a build-inlined metricViewsMetadata constant.
  • Accept generated metadata in the Analytics plugin and attach only the requested measures and dimensions to each result.
  • Keep metadata outside SQL generation and cache identity, applying the latest metadata after cached query execution.
  • Migrate the generated artifact from .d.ts to .ts and remove stale declarations safely.
  • Preserve committed generated artifacts during warehouse outages and require every configured artifact before using the fallback.
  • Cover generator output, payload scoping, cache behavior, migration, and outage handling with focused tests.

Base for #488.

Signed-off-by: Atila Fassina <atila@fassina.eu>
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 813 KB (+3.8 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 835 KB (+3.2 KB) 291 KB (+1.3 KB)
Type declarations 307 KB (+1.4 KB) 106 KB (+671 B)
Source maps 1.6 MB (+6.6 KB) 547 KB (+2.6 KB)
Other 11 KB 3.7 KB
Total 2.8 MB (+11 KB) 947 KB (+4.6 KB)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 87 KB (+466 B) 2.5 KB 90 KB (+466 B) external 285 KB (+1.2 KB)
./beta 45 KB 458 B 45 KB external 129 KB
./type-generator 20 KB (+320 B) 0 B 20 KB (+320 B) external 57 KB (+568 B)

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 83 KB
. utils.js initial 4.0 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 29 KB
./beta stream-manager.js initial 5.8 KB
./beta wide-event-emitter.js initial 3.2 KB
./beta databricks.js initial 3.0 KB
./beta configuration.js initial 2.1 KB
./beta service-context.js initial 1.3 KB
./beta client.js initial 431 B
./beta client-options.js initial 220 B
./beta supervisor-api.js lazy 193 B
./beta databricks.js lazy 142 B
./beta index.js lazy 123 B
./type-generator index.js initial 20 KB

@databricks/appkit-ui

npm tarball (packed): 305 KB (+66 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 360 KB 119 KB
Type declarations 205 KB (+206 B) 74 KB (+66 B)
Source maps 686 KB (+14 B) 224 KB (+5 B)
CSS 16 KB 3.3 KB
Total 1.2 MB (+220 B) 422 KB (+71 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 4.3 KB 49 KB 54 KB 208 KB 12 KB
./js/beta 20 B 0 B 20 B 0 B 0 B
./react 429 KB 49 KB 478 KB 1.3 MB 168 KB
./react/beta 20 B 0 B 20 B 0 B 0 B

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 4.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 20 B
./react index.js initial 427 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 20 B

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the AppKit type-generation + analytics SSE pipeline to carry Unity Catalog Metric View display metadata (labels/formats/types) from build-time generated artifacts into metric query SSE results, so clients can render consistently without runtime DESCRIBE calls and without polluting cache identity.

Changes:

  • Adds build-generated Metric View metadata types and threads an optional per-column metadata map through the shared analytics SSE wire contract.
  • Migrates the generated Metric Views artifact from metric-views.d.ts to metric-views.ts (module augmentation + runtime metricViewsMetadata export), including upgrade/cleanup behavior and Vite/CLI plumbing.
  • Updates analytics metric route handling to slice injected metadata to requested columns and stamp it onto result messages after cached execution, with focused tests and docs updates.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
packages/shared/src/sse/analytics.ts Extends SSE result contract to optionally include per-column metric metadata.
packages/shared/src/metric-metadata.ts Introduces shared MetricColumnMeta / MetricViewsMetadata types.
packages/shared/src/index.ts Re-exports metric metadata types from the shared package barrel.
packages/shared/src/cli/commands/generate-types.ts Updates CLI logging to reference metric-views.ts artifact name.
packages/shared/src/cli/commands/generate-types.test.ts Adjusts tests for the renamed metric artifact.
packages/appkit/src/type-generator/vite-plugin.ts Updates Vite plugin docs/behavior and adds validation rejecting .d.ts mvOutFile.
packages/appkit/src/type-generator/tests/vite-plugin.test.ts Updates tests for .ts mvOutFile and adds rejection coverage for .d.ts.
packages/appkit/src/type-generator/tests/unreachable-warehouse-gate.test.ts Refines tests around the blocking fallback gate via the real query path.
packages/appkit/src/type-generator/tests/sync-metric-views-types.test.ts Updates metric artifact expectations to .ts and adds stale .d.ts cleanup test.
packages/appkit/src/type-generator/tests/mv-registry.test.ts Refreshes comments/fixtures and adds runtime metricViewsMetadata emission assertions.
packages/appkit/src/type-generator/tests/index.test.ts Updates end-to-end typegen tests for .ts metric artifact + committed-artifact gating.
packages/appkit/src/type-generator/tests/snapshots/mv-registry.test.ts.snap Snapshot updates for the new generated header + runtime metadata export.
packages/appkit/src/type-generator/query-registry.ts Simplifies environmental-failure tracking for blocking-mode gate behavior.
packages/appkit/src/type-generator/mv-registry/render-types.ts Emits runtime metricViewsMetadata alongside type augmentation and refactors metadata rendering.
packages/appkit/src/type-generator/index.ts Tightens fallback gating to require all configured committed artifacts and adds stale .d.ts sweep.
packages/appkit/src/type-generator/errors.ts Simplifies blocking failure classification comments/structure.
packages/appkit/src/plugins/analytics/types.ts Extends analytics config and stream message types to include optional metric metadata.
packages/appkit/src/plugins/analytics/tests/metric.test.ts Adds coverage for metadata stamping/scoping and cache behavior invariants.
packages/appkit/src/plugins/analytics/mv/metadata.ts Implements selectMetricMetadata helper to scope metadata to requested columns.
packages/appkit/src/plugins/analytics/mv/index.ts Re-exports selectMetricMetadata from the metric-view module surface.
packages/appkit/src/plugins/analytics/analytics.ts Stamps scoped metadata onto result messages after cached query execution.
packages/appkit-ui/src/react/hooks/types.ts Adds empty MetricRegistry interface for generated module augmentation to target.
packages/appkit-ui/src/react/hooks/index.ts Exports MetricRegistry from the hooks entrypoint.
docs/docs/development/type-generation.md Updates docs for .ts metric artifact, committed fallback rules, and metadata export usage.
Suppressed comments (1)

packages/appkit/src/type-generator/mv-registry/render-types.ts:221

  • Metric keys are emitted as plain object-literal keys in the runtime metricViewsMetadata. If a metric key is ever "__proto__" (or similar), it can trigger object-literal prototype semantics. Emit the key as a computed property (["..."]) so it’s always treated as a data property.
  const entries = schemas
    .map((schema) => {
      const measures = renderMetadataValueMap(schema.measures, "    ");
      const dimensions = renderMetadataValueMap(schema.dimensions, "    ");
      return `  ${JSON.stringify(schema.key)}: {
    measures: ${measures},
    dimensions: ${dimensions},
  }`;

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/appkit/src/type-generator/vite-plugin.ts
Comment thread packages/appkit/src/type-generator/index.ts Outdated
Comment thread packages/appkit/src/plugins/analytics/mv/metadata.ts
Comment thread packages/appkit/src/type-generator/mv-registry/render-types.ts
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 31008869618 -R databricks/appkit -n appkit-template-0.53.1-pr.2f9778e-mv-metadata-pipeline-512 -D appkit-pr-512 \
  && unzip -o "appkit-pr-512/appkit-template-0.53.1-pr.2f9778e-mv-metadata-pipeline-512.zip" -d "appkit-pr-512" \
  && databricks apps init --template "appkit-pr-512"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

Comment and documentation cleanup only; no behavior change.

- Collapse the "metadata never affects SQL or cache identity" invariant
  from six sites down to the one that owns it (selectMetricMetadata).
- Reduce the duplicated "never emit a side-effect import" rationale to
  the emitter plus the test that asserts it.
- Drop caps-emphasis and compress the render-types helper preambles to
  match the density of their pre-existing siblings.
- De-narrate the cache-hit metadata test, which described the branch's
  own development arc rather than the invariant.
- Fix a broken docs cross-reference that pointed at hook and
  format-utility wiring the analytics plugin docs do not contain.
- Normalize metric-metadata.ts JSDoc and export spacing to the
  conventions used elsewhere in packages/shared.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
…he write

The stale-sibling sweep unlinked metric-views.d.ts unconditionally once
the out file was a .ts. In blocking mode a degraded pass suppresses the
replacement write, so an app still carrying a legacy metric-views.d.ts
lost its only committed metric types and the --wait gate had nothing to
fall back on.

Guard the sweep on the new file actually existing, and pin the behavior
with a test covering the suppressed-write path.

Signed-off-by: Atila Fassina <atila@fassina.eu>
Comment-only follow-up to b58b51c; no behavior change.

Collapse the stale metric-views.d.ts sweep comment from eight lines to
four, dropping the re-derivation of the emitter's type-only-import
contract that render-types.ts already documents, and describe the
existsSync guard by what it observably does rather than restating a
rationale the derived staleDts path does not support.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>

@pkosiec pkosiec left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed, before the merge, please run deslop to reduce/get rid of some of the overly descriptive / unnecessary comments. Thanks!

Comment thread packages/shared/src/metric-metadata.ts
The .d.ts guard threw with an appKitAnalyticsTypesPlugin prefix, but the
exported plugin is appKitTypesPlugin, sending anyone tracing the failure
after a name the codebase no longer has.

Signed-off-by: Atila Fassina <atila@fassina.eu>
MetricColumnMeta and the build-time MetricColumnMetadata differed only in
a suffix, so the pair read as if Meta vs Metadata encoded the wire/build
split. It does not: MetricViewsMetadata is wire-side too. Reviewers went
looking for the distinction and found nothing.

Name the wire type after what it carries — the display attributes the
doc comment already describes — and put the difference in the prefix,
where the two names no longer collide at a glance.

Signed-off-by: Atila Fassina <atila@fassina.eu>
PR #511 landed the same committed-types gate this branch had reworked.
Both wanted the metric surface to participate in the gate; #511's version
is strictly more precise, tracking the failure per surface so a committed
artifact for one surface can never stand in for a missing one elsewhere.

Resolve to #511's design and drop this branch's all-artifacts variant
(hasRequiredCommittedTypes and the now-unread metricTypesRequired flag).
Keep the .ts rename on top: the gate message says "type files" rather
than ".d.ts files", the two fixtures point at metric-views.ts, and the
committed-metric-types fixture carries the runtime const so a preserved
fallback stays a loadable module.

Signed-off-by: Atila Fassina <atila@fassina.eu>
…omments

Third deslop pass, covering the comments the earlier two did not reach.
Each of these said the same thing more than once:

- the "it's a real .ts because it carries a runtime const" rationale was
  stated in three places; keep it where the header is emitted and where
  the .d.ts path is rejected, drop the restatement
- the SSE metadata field re-derived the "keep client validation cheap"
  reasoning the adjacent interface already documents
- the cache-hit rationale appeared at both the definition and the yield
  site; keep the definition-site one
- "value twin" was echoed across three consecutive render helpers

Comment-only: no behavior, no generated output, no test changes.

Signed-off-by: Atila Fassina <atila@fassina.eu>
@atilafassina
atilafassina merged commit ba3d3ed into main Aug 5, 2026
10 checks passed
@atilafassina
atilafassina deleted the mv-metadata-pipeline branch August 5, 2026 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants