feat(appkit): add testing kit

[IamGalymzhan]

Testing Kit

1. Made PluginContext telemetry injectable — the one production edit. It now takes an optional { telemetry }, defaulting to the real provider for testing.
2. Created the @databricks/appkit/testing package — moved the fixtures out of tools/test-helpers.ts into packages/appkit/src/testing/
3. Added the two headline helpers — createTestPluginContext() (wraps the real PluginContext with fake tool providers) and expectStream(...).toEmit(...).
4. Shipped it to users + added docs page (docs/development/testing.md), a template example test.
5. Added new tests for analytics and genie plugins using this kit

[github-actions]

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 873 KB (+33 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	893 KB (+24 KB)	312 KB (+8.6 KB)
Type declarations	340 KB (+25 KB)	118 KB (+9.1 KB)
Source maps	1.7 MB (+57 KB)	586 KB (+20 KB)
Other	11 KB	3.7 KB
Total	3.0 MB (+106 KB)	1019 KB (+38 KB)

Per-entry composition (own code — deps external (as shipped))

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
.	88 KB (+15 B)	2.5 KB	91 KB (+15 B)	external	288 KB (+46 B)
./beta	49 KB	458 B	49 KB	external	143 KB
./testing	17 KB	0 B	17 KB	external	50 KB
./type-generator	21 KB	0 B	21 KB	external	60 KB

Chunks:

Entry	Chunk	Load	Size (gz)
.	index.js	initial	84 KB
.	utils.js	initial	4.0 KB
.	remote-tunnel-manager.js	lazy	2.5 KB
./beta	beta.js	initial	33 KB
./beta	stream-manager.js	initial	5.8 KB
./beta	wide-event-emitter.js	initial	3.2 KB
./beta	databricks.js	initial	3.0 KB
./beta	configuration.js	initial	2.1 KB
./beta	service-context.js	initial	1.3 KB
./beta	client.js	initial	431 B
./beta	client-options.js	initial	219 B
./beta	supervisor-api.js	lazy	194 B
./beta	databricks.js	lazy	142 B
./beta	index.js	lazy	122 B
./testing	index.js	initial	17 KB
./type-generator	index.js	initial	21 KB

@databricks/appkit-ui

npm tarball (packed): 342 KB — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	390 KB	130 KB
Type declarations	228 KB	83 KB
Source maps	753 KB	248 KB
CSS	16 KB	3.3 KB
Total	1.4 MB	464 KB

Per-entry composition (consumer bundle — deps bundled, peerDeps external)

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
./js	5.3 KB	49 KB	55 KB	208 KB	14 KB
./js/beta	20 B	0 B	20 B	0 B	0 B
./react	432 KB	49 KB	480 KB	1.3 MB	175 KB
./react/beta	1.0 KB	0 B	1.0 KB	0 B	1.9 KB

Chunks:

Entry	Chunk	Load	Size (gz)
./js	index.js	initial	5.2 KB
./js	chunk	initial	120 B
./js	apache-arrow	lazy	49 KB
./js/beta	beta.js	initial	20 B
./react	index.js	initial	430 KB
./react	tslib	initial	2.1 KB
./react	apache-arrow	lazy	49 KB
./react/beta	beta.js	initial	1.0 KB

[github-actions]

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 31701596628 -R databricks/appkit -n appkit-template-0.60.0-pr.45dec48-feat-testing-kit-530 -D appkit-pr-530 \
  && unzip -o "appkit-pr-530/appkit-template-0.60.0-pr.45dec48-feat-testing-kit-530.zip" -d "appkit-pr-530" \
  && databricks apps init --template "appkit-pr-530"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

[pkosiec]

Testing kit - review

I tried this kit two ways: read it closely, then used it to write real tests for the analytics and genie plugins. Short version: I'd adopt it. Wrapping the real PluginContext and faking only the edges (telemetry, tool providers, routes) gives you things a hand-rolled { executeTool } stub can't - real timeout composition, the real asUser token check, real route recording. attach() and OBO both worked cleanly in a fresh test.

One gap stood out in real use, and a few smaller things are worth a look. Details are in the inline comments; here's the map:

Biggest lever

• The kit tests generators well, but you can't assert a plugin's real SSE route - createMockResponse throws the written bytes away. Fixing this is the single change that makes plugin testing feel complete. (see comment on fixtures.ts)

Worth deciding before wide use

• vitest as a plain runtime dependency ships a test framework into every production install. An optional peer avoids that. (see package.json)

Convenience

• An obo option on createMockRequest (+ show it in the docs example).
• A resetTestCache() so the documented cache advice is actually followable.

Polish

• The shipped example test uses an any-cast instead of the documented attach().
• Two service-context helpers differ only by a create prefix.
• A useServiceContextMock() hook could remove the repeated beforeEach/afterEach block.

[pkosiec]

createMockResponse().write() throws the bytes away - it only flips headersSent. So when a plugin streams SSE from a route, there's no way to feed that output into expectStream. You're left scraping res.write.mock.calls and matching strings by hand.

Why it matters: most plugins stream from a route, not a bare generator. Today the kit tests generators well but can't assert what a real route emits. This was the #1 rough spot when I used the kit on the analytics plugin.

Suggestion: have createMockResponse collect the written chunks, then let expectStream read them directly:

await handler(req, res);
await expectStream(res).toEmit("warehouse_status", "result");

The cleanest shape is for expectStream to accept the mock response (it's the kit's own type) and parse its captured buffer as SSE. If you'd rather leave expectStream untouched, expose res.sseResponse() that returns a real Response and pass that.

One gotcha: don't hand expectStream the raw body string - a string is an iterable of characters, so it would be consumed one char at a time. It needs the mock response or a Response, not text.

[pkosiec]

vitest is a plain dependency, so it's installed into every app - including production installs that never import /testing. That pulls a whole test framework (and a second copy of vite) into code that never runs in production.

Why it matters: bigger installs, slower cold starts, and more supply-chain surface for nothing. The exact pin 3.2.4 can also clash with an app's own vitest, leaving two copies - and then the app's clearMocks/restoreMocks won't reset the spies this kit creates.

Heads-up: optionalDependencies would not fix this - npm and pnpm still install those by default (only --no-optional skips them).

Suggestion: make it an optional peer, and keep a dev copy for AppKit's own tests:

"peerDependencies":     { "vitest": ">=3" },
"peerDependenciesMeta": { "vitest": { "optional": true } },
"devDependencies":      { "vitest": "3.2.4" }

An optional peer isn't auto-installed, so it stays out of apps that don't test. Anyone who imports /testing is already writing vitest tests, so they have it - and there's only ever one vitest instance. (A devDependency alone won't work: it isn't installed for consumers, so their own tests couldn't resolve vitest.) Pick a range that covers the vitest majors you test against - the vi.fn/vi.spyOn APIs the kit uses are the same in v3 and v4.

Bigger alternative, if we'd rather /testing need no test framework at all: drop import { vi } from fixtures.ts and hand-roll the small spy surface it uses. More work - mockServiceContext relies on vi.spyOn(ServiceContext, ...) - so the optional peer is the smaller, idiomatic fix.

[pkosiec]

createMockRequest() ships with empty headers, so every on-behalf-of test has to hand-add the same two headers (x-forwarded-access-token, x-forwarded-user) to make asUser resolve. The cross-plugin example in the docs doesn't show them, so copying it throws Missing user token.

Why it matters: OBO is the kit's headline feature. The first thing a reader tries fails, and everyone repeats the same header boilerplate.

Suggestion: add an obo option to createMockRequest so OBO is one flag, not hand-rolled headers - it keeps a single entry point instead of a second function to discover:

createMockRequest({ obo: true });                  // default test user + token
createMockRequest({ obo: { userId: "alice" } });   // pick the user

It would set the two headers for you and still merge the other overrides. Then use it in the docs cross-plugin example. (A standalone createOboRequest(userId, token?) would also work, but the option keeps one request builder.)

[pkosiec]

The docs note the cache is a shared singleton: isolated between test files, but shared by every test within one file, and vitest does not clear it between those tests. The kit ships no helper to clear it.

Why it matters: a value one test caches can leak into the next test in the same file - a stale read or a false pass. The docs say to "clear the cache in beforeEach", but there is no API to do it. A reset is also handy within a single test - e.g. to check a cache miss, then a hit.

Suggestion: export an async resetTestCache() that wraps CacheManager.getInstanceSync().clear() (and no-ops when the cache isn't initialized yet, since getInstanceSync() throws otherwise). Show it in the beforeEach here.

[pkosiec]

This sets (plugin as any).context = mock.ctx instead of the documented await mock.attach(plugin).

Why it matters: this is the example people copy. The cast skips cache seeding, the isReady flip, and registry wiring - so pasted onto a plugin that reads this.cache, it breaks. In a fresh test attach() just works (I confirmed it while dogfooding).

Suggestion: use attach() here. If you genuinely need to skip the wiring, add a small documented setter so the shipped pattern isn't an any-cast.

[pkosiec]

createMockServiceContext builds a value and installs nothing; mockServiceContext installs spies and returns restore(). They sit next to each other and differ only by the create prefix.

Why it matters: in autocomplete they look interchangeable. Pick the builder by mistake and the real singleton stays live, so the test fails in a confusing way.

Suggestion: rename to say what each does (e.g. buildServiceContextState vs installServiceContextSpies), or drop the unused builder from the public exports. If you add useServiceContextMock() (see the other comment on this file), these two become its lower-level building blocks - name them so that reads clearly. Renaming is free now, while the kit is new.

[pkosiec]

Three edges the kit deliberately handles have no test pinning them:

• the Object.hasOwn guard here (a tool named constructor/toString must report "missing", not resolve to a prototype method),
• the dev-mode branch of the fake asUser (test-plugin-context.ts:277), which never runs because the suite is under NODE_ENV=test,
• the scalar / array / non-JSON / multi-line data: paths in parseSSEBody (expect-stream.ts:80).

Why it matters: each is a corner someone could quietly regress - the guard could be weakened to in/=== undefined and CI would stay green.

Suggestion: add one small case for each (a constructor tool name rejects; a NODE_ENV=development token-less request is allowed through; a data: 42 / data: [1] / data: plain text frame parses as expected).

[pkosiec]

Every suite that mocks the service context repeats the same three steps: call mockServiceContext() in beforeEach, hold the handle, and call restore() in afterEach.

Why it matters: it's boilerplate in every service-context test, and forgetting the restore() leaks spies into the next test - a confusing, order-dependent failure.

Suggestion (small enough to add here): ship a useServiceContextMock(options?) that registers the setup and teardown for you:

useServiceContextMock();   // one line: auto beforeEach setup + afterEach restore

It just wraps the existing mockServiceContext and calls vitest's beforeEach/afterEach, so it's a thin helper, not a rewrite. (This also leans on vitest being present - one more reason to settle it as the optional peer from the package.json comment.)

Two things to get right so it's actually usable: call it at the top of a describe, not inside a test (beforeEach/afterEach only register during collection); and return a live accessor, not the handle itself - each beforeEach builds fresh spies, so a returned snapshot goes stale. Hand back something like { get current() { ... } } so a test can still assert on the spies.

[pkosiec]

I think it should be placed under plugins docs? as "Testing plugins". What do you think?

[pkosiec]

Round 2 - the new dogfood tests

Thanks for adding these. Both pass, and the analytics one adds real coverage that was missing (cross-plugin executeTool OBO identity).

Two asks:

1. Don't keep them as separate *.kit.test.ts files. The testing kit should be the default way to test a plugin, not a parallel track. Fold these into analytics.test.ts / genie.test.ts and drop the .kit split. For genie, that also means merging with the existing (presence-only) SSE test into one ordered assertion on the real event names.
2. Trim the comments. Both files carry a lot of explanatory narration that reads as slop; a /deslop pass when merging would cut it to what's needed.

Bonus signal: these files independently re-hit round 1 - a hand-rolled capturing response (C1), a re-invented service-context mock (C8), and hand-rolled OBO headers (C3). Two authors needing the same workarounds is the strongest case yet to land those kit ergonomics.

[pkosiec]

Good tests, but two things about the file itself.

Don't keep them in a separate analytics.kit.test.ts. The kit should be the default way to test a plugin, not a parallel "kit" track. A separate file signals the kit is optional, splits the plugin's suite in two, and leaves it unclear where a new test should go. Fold these three tests into analytics.test.ts and delete this file - the coverage (cross-plugin executeTool OBO identity) is genuinely new, so keep it, just not as its own file.

Trim the narration. The docblock and inline comments explain the exercise at length; it reads as slop. A /deslop pass when you merge would cut it to the essentials.

Why it matters: if this is how we want people to test, the tests should look like normal plugin tests - same file, minimal ceremony - not a heavily annotated "kit demo".

[pkosiec]

Same as the analytics file: merge this into genie.test.ts rather than keep a separate genie.kit.test.ts, and trim the heavy comment narration (/deslop) while you do.

One extra reason to merge: genie.test.ts already tests this SSE path ("should call startConversation and stream SSE events"), but only checks event presence via mockRes.write.mock.calls.join("").toContain(...) - no ordering, brittle substrings. This file adds ordering via expectStream. Two tests for one path is worse than one good one.

Suggestion: replace that existing test's assertion block with expectStream(...).toEmit(...) on genie's real event names, and delete this file.

[pkosiec]

For whoever lands the kit changes: this local createCapturingResponse is exactly the createMockResponse gap from the first review (C1), now hand-rolled a second time. Two dogfood files needing the same bridge is the signal to ship a capturing response in the kit and drop these local copies.

[pkosiec]

Also re-invents mockServiceContext plus a manual restore - the boilerplate the proposed useServiceContextMock() (C8) removes. When this folds into genie.test.ts, use the kit fixture instead of a local stub.

[pkosiec]

When you merge this in, assert genie's real SSE event names. The fake connector here yields status/message/complete, but genie actually emits message_start/status/message_result/query_result (see the existing test). Ordering on invented names won't catch a reorder or drop of the real ones.

[pkosiec]

Minor: another hand-rolled OBO request with the two headers - more evidence for a createMockRequest({ obo }) option (C3).