Skip to content

Entrypoint perf: cold-cache dagger call is 16.4s for a one-line function — trace breakdown + ideas to get under 10s #65

Description

@TomChv

What this is

I benchmarked a module whose only function returns a string literal, on a pruned local cache, against v1.0.0-beta.14 with a generated dang entrypoint:

@object()
export class HelloWorld {
  @func()
  hello(): string {
    return "hello"
  }
}

The function does no work and nothing is cached, so every second measured is Dagger preparing the execution environment — none of it is user code. Numbers are from a Docker engine on linux/arm64 (macOS host).

Repro tree and raw data: TomChv/dagger-playground → typescript/benchmarks

  • beta-entrypoint/ — the generated tree, including entrypoint/main.dang
  • README.md — method and results
  • analyze.md — full per-span breakdown (also covers the v0.21.9 and beta-without-entrypoint baselines, which I'm leaving out here since they're not the current state of the SDK)

Any table below can be reproduced with dagger --x-release=1.0.0-beta.14 trace <traceID> -vvv -d --progress=plain.

Numbers

Command Cold cache Trace
dagger -m hello-world api functions 2.2s 425e77dc…
dagger -m hello-world api call hello 16.4s 231beb44…

functions is essentially solved. types() is a literal evaluated in the engine, so no container is built, nothing is pulled and no process starts — 1.5s of the 2.2s is just uploading the workspace directory (and that's inflated here: this benchmark repo holds three copies of the module with their full clients//sdk/ trees).

This issue is about the other one.

Where the 16.4s goes

Span Time
load workspace: . 1.2s
helloWorld (constructor) → call module entrypoint 9.1s
├─ requireGenerated (7 × Directory.exists) 0.1s
├─ Container.from(node:24.13.1-alpine) 4.9s (4.1s pull + 0.8s unpack)
├─ apk add --no-cache ca-certificates 0.9s
├─ npm install -g tsx@4.22.4 1.8s
├─ yarn install --prod 1.4s
├─ tsx __dagger.dispatch.ts engine-call (constructor) 1.0s
└─ not attributed to any child span ~1.9s
HelloWorld.hellocall module entrypoint 5.9s
├─ tsx __dagger.dispatch.ts engine-call 1.0s
└─ not attributed to any child span ~4.7s

Two things to note before reading the ideas:

  • The exec chain is already parallel. apknpm i -g tsx and yarn install branch off base and overlap: 5.1s of serial exec time lands in ~2.3s of wall clock. There is no win left in reordering it.
  • ~6.6s — 40% of the run — is in call module entrypoint with no child span accounting for it. That is the single largest line item and I can't explain it from the trace alone.

Budget

16.4s → under 10s means finding ~7s. Here's what I'd look at, roughly in order of expected payoff.


1. Instrument call module entrypoint — ~6.6s unexplained

Every child span under the second call module entrypoint is either 0.0s or the 1.0s tsx exec, yet the parent reports 5.8s. Same shape on the constructor dispatch (~1.9s). Whatever this is, it's bigger than the process it wraps, and it's the difference between hitting 10s and not.

My hypothesis is that each call() re-evaluates the whole dang program — base(), dependencies(), runtime() — and the engine re-derives cache keys over workspace.directory("/", exclude: ["**/node_modules"]), i.e. the entire workspace, on every dispatch. That would also explain why the second dispatch is worse than the first despite every layer being warm.

Worth adding spans before anything else, because ideas 2–7 are all guesswork until this is attributed.

2. Mount the module directory, not the workspace root

dangRuntimeChain mounts workspace.directory("/", exclude: ["**/node_modules"]) — the whole workspace — then withWorkdir into the module. For a monorepo that's a large directory hashed on every call to produce a mount the module reads one subtree of.

If §10.3 of design/module-entrypoint.md resolves toward mounting the module source, this gets much cheaper. If the workspace root genuinely has to be reachable (root package.json, pnpm-workspace.yaml), then narrowing the mount to module dir + the manifest files above it would still cut most of it.

Expected: unknown until (1), but this is my prime suspect for the 6.6s.

3. Don't dispatch a no-arg constructor

helloWorld is a full dispatch — container build + tsx process — whose entire output is {}. HelloWorld has no user-defined constructor; the SDK synthesized one (dangObjectEntry always emits withConstructor for the main object).

Generation time already knows whether the user wrote a constructor and whether it takes arguments. When it didn't, call() could return {} directly from dang without touching runtime():

pub call(..., fnName: String!, ...): JSON! {
  if (receiverType == "HelloWorld" && fnName == "") {
    {{}}
  } else {
    ... exec ...
  }
}

Expected: ~2–5s here (one tsx process at 1.0s plus one dispatch's worth of whatever (1) turns out to be). It compounds on chained calls — today every object hop in dagger call a b c costs a process.

4. Drop the dead typescript pin — the default module should install nothing

config-updater/main.go:295 still writes dependencies.typescript into every generated package.json. My generated module has exactly two dependencies:

{
  "dependencies": {
    "typescript": "5.9.3",
    "@dagger.io/dagger": "file:./clients/dagger"
  }
}

Since #42 the bundle contains zero from "typescript" imports, so that 22MB is downloaded and installed on every cold call to be never imported. This is already Rollout step 1 in the design doc — the trace just puts a price on it (most of yarn install --prod, 1.4s).

The follow-on is the interesting half: with the pin gone, the default generated module's only dependency is a file: specifier pointing inside its own directory. No package manager is needed to resolve that. dangDependenciesChain could detect at generation time that the closure is local-only and emit a directory assembly instead of an install exec:

directory.withDirectory("@dagger.io/dagger", workspace.directory("<mod>/clients/dagger"))

Expected: ~1.4s, and it removes the registry from the hot path for the default template — which also answers §10.8 (registry reachability) for the common case.

5. Publish a prebuilt runtime image

Container.from(node) 4.9s + apk add ca-certificates 0.9s + npm install -g tsx@4.22.4 1.8s. §8.1 already lists this "in reserve": a digest-pinned node + tsx + ca-certificates image published from this repo folds three steps into one pull.

Expected: ~1.5–2s, and it removes npm from the first-call path too. Cost is a release artifact to maintain and version alongside dangNodeImageRef/dangTsxVersion.

6. Make ca-certificates conditional

apk add --no-cache ca-certificates + NODE_OPTIONS=--use-openssl-ca is in dangBaseChain unconditionally, for custom-CA support the large majority of modules never use. alpine already ships ca-certificates-bundle, so the public-CA case is covered without it.

Expected: 0.9s. Subsumed by (5) if that lands. Needs a check on what breaks for users behind a corporate proxy.

7. Cheaper dispatch than a tsx process

1.0s per dispatch is mostly tsx starting up and transforming. Node 24 can't replace it — type stripping refuses legacy decorators, as §8.1 notes. But the transform is deterministic given the source, so it could move out of the hot path: emit a precompiled __dagger.dispatch.js (plus compiled sources) at dagger generate, and exec plain node. That trades a slower dagger generate and a regeneration-is-load-bearing footgun (§10.6) for ~1.0s × number of dispatches.

Probably only worth it behind a flag, or once (3) has reduced the dispatch count to one.


Rough arithmetic

today 16.4s
− (3) skip the constructor dispatch ~2–5s
− (4) no install ~1.4s
− (5) prebuilt image (includes (6)) ~1.5–2s
~8–11.5s

So the mechanical wins roughly get there, but only barely and only at the optimistic end — which is why (1) matters most. If the 6.6s is real per-dispatch overhead rather than a tracing gap, fixing it alone gets under 10s and everything else is margin.

Ask

Does the ~6.6s in call module entrypoint match anything known engine-side, or is it worth me instrumenting the dang evaluation path and re-measuring? Happy to take any of (3)–(6) if the direction looks right.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions