contrib/dbt-factory: add job cluster compute option + Lakeflow Jobs rename

contrib/dbt_factory/CONTRIBUTING.md

@@ -0,0 +1,110 @@
+# Contributing to dbt-factory
+
+Thanks for contributing! This guide covers the **dbt-factory** template
+([`contrib/templates/dbt-factory`](../templates/dbt-factory)) and this ready-to-run example
+([`contrib/dbt_factory`](.)). Most development happens from this example directory — it has the
+`Makefile`, the unit tests, and the end-to-end test.
+
+## First principles
+
+- Keep a generated project **self-contained and dependency-light** — it should run with just `uv`
+  and the Databricks CLI, with no per-model YAML to maintain.
+- The factory core under `src/databricks_dbt_factory/` is **vendored**, not developed here — see
+  [Vendored factory](#vendored-factory).
+- **Serverless is the default** compute and the **job cluster** path is opt-in; keep both working.
+
+## Local setup
+
+Requirements:
+
+- Python (`>=3.10,<3.13`, per `pyproject.toml`)
+- The [`uv`](https://docs.astral.sh/uv/) package manager
+- The [Databricks CLI](https://docs.databricks.com/dev-tools/cli/databricks-cli.html)
+
+Install the development environment:
+
+```
+$ make setup      # == uv sync --dev  (into .venv, from the frozen uv.lock)
+```
+
+Prefer the `make` targets over calling `uv`/`dbt`/`databricks` directly where one exists:
+`make manifest`, `make validate`, `make deploy`, `make run`, `make test`, `make test-e2e`.
+
+## Before you open a PR (Definition of Done)
+
+Run these locally — they mirror the CI gates plus the manual checks:
+
+1. **Format** — `uv run ruff format .` The `fmt` CI runs `ruff format --check`, so unformatted code
+   fails the build.
+2. **Unit tests** — `make test` (offline; no workspace needed). This is the fast gate.
+3. **End-to-end** — `make test-e2e` deploys, runs, verifies, and tears down a real job for **both**
+   compute modes (serverless and job cluster). It is **not** a CI gate, so run it locally before
+   merging any change to the factory or template. It needs `DATABRICKS_*` / `DBT_FACTORY_*`
+   environment variables — see [`tests/e2e/README.md`](tests/e2e/README.md).
+4. **Snapshot** — if you intentionally changed the generated task output, refresh the saved snapshot
+   with `make test-update-expected-tasks` and commit the result.
+5. **Docs** — update the relevant README(s) for any user-facing change.
+
+## Testing
+
+- **`make test`** — runs the vendored factory's unit tests plus an offline test of the PyDABs
+  integration against the committed manifest; no workspace is required. One test compares the
+  generated tasks with `tests/test_data/expected_tasks.json`, so unintended changes to the generated
+  job fail the suite.
+- **`make test-e2e`** — the real end-to-end check (generate → deploy → run → verify → destroy) for
+  both compute modes. See [`tests/e2e/README.md`](tests/e2e/README.md) for the required environment
+  and details.
+
+### Scaffolding check
+
+The unit tests run against the committed (pre-rendered) example, so they do **not** exercise the
+template's prompts or rendering. When you change the template, scaffold a throwaway project with
+`databricks bundle init` to confirm it still scaffolds correctly — do it once per compute mode
+(`use_serverless = yes` and `= no`):
+
+```
+# from a neutral directory (not inside a bundle), pointing at the template:
+$ databricks bundle init <repo-root>/contrib/templates/dbt-factory --output-dir /tmp/dbt-factory-check
+$ cd /tmp/dbt-factory-check/<project_name>
+$ make setup && make manifest && make test
+```
+
+This needs a configured Databricks CLI profile — the template resolves workspace values (such as
+`smallest_node_type` and `workspace_host`) at init time. `make test-e2e` already runs `bundle init`
+for both modes end-to-end, but this quick check catches rendering/prompt errors without a full
+deploy.
+
+## Keeping the template and example in sync
+
+The template ([`contrib/templates/dbt-factory`](../templates/dbt-factory)) and this example are two
+copies of the same project (the template's files are `*.tmpl`; this example is a rendered, serverless
+copy). When you change a shared file — the factory core, `resources/__init__.py`, the tests, the
+`Makefile`, `dbt_profiles/profiles.yml` — update **both** so the template's serverless rendering
+still matches this example.
+
+### Vendored factory
+
+`src/databricks_dbt_factory/` is vendored from
+[databricks-dbt-factory](https://github.com/mwojtyczka/databricks-dbt-factory) and pinned to the
+commit recorded in [`NOTICE`](NOTICE). The `dbt-factory vendor-sync` CI check enforces that:
+
+- each vendored file is byte-identical to that upstream commit, and
+- the example and template copies are byte-identical to each other.
+
+So **don't edit the factory here** — make the change upstream (see the factory's
+[contributing guide](https://github.com/mwojtyczka/databricks-dbt-factory/blob/main/CONTRIBUTING.md),
+which uses its own Hatch-based `make dev`/`fmt`/`lint`/`test`/`integration` workflow), then re-vendor
+into both copies and update the commit/version in both `NOTICE` files.
+
+## Opening the PR
+
+Unless you have write access to
+[`databricks/bundle-examples`](https://github.com/databricks/bundle-examples), work from a **fork**:
+
+1. Fork the repo and clone your fork (keep its `main` in sync with upstream). If you have write
+   access, you can branch directly in the upstream repo instead.
+2. Create a feature branch from `main`, make your change, and run the Definition-of-Done checks
+   above.
+3. Push the branch (to your fork, or upstream if you have access) and open a PR against
+   `databricks/bundle-examples` `main`, with a clear description and `Resolves #NNN` if it fixes an
+   issue.

contrib/dbt_factory/README.md

@@ -1,7 +1,7 @@
 # dbt_factory
 
 This example runs a [dbt](https://docs.getdbt.com/) project on Databricks as a
-**Databricks Workflow with one task per dbt object** (model, seed, snapshot, test) instead of
+**Databricks Lakeflow Job with one task per dbt object** (model, seed, snapshot, test) instead of
 running the whole project as a single opaque task.
 
 It does this by combining two pieces:
@@ -20,13 +20,13 @@ the dbt manifest each time you deploy.
 
 ## Why one task per dbt object?
 
-By default dbt's integration with Databricks Workflows treats the whole project as a single
+By default dbt's integration with Databricks Lakeflow Jobs treats the whole project as a single
 task — a black box. Expanding it into one task per object gives:
 
 * **Faster execution** — independent models run in parallel, and the notebook task type runs dbt
   from a pre-built serverless base environment, avoiding a dependency install on every task.
 * **Visibility & simplified troubleshooting** — pinpoint and fix issues at the model level right
-  in the Databricks Workflows UI.
+  in the Databricks Lakeflow Jobs UI.
 * **Enhanced logging & notifications** — per-task logs and precise, model-level error alerts.
 * **Improved retriability** — retry only the failed model tasks without rerunning the whole project.
 * **Seamless testing** — dbt data tests run as their own tasks right after each model finishes,
@@ -40,7 +40,7 @@ for more.
 ## How it works
 
 The [`dbt-factory` template](../templates/dbt-factory) scaffolds a self-contained project.
-From then on, each `databricks bundle deploy` regenerates the Workflow from your current dbt
+From then on, each `databricks bundle deploy` regenerates the job from your current dbt
 manifest — add or remove a model and the task graph follows on the next deploy, with no per-model
 YAML to maintain.
 
@@ -56,7 +56,7 @@ flowchart TD
       E --> F["PyDABs load_resources reads the<br/>manifest and generates the job"]
     end
     subgraph runtime["At run time — serverless"]
-      G["Databricks Workflow:<br/>one task per model / seed / snapshot / test"] --> H["Each task triggers dbt<br/>via the runner notebook"]
+      G["Databricks Lakeflow Job:<br/>one task per model / seed / snapshot / test"] --> H["Each task triggers dbt<br/>via the runner notebook"]
       H --> I[("SQL warehouse")]
     end
     B --> C
@@ -169,55 +169,6 @@ The version is shipped as a small `dbt_serverless_env.yaml` [base environment](h
 that the bundle generates and syncs on every deploy (git-ignored), so Databricks pre-builds the
 environment once instead of installing dbt on every task.
 
-## Migrating an existing dbt project
-
-Bring your own dbt project by **generating a fresh project from the template and moving your dbt
-files into it.** You don't touch dependencies, the vendored factory, or any paths — the generated
-project already ships all of that.
-
-1. Generate a new project (or copy this `dbt_factory` example):
-
-   ```
-   $ databricks bundle init https://github.com/databricks/bundle-examples --template-dir contrib/templates/dbt-factory
-   ```
-
-2. Remove the starter models and copy your dbt sources into the matching `src/` subdirectories:
-
-   ```
-   $ rm -r src/models/example
-   # Copy whichever of these your project has (skip the ones you don't use):
-   $ cp -R /path/to/your/dbt/models/*     src/models/
-   $ cp -R /path/to/your/dbt/seeds/*      src/seeds/
-   $ cp -R /path/to/your/dbt/snapshots/*  src/snapshots/
-   $ cp -R /path/to/your/dbt/macros/*     src/macros/
-   $ cp -R /path/to/your/dbt/tests/*      src/tests/
-   ```
-
-   The generated `dbt_project.yml` already points `model-paths`, `seed-paths`, etc. at these
-   `src/` folders, so your files are picked up as-is. Merge any model/seed configuration from your
-   own `dbt_project.yml` into the generated one (keep the generated `name`/`profile`), and remove
-   the leftover `models: dbt_factory: example:` block that referenced the deleted starter models —
-   otherwise `dbt parse` warns that those config paths don't apply to any resource. If you use dbt
-   packages, copy your `packages.yml` to the project root too: `make manifest` installs them
-   (`dbt deps`) and the bundle syncs the resulting `dbt_packages/` to the workspace, so the job
-   never installs packages at runtime.
-
-3. Point `dbt_profiles/profiles.yml` at your warehouse (`http_path`, `catalog`, `schema`). Leave
-   the `host`/`token` lines as they are — the runner notebook sets those at runtime.
-
-4. Generate the manifest and deploy:
-
-   ```
-   $ make setup
-   $ make manifest      # dbt parse -> target/manifest.json
-   $ databricks bundle deploy --target dev
-   ```
-
-That's the whole migration: no dependency wrangling and no path edits, because your project keeps
-the generated layout (dbt project at the bundle root, factory under `src/`). If you'd rather keep
-your project's existing directory structure instead of `src/`, edit the `*-paths` in
-`dbt_project.yml` to point at your folders — nothing else changes.
-
 ## Tests
 
 ```

contrib/dbt_factory/tests/e2e/README.md

@@ -1,11 +1,12 @@
 # End-to-end test
 
 `make test-e2e` (from the example root) is the check that a change to the factory won't break real dbt
-execution. It generates a fresh project from the [`dbt-factory` template](../../../templates/dbt-factory)
-with the factory's default options, points it at **your** Databricks workspace, drops in the fixture
-dbt project in this directory, then **deploys the factory-generated job, runs it, verifies the
-output, and tears everything down**. Every run destroys its bundle and drops its schema, pass or
-fail, so nothing is left behind.
+execution. It generates a fresh project from the [`dbt-factory` template](../../../templates/dbt-factory),
+points it at **your** Databricks workspace, drops in the fixture dbt project in this directory, then
+**deploys the factory-generated job, runs it, verifies the output, and tears everything down**. It
+does this once for **each compute mode** — serverless and a job cluster — so both renderings of the
+template are deployed and executed for real. Every run destroys its bundle and drops its schema, pass
+or fail, so nothing is left behind.
 
 Unlike `make test` (fast, offline unit tests), this one deploys and runs on a real workspace — so
 it's not a CI gate; run it locally before merging a change to the factory.
@@ -14,6 +15,10 @@ it's not a CI gate; run it locally before merging a change to the factory.
 
 - A Databricks CLI profile for your workspace: `databricks auth login --host <your-workspace-url>`.
 - A SQL warehouse, and a catalog you can create schemas/tables in.
+- Permission to **create clusters** (the cluster-create entitlement) — the job-cluster mode
+  provisions a single-node job cluster. The serverless mode doesn't need this; without the
+  entitlement the serverless half passes and the job-cluster half fails with a `PERMISSION_DENIED`
+  cluster-creation error.
 
 ## Run it
 

contrib/dbt_factory/tests/e2e/run_e2e.py

@@ -1,17 +1,22 @@
 """
 End-to-end test for the dbt-factory example.
 
-It generates a fresh project from the ``dbt-factory`` template with the factory's default options,
-points it at YOUR workspace, drops in the fixture dbt project next to this file, then **deploys the
-factory-generated job, runs it, and verifies it succeeded** — before tearing everything down. This
-is the check that a change to the factory won't break real dbt execution.
+It generates a fresh project from the ``dbt-factory`` template, points it at YOUR workspace, drops
+in the fixture dbt project next to this file, then **deploys the factory-generated job, runs it, and
+verifies it succeeded** — before tearing everything down. It runs the whole flow once per **compute
+mode** (serverless and a job cluster), so a change to the factory is exercised end-to-end on both.
+This is the check that a change to the factory won't break real dbt execution.
 
 Run it with ``make test-e2e`` from the example root. Required environment:
 
     DATABRICKS_CONFIG_PROFILE   CLI profile for your workspace (auth + host).
     DBT_FACTORY_HTTP_PATH           SQL warehouse HTTP path, e.g. /sql/1.0/warehouses/<id>.
     DBT_FACTORY_CATALOG             Catalog to create the throwaway schema + tables in (write access).
 
+The job-cluster mode additionally needs permission to create clusters (the cluster-create
+entitlement) — the serverless mode does not. Without it, the serverless half passes and the
+job-cluster half fails at deploy/run with a PERMISSION_DENIED cluster-creation error.
+
 Optional:
     DBT_FACTORY_SCHEMA_PREFIX       Schema-name prefix (default: dbt_factory_e2e). A unique
                                 <prefix>_<timestamp> schema is created and dropped per run.
@@ -108,18 +113,19 @@ def _sql(cfg: Config, statement: str, schema: str | None = None) -> list[list]:
     return payload.get("result", {}).get("data_array", []) or []
 
 
-def _init_project(cfg: Config, out_dir: Path, schema: str) -> Path:
-    # Set only the values that point the project at this workspace. Everything else (test bundling,
-    # environment key, extra dbt options) is left at the template's defaults — the factory's
-    # out-of-the-box behavior, which is exactly what we want the e2e to exercise.
+def _init_project(cfg: Config, out_dir: Path, schema: str, use_serverless: str, project_name: str) -> Path:
+    # Set the values that point the project at this workspace, plus the compute mode under test
+    # (use_serverless). Everything else (test bundling, environment key, extra dbt options) is left
+    # at the template's defaults — the factory's out-of-the-box behavior, which is what we exercise.
     config_file = out_dir / "init-config.json"
     config_file.write_text(
         json.dumps(
             {
-                "project_name": PROJECT_NAME,
+                "project_name": project_name,
                 "default_catalog": cfg.catalog,
                 "dev_schema": schema,
                 "http_path": cfg.http_path,
+                "use_serverless": use_serverless,
             }
         )
     )
@@ -138,7 +144,7 @@ def _init_project(cfg: Config, out_dir: Path, schema: str) -> Path:
         ],
         cwd=out_dir,
     )
-    return out_dir / PROJECT_NAME
+    return out_dir / project_name
 
 
 def _fill_fixture(project: Path) -> None:
@@ -200,17 +206,20 @@ def _verify_output(cfg: Config, schema: str) -> list[str]:
     return failures
 
 
-def run(cfg: Config) -> bool:
-    schema = f"{cfg.prefix}_{time.strftime('%Y%m%d_%H%M%S')}"
-    job = f"{PROJECT_NAME}_job"
+def run(cfg: Config, label: str, use_serverless: str) -> bool:
+    schema = f"{cfg.prefix}_{label}_{time.strftime('%Y%m%d_%H%M%S')}"
+    # Each mode gets its own project (and therefore its own bundle path + job name), so the two runs
+    # are fully isolated — a silently-failed teardown of one can't collide with the other's deploy.
+    project_name = f"{PROJECT_NAME}_{label}"
+    job = f"{project_name}_job"
     work = Path(tempfile.mkdtemp(prefix="dbtfactory_e2e_"))
     project = None
-    print(f"\n===== dbt-factory e2e — schema {cfg.catalog}.{schema} =====")
+    print(f"\n===== dbt-factory e2e [{label}] — schema {cfg.catalog}.{schema} =====")
     try:
         print("  [1/6] create throwaway schema")
         _sql(cfg, f"CREATE SCHEMA IF NOT EXISTS {cfg.catalog}.{schema}")
         print("  [2/6] init project from template + drop in fixture")
-        project = _init_project(cfg, work, schema)
+        project = _init_project(cfg, work, schema, use_serverless, project_name)
         _fill_fixture(project)
         print("  [3/6] install deps + generate dbt manifest")
         _run(["uv", "sync", "--dev"], cwd=project)
@@ -248,9 +257,17 @@ def run(cfg: Config) -> bool:
 
 
 def main() -> None:
-    passed = run(Config())
-    print(f"\n===== e2e {'PASS' if passed else 'FAIL'} =====")
-    if not passed:
+    cfg = Config()
+    # Run the whole flow once per compute mode so both the serverless and the job-cluster
+    # rendering of the template are deployed and executed for real.
+    results = {
+        label: run(cfg, label, use_serverless)
+        for label, use_serverless in (("serverless", "yes"), ("job_cluster", "no"))
+    }
+    print("\n===== e2e summary =====")
+    for label, ok in results.items():
+        print(f"  {label:12} {'PASS' if ok else 'FAIL'}")
+    if not all(results.values()):
         sys.exit(1)
 
 

contrib/dbt_factory/tests/test_load_resources.py

@@ -94,8 +94,8 @@ def test_load_resources_registers_the_job():
 
 
 def test_dependency_pin_rejects_non_pypi_versions(monkeypatch):
-    # A local or dev build cannot be pip-installed from PyPI when Databricks builds the
-    # serverless environment, so the deploy must fail early with a clear message.
+    # A local or dev build cannot be pip-installed from PyPI, so the deploy must fail early
+    # with a clear message.
     for installed in ("1.9.0+custom", "1.13.0.dev0", "not-a-version"):
         monkeypatch.setattr(resources, "version", lambda name, v=installed: v)
         with pytest.raises(RuntimeError, match="PyPI"):