Skip to content

[air] Show retry progress while streaming workload logs - #6778

Merged
caroline-db merged 2 commits into
mainfrom
air/retry-progress
Sep 22, 2026
Merged

caroline-db merged 2 commits into
mainfrom
air/retry-progress

Conversation

@caroline-db

@caroline-db caroline-db commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Changes

  • Show the configured retry policy when experimental air run --watch starts.
  • Emit each observed retry once in text and JSON output, detecting WAITING_FOR_RETRY early and preserving attempt-number detection across Bricklens and MLflow.

Why

Retry backoff was silent until Jobs created the next attempt, making active workloads appear hung after a failed attempt.

Tests

  • ./task checks
  • ./task lint (all Go linters passed with zero issues; the Python-only Ruff step could not download Ruff because PyPI DNS was unavailable, and this PR changes no Python files)
  • Focused retry tests passed normally and under the race detector
  • Bricklens-to-MLflow fallback test passed 20 consecutive runs
  • ./task build
  • Full test suite deferred to GitHub CI
Screenshot 2026-09-21 at 3 33 53 PM

This PR was written with Codex.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: e978741

Run: 35659083794

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux 276 15 4:00
✅​ aws windows 278 13 3:28
🔄​ azure linux 2 273 15 4:17
✅​ azure windows 277 13 3:26
✅​ gcp linux 276 15 4:39
✅​ gcp windows 278 13 3:24
Test Name azure linux
🔄​ TestFsCpFileToDir 🔄​f
🔄​ TestFsCpFileToDir/uc-volumes_to_uc-volumes 🔄​f
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:27 aws windows TestAccept
3:24 azure windows TestAccept
3:22 gcp windows TestAccept

@caroline-db caroline-db added the AIR Databricks AI Runtime CLI label Sep 21, 2026
@caroline-db
caroline-db marked this pull request as ready for review September 21, 2026 19:31
Comment thread experimental/air/cmd/logstream.go
@caroline-db
caroline-db added this pull request to the merge queue Sep 22, 2026
Merged via the queue into main with commit 2b66bfc Sep 22, 2026
34 checks passed
@caroline-db
caroline-db deleted the air/retry-progress branch September 22, 2026 03:03
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 2b66bfc

Run: 35681767988

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux 1653 1084 151:46
✅​ azure linux 1466 1133 117:55
🔄​ gcp linux 3 1449 1137 140:53
Test Name gcp linux
🔄​ TestAccept 🔄​f
🔄​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🔄​f
🔄​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name/DATABRICKS_BUNDLE_ENGINE=direct/DMS= 🔄​f
Top 50 slowest tests (at least 2 minutes):
duration env testname
12:29 gcp linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
11:38 gcp linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
11:12 aws linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
11:03 aws linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
9:04 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
9:01 gcp linux TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
8:27 aws linux TestAccept/bundle/resources/clusters/resize-terminated-fallback/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
8:27 aws linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
8:10 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
8:02 aws linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
7:40 aws linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
7:20 aws linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
7:20 aws linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:08 azure linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
7:00 azure linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
6:58 gcp linux TestAccept/bundle/resources/clusters/resize-terminated-fallback/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
6:58 azure linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:57 gcp linux TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
6:46 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
6:27 gcp linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:25 aws linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
6:23 aws linux TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
6:20 aws linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:20 gcp linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
6:14 azure linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:01 azure linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
5:13 azure linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
5:06 aws linux TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
5:05 azure linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:04 gcp linux TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:02 gcp linux TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
5:01 gcp linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:58 gcp linux TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:56 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:45 gcp linux TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:41 gcp linux TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
4:37 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
4:36 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:34 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:33 gcp linux TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
4:27 gcp linux TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:12 aws linux TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:04 azure linux TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
3:36 aws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
3:32 azure linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:31 gcp linux TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
3:26 azure linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:25 aws linux TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:21 aws linux TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:20 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct/DMS=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants