Carry a schedule onto the bundle job - #6240
Open
vinchenzo-db wants to merge 3 commits into
Open
Conversation
A run YAML may now set a `schedule` block (quartz_cron_expression, timezone_id, and optional pause_status). `air run` submits a one-time run and can't honor it, so it detects the schedule and offers to convert the config to a bundle instead — where `bundle deploy` creates the persistent, scheduled job. convert-to-dabs maps the block 1:1 onto the bundle job's schedule. Co-authored-by: Isaac
Contributor
Approval status: pending
|
Collaborator
Integration test reportCommit: 5aa03c8
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 6 slowest tests (at least 2 minutes):
|
Co-authored-by: Isaac
Pivot the schedule handling: instead of offering to convert to a bundle, a run config with a `schedule` block now makes `air run` create (or update) a persistent, scheduled Databricks job directly via jobs/create. `air run` already uploads the code snapshot + launch scripts to a durable workspace path and builds the ai_runtime_task; this reuses that prep and swaps the ephemeral jobs/runs/submit for a jobs/create carrying a CronSchedule. Since the CLI keeps no state, the job is upserted by name: an existing job with the same experiment_name is reset in place (so re-running doesn't pile up duplicates), and an ambiguous name (>1 match) errors rather than guessing. convert-to-dabs still emits the schedule block, so converting a scheduled config to a bundle keeps the schedule. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
air run on a config with a schedule block creates a persistent, scheduled Databricks job (via jobs/create) instead of submitting a one-time run.
It reuses the existing upload path — the code snapshot + launch scripts already go to a durable workspace path and build the ai_runtime_task — and swaps jobs/runs/submit for a jobs/create carrying a CronSchedule. The CLI keeps no state, so the job is upserted by name: an existing job with the same experiment_name is updated in place (re-running doesn't pile up duplicates); an ambiguous name (>1 match) errors rather than guessing.
Schedule YAML mirrors the Jobs cron block (quartz_cron_expression, timezone_id, optional pause_status). convert-to-dabs also carries the schedule, so converting a scheduled config to a bundle keeps it.
Co-authored-by: Isaac
Changes
Why
Tests