Skip to content

fix(ci): Remove hardcoded cloud_runner from maxtext_jupyter_notebooks#4355

Draft
mesakhcienet wants to merge 3 commits into
mainfrom
fix/notebook-ci-runner
Draft

fix(ci): Remove hardcoded cloud_runner from maxtext_jupyter_notebooks#4355
mesakhcienet wants to merge 3 commits into
mainfrom
fix/notebook-ci-runner

Conversation

@mesakhcienet

@mesakhcienet mesakhcienet commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR fixes two critical issues blocking the maxtext_jupyter_notebooks CI job from running successfully, particularly for Pull Requests originating from forks.

  1. Reverted Notebook Runner to v6e-4 to Unblock the CI Queue
    In commit f940dfd (April 29th), the notebook CI was updated to use v6e-8 and request a linux-x86-ct6e-180-8tpu runner to accommodate rl_llama3_demo.ipynb. However, because there are no 8-chip runners available in the CI pool, GitHub Actions queues the job indefinitely, blocking all PRs.
  • Fix: Fully reverted the notebook CI configuration back to the known-working 4-chip setup.
    • Downgraded device_name back to v6e-4.
    • Explicitly requested the available linux-x86-ct6e-180-4tpu cloud runner.
    • Re-added rl_llama3_demo.ipynb to the skip list in run_jupyter_notebooks.yml (as it OOMs on 4 chips).
  1. Gracefully Handle Empty Hugging Face Tokens in Fork PRs
    Due to GitHub Actions security restrictions, PRs originating from forks cannot access repository secrets. Consequently, ${{ secrets.HF_TOKEN }} evaluates to an empty string. This caused the hf auth login --token "$HF_TOKEN" command to crash with an Illegal header value b'Bearer ' exception.
  • Fix: Added a conditional check around the Hugging Face authentication step in run_jupyter_notebooks.yml. If the token is empty, the script now gracefully logs a warning and skips authentication rather than crashing the entire
    workflow.

Motivation and Context
These fixes are necessary to restore a passing baseline for the MaxText CI pipeline. Currently, the notebook job either hangs forever (due to the missing runner) or crashes immediately (due to the empty token on fork PRs).

How has this been tested?

  • Verified that the CI correctly requests the self-hosted, tpu, v6e-4 labels.
  • Verified that the workflow proceeds past the Hugging Face authentication step gracefully when HF_TOKEN is empty.

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@mesakhcienet mesakhcienet changed the title Fix/notebook ci runner fix(ci): Remove hardcoded cloud_runner from maxtext_jupyter_notebooks Jul 6, 2026
@mesakhcienet mesakhcienet force-pushed the fix/notebook-ci-runner branch from a8b0d7b to 07b060e Compare July 6, 2026 03:48
@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@mesakhcienet mesakhcienet force-pushed the fix/notebook-ci-runner branch from 07b060e to 769e0a3 Compare July 6, 2026 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant