CI: add timeout and retry to apt-get install steps - #840
Closed
mmcky wants to merge 3 commits into
Closed
Conversation
Bare `sudo apt-get install -y` with no `timeout-minutes` and no retry let a stalled Ubuntu mirror hang CI for hours — two 358-minute hangs on 2026-08-18 across two edition repos ten minutes apart, and four more hangs on 2026-08-19 each cleared only by cancel-and-rerun. Every apt-installing step now carries a step-level `timeout-minutes` and runs up to three attempts with a per-attempt `timeout`, so a stalled fetch fails fast and a transient mirror outage is ridden out rather than burning the job. Package lists are unchanged. Tracked in QuantEcon/project-translation#43.
✅ Deploy Preview for taupe-gaufre-c4e660 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens GitHub Actions apt-get install steps to avoid long CI hangs caused by stalled Ubuntu mirrors by adding step-level timeouts plus a retry loop with per-attempt timeouts.
Changes:
- Added
timeout-minutesto apt-installing steps (10m for graphviz, 30m for TeX dependencies). - Wrapped
apt-get update && apt-get installin a 3-attempt retry loop withtimeoutper attempt and a 30s backoff. - Consolidated LaTeX package lists into a single
pkgsvariable (no package list changes).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| .github/workflows/ci.yml | Adds step timeouts + retry wrapper around graphviz and LaTeX apt installs. |
| .github/workflows/cache.yml | Adds step timeout + retry wrapper around LaTeX apt install (graphviz step still un-hardened). |
| .github/workflows/publish.yml | Adds step timeout + retry wrapper around LaTeX apt install. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
27
to
+31
| - name: Install latex dependencies | ||
| timeout-minutes: 30 | ||
| run: | | ||
| sudo apt-get -qq update | ||
| sudo apt-get install -y \ | ||
| texlive-latex-recommended \ | ||
| texlive-latex-extra \ | ||
| texlive-fonts-recommended \ | ||
| texlive-fonts-extra \ | ||
| texlive-xetex \ | ||
| latexmk \ | ||
| xindy \ | ||
| dvipng \ | ||
| cm-super | ||
| # Retried with a per-attempt timeout: a stalled Ubuntu mirror otherwise | ||
| # hangs this step for hours (two 358-minute hangs on 2026-08-18). |
| environment-file: environment.yml | ||
| activate-environment: quantecon | ||
| - name: Graphics Support #TODO: Review if graphviz is needed | ||
| timeout-minutes: 10 |
…er-attempt timeouts (4m/14m) Addresses Copilot review on QuantEcon/lecture-python.zh-cn#263: the cache.yml graphviz step was missed by a name match, and three 3m/9m attempts could never let a slow-but-progressing install finish. Now two attempts at 4m (graphviz) / 14m (texlive) inside the unchanged 10/30-minute step budgets.
…-get Addresses Copilot review on QuantEcon/lecture-intro.zh-cn#301: with timeout wrapping sudo, a timed-out attempt terminated sudo and the bash -c shell but could orphan apt-get holding the dpkg lock, making the retry block instead of retry. timeout is now the direct parent of each apt-get (inside sudo, -k 30s escalation): update 1m/2m, install 3m/12m for graphviz/texlive. Step budgets unchanged.
Contributor
Author
|
Closing unmerged (@mmcky, 2026-08-21): the apt hangs are accepted as a known flake for now rather than investing further here — the QuantEcon/actions container templates remove the apt step entirely, and that deployment is the medium-term fix. Branch left in place for reference; tracked in QuantEcon/project-translation#43. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bare
sudo apt-get install -ywith notimeout-minutesand no retry lets a stalled Ubuntu mirror hang CI for hours: two 358-minute hangs on 2026-08-18 across two edition repos ten minutes apart (a mirror outage), four more hangs on 2026-08-19 (Install latex dependencies77/15/16 min,Graphics Support21 min) each cleared only by cancel-and-rerun, and a 59-minute slow-but-green install on 2026-08-20.This PR hardens every apt-installing step in the workflows that carry one (
ci.yml,cache.yml,publish.ymlwhere present):timeout-minutes(10 for graphviz, 30 for the texlive set), so a hang can no longer take the six-hour job limit;timeout(3m / 9m) and a 30-second pause, so a transient mirror outage is ridden out rather than failing the build — a timeout alone would only turn a six-hour hang into a fast red build.Package lists are unchanged. The apt steps are edition-local (the shared
QuantEcon/actions/templatescarry none), so this lands per repo. Part of the five-repo sweep tracked in QuantEcon/project-translation#43.