Skip to content

Move _unittest.yml to linux_job_v3 - #22245

Open
huydhn wants to merge 1 commit into
mainfrom
osdc/v3-plumbing
Open

Move _unittest.yml to linux_job_v3#22245
huydhn wants to merge 1 commit into
mainfrom
osdc/v3-plumbing

Conversation

@huydhn

@huydhn huydhn commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

First of six splitting up #22107, which was too large to review in one piece. Each PR stacks on the previous one.

_docker-image.yml, which this depends on, now lands in #22106 instead.

_unittest.yml is its first caller, so pull.yml and trunk.yml exercise the resolution on every PR. The docker-image input still takes ci-image:<name>, so its own callers are unchanged; this workflow expands that into the full ECR reference.

Authored with Claude Code.

@pytorch-bot

pytorch-bot Bot commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22245

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 28, 2026
@huydhn huydhn mentioned this pull request Aug 28, 2026
Draft
@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@huydhn huydhn mentioned this pull request Aug 28, 2026
Closed
@huydhn huydhn changed the title Add _docker-image.yml and move _unittest.yml to linux_job_v3 Move _unittest.yml to linux_job_v3 Aug 28, 2026
huydhn added a commit that referenced this pull request Aug 28, 2026
@huydhn
Add _docker-image.yml
e07bd02 
Consumer-side counterpart of the docker-builds move, landing here so the
linux_job_v3 migration can build on it.

linux_job_v3 passes docker-image straight to the runner pod's container, which
is pulled before any step runs, so v2's in-job calculate-docker-image -- which
resolved ci-image:<name> to an ECR reference -- has nowhere left to run. This
emits the same tag docker-builds computes and callers name the image with it.

Like pytorch's OSDC jobs, nothing waits for the images: a commit that changes
.ci/docker needs the ciflow/docker label, and without it the pod fails to pull.

Nothing calls this yet; the callers arrive in #22245 onward.

Authored with Claude Code.
@huydhn huydhn mentioned this pull request Aug 28, 2026
Merged
huydhn added a commit that referenced this pull request Aug 28, 2026
@huydhn
Move docker-builds to OSDC runners (#22106)
db8b643 
Part 1 of moving ExecuTorch's Linux CI to
[OSDC](https://github.com/pytorch/test-infra/blob/main/docs/osdc_runners.md).
The `linux_job_v2` -> `v3` migration follows in #22245 onward; #22108
covers the native jobs.

OSDC pods have no docker daemon, so the builds move to the in-cluster
BuildKit pool: `build.sh` switches to `docker buildx build --push` under
`REMOTE_BUILDKIT`, driven by test-infra's `docker-build-remote-buildkit`
action. The tag is computed in the workflow now, but it is the same
string `calculate-docker-image` produced, so jobs still on v2 keep
resolving. That action only retries failures from before BuildKit
starts, so the three retries `calculate-docker-image` wrapped the build
in are kept around the SDK downloads.

sccache resolved its S3 credentials from EC2 instance metadata, which a
BuildKit pod has none of, so every compiler invocation died on a
metadata timeout. They now reach the build as a BuildKit secret, and
`install_pytorch.sh` caches locally when it is absent. Dropping the S3
backend would have been the smaller change, but the PyTorch build layer
takes ~19 minutes only because sccache serves nearly all of it, and
uncached it would not fit the job timeout.

`pull_request` gives way to `ciflow/docker`, since a fork PR gets no
OIDC token and could never push. The tag is registered in
`pytorch-probot.yml` and auto-applied on the paths that need a rebuild,
so such a PR still rebuilds its images without anyone having to remember
— the same rule pytorch has.

Last commit adds `_docker-image.yml`, the consumer-side counterpart: it
emits `git rev-parse HEAD:.ci/docker` for the v3 jobs to name their
image with. Nothing calls it yet.

All 16 images build and push green on OSDC in [run
32799247406](https://github.com/pytorch/executorch/actions/runs/32799247406):
16–48 min each against 37–56 on EC2, arm64 included.

Authored with Claude Code.
@huydhn huydhn mentioned this pull request Aug 28, 2026
Draft
@huydhn
huydhn marked this pull request as ready for review August 28, 2026 02:40
@huydhn huydhn mentioned this pull request Aug 28, 2026
Merged
@huydhn

huydhn commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

The two unittest / linux failures are not in this diff — they are linux_job_v3's own first step:

rm: cannot remove '/__w/executorch/executorch': Permission denied

The GH runner image creates GITHUB_WORKSPACE as uid 1001, our ci-image ends with USER ci-user at uid 1000 ("Same as ec2-user"), so that user cannot unlink it and every later step skips.

linux_job_v2 ran the same cleanup under sudo by default, gated on its no-sudo input; v3 dropped both, so this is a regression that only surfaces with a non-root image — which is why cuda.yml's v3 jobs on pytorch/almalinux-builder never hit it, and why this PR is the first to. pytorch's own images have the identical USER jenkins/uid 1000 setup and work around it with sudo chmod -R 777 "$GITHUB_WORKSPACE" as the first step of setup-linux, which they can do because they own the workflow; v3 hardcodes Clean workspace first.

Fix: pytorch/test-infra#8665. The last commit here temporarily points _unittest.yml at that branch to prove it, and comes off before landing.

huydhn
huydhn commented Aug 28, 2026
View reviewed changes
Comment thread .github/workflows/_unittest.yml Outdated
linux:
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
needs: docker-image
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@osdc/v3-workspace-permissions

@huydhn huydhn Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll switch this back to main before landing. This needs a fix from pytorch/test-infra#8665

atalman pushed a commit to pytorch/test-infra that referenced this pull request Aug 28, 2026
@huydhn
linux_job_v3: fix the workspace permissions instead of recreating it (#…
f0a28c4 
…8665)

`Clean workspace`, the first step of every v3 job, dies with a non-root
image:

```
rm: cannot remove '/__w/executorch/executorch': Permission denied
```

and every later step skips. The GH hook creates `GITHUB_WORKSPACE` as
uid 1001
([runner-images#10936](actions/runner-images#10936)),
while CI images commonly default to a uid 1000 user mirroring `ec2-user`
— pytorch's `USER jenkins` and executorch's `USER ci-user` both do. It
only surfaces with a non-root image, which is why jobs on
`pytorch/almalinux-builder` have never hit it. Reproducer:
[pytorch/executorch#22245](pytorch/executorch#22245),
the first v3 caller to pass a repo CI image.

**An OSDC pod is ephemeral, so there is nothing to clean.** The
workspace starts empty — the failing `rm -rfv` printed no `removed`
lines before erroring — and the step is carried over from v2, where EC2
runners were reused between jobs. What a job actually needs is to be
able to write into the directory the hook created, so this chmods it and
drops the removal, which is what pytorch/pytorch does in
[`.github/actions/setup-linux`](https://github.com/pytorch/pytorch/blob/d3b8bf0a11cba909aac9e8cdf8b81af8d59a251e/.github/actions/setup-linux/action.yml#L53-L63)
for its ARC path
([pytorch/pytorch#178973](pytorch/pytorch#178973)).

Worth recording: adding a chmod *before* the `rm` does not work, and the
test below caught it. Unlinking the workspace needs write on its parent
`/__w/<repo>`, which chmodding the workspace does not grant — the chmod
succeeded and the `rm` failed anyway.

Root images can already write and often ship no `sudo` (`nvidia/cuda`,
bare distro images), so the chmod tolerates failure rather than being
gated on a user check.

## Tests

`test_linux_job_v3.yml` covers both paths:

- `test-docker-image` (`ghcr.io/pytorch/test-infra:cpu-x86_64-latest`)
asserts `id -u` is 0, pinning the root path that must not require sudo.
- `test-non-root-docker-image` is new, on
`ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3.10-clang21` — a uid
1000 `USER jenkins` image. It asserts a non-root user, probes that the
workspace is writable, and checks the checkout produced a git repo. It
fails on `main` today.

Authored with Claude Code.
@huydhn
Move _unittest.yml to linux_job_v3
7b62802 
First caller of _docker-image.yml. pull.yml and trunk.yml call this workflow,
so they exercise the resolution on every PR.

_docker-image.yml also gains a docker-registry output, so the ECR account and
region are written once rather than at every call site. An output rather than
an env, which GitHub does not allow in a reusable workflow's `with`. It stops
short of the ci-image repository so that callers can keep passing
`ci-image:<name>` as their docker-image input, which keeps each PR in this
stack landable on its own.

Authored with Claude Code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shoumikhin shoumikhin shoumikhin approved these changes
@kirklandsign kirklandsign Awaiting requested review from kirklandsign

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. topic: not user facing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@huydhn @shoumikhin