ci: migrate iOS tests to CircleCI M4 Pro - #237
Conversation
5961801 to
8fa3cde
Compare
Run the iOS unit suite and two serial snapshot shards concurrently on managed M4 Pro executors while the existing GitHub Actions controls remain in place. Validate the exact Xcode and simulator runtime, keep checkout access credential-free, hydrate LFS references, export test results, and retain cold-run diagnostics without caching DerivedData. Validation: - circleci config validate .circleci/config.yml - circleci config process .circleci/config.yml - ./swiftformat --lint
8fa3cde to
f175a7c
Compare
Remove the long iOS and snapshot jobs from GitHub Actions after the parallel M4 Pro workflow passed all 1,821 iOS tests and 377 snapshot references. Keep SwiftFormat, architecture, and native-macOS coverage on GitHub Actions. Validation: - circleci config validate .circleci/config.yml - circleci config process .circleci/config.yml - ./swiftformat --lint - CircleCI test-ios, snapshot-1, and snapshot-2
Here’s the complete configuration inventory, including temporary changes that were subsequently reverted. Repository configurationAdded .circleci/config.yml:
Added .circleci/xcresult_to_junit.py:
Updated .github/workflows/ci.yml:
CircleCI organization/project configurationI configured the CircleCI project as follows:
The installed CircleCI GitHub App is scoped to the Stuff repository. Its vendor-declared GitHub permissions are:
Those are installation-level app permissions. The CircleCI jobs are not given an API token, context secret, SSH key, or retained Git checkout credential. GitHub repository configurationInstalled/authorized the CircleCI GitHub App for the Stuff repository so it could:
Updated classic branch protection for Before:
After:
I preserved the other classic protection settings, including admin enforcement, PR review configuration, force-push prevention, and deletion prevention. The separate pre-existing Temporary configuration changesFor completeness, two GitHub ruleset changes occurred during the protection audit:
Final state:
|
Use workflow-level display names matching the retired GitHub Actions iOS and snapshot jobs while preserving CircleCI's reusable job keys.\n\nValidation:\n- circleci config validate .circleci/config.yml\n- circleci config process .circleci/config.yml\n- git diff --check
| type: string | ||
| steps: | ||
| - run: | ||
| name: Convert Xcode results to JUnit |
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Simplified in e5b3fd94: CircleCI now runs the complete snapshot scheme serially in one M4 Pro job, and the checked-in shard map, balancer, CLI flags, timing-report plumbing, and their tests have been removed. The remaining configuration owns runner setup, exact-renderer/runtime gates, LFS hydration, the canonical ./test invocations, JUnit reporting, real wall-clock timeouts, credential checks, and failure diagnostics. The updated file-level walkthrough is here.
kyleve
left a comment
kyleve
left a comment
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Configuration walkthrough: one inline note on each changed file.
| @@ -0,0 +1,254 @@ | |||
| version: 2.1 | |||
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
This file is intentionally more explicit than the retired GitHub workflow. It:
- defines the managed Xcode 27 M4 Pro executor and two concurrent jobs: the iOS unit suite and complete serial snapshot suite;
- hydrates and verifies LFS snapshot references;
- installs and download-caches the pinned
misetoolchain without caching DerivedData or test output; - gates execution on Xcode build
27A5228h, an iOS 27 runtime, and the repository-selected simulator; - verifies checkout did not retain a Git credential;
- supplies real process-group wall-clock limits because CircleCI's
no_output_timeoutonly detects silence; - publishes JUnit test results and per-phase snapshot timings;
- stages complete test, crash, and snapshot diagnostics only after failure; and
- filters execution to PRs targeting
mainand pushes tomain.
Some of this length reflects CircleCI differences: no direct equivalents of GitHub's timeout-minutes, checkout LFS option, or jdx/mise-action. The rest is migration hardening and observability that the old workflow did not provide. Orbs could shorten the YAML, but would hide behavior behind additional dependencies. The earlier two-shard duplication has been removed: one snapshot job now pays checkout, tool installation, simulator boot, project generation, and build overhead once.
| @@ -0,0 +1,98 @@ | |||
| #!/usr/bin/env python3 | |||
| """Convert Xcode 27 test result bundles into CircleCI-compatible JUnit XML.""" | |||
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
This is a reporting-only adapter. ./test writes Apple's .xcresult bundles, which CircleCI's Tests UI cannot interpret directly. The script:
- asks
xcresulttoolfor every result bundle's test tree; - extracts bundle, suite, test name, duration, failure, and skip data;
- writes standard JUnit XML; and
- hands that XML to CircleCI's
store_test_resultsstep.
It enables structured test counts, failures, durations, and timing history in CircleCI. It does not run tests or determine job success—the canonical ./test command still owns pass/fail—and could be removed if the native Tests UI is not useful.
| test-macos: | ||
| name: Build & Test (macOS) | ||
| # Same image as the other jobs so every target builds against one toolchain. | ||
| # Keep the native-macOS target on GitHub's pinned Xcode image. |
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
This file now expresses the cutover boundary: GitHub Actions retains SwiftFormat Lint, Bumper Bowling, and Build & Test (macOS) on the pinned xcode-27 image. The long Build & Test (iOS) job and snapshot job were removed after their CircleCI replacements passed unchanged; CircleCI now runs the complete snapshot scheme serially in one job. Snapshot references, assertions, and tolerances were not modified.
Run the complete snapshot scheme serially on one CircleCI M4 Pro executor and retire the checked-in shard map, balancer, CLI selection flags, per-suite report plumbing, and their tests.\n\nPreserve the same canonical snapshot command, renderer gates, LFS hydration, assertions, review output, and failure diagnostics while avoiding duplicated setup and build compute.\n\nValidation:\n- circleci config validate .circleci/config.yml\n- circleci config process .circleci/config.yml\n- ./swiftformat --lint\n- bash -n test profile\n- ./test --snapshots --review --timings (41 suites, 377 images passed in 7m04s)\n- git diff --check
Stop the retained GitHub lint job from invoking the deleted snapshot shard tests and configuration checker.\n\nValidation:\n- ruby YAML parse of .github/workflows/ci.yml\n- rg --hidden for shard infrastructure references\n- git diff --check
The benchmark completed successfully. Large is faster, but probably not worthwhile for this workload.
Large consumed approximately 5,980 credits versus Medium’s 3,480. So we pay 72% more to save 55 seconds from the actual workflow critical path. My recommendation: keep PR #239 remains open and unchanged. The monitor is now stopped. |
1 participant
Summary
m4pro.mediumjobs./testentry point, iOS 27 simulator selection, snapshot tolerances, and all 377 references while retiring the checked-in shard map27A5228hand an available iOS 27 runtimeFinal state
CircleCI owns these stable required contexts:
ci/circleci: Build & Test (iOS)ci/circleci: Snapshot Tests (iOS)GitHub Actions retains:
SwiftFormat LintBumper BowlingBuild & Test (macOS)The CircleCI project uses its GitHub App pipeline definition, runs for pull requests targeting
mainand pushes tomain, and has auto-cancel enabled for superseded branch workflows.Implementation
.mise.toml,Package.swift, andPackage.resolved; never cache DerivedData or recorded snapshot output./simulatorplus./testxcresultsummaries as JUnitTEST_WORKDIRdiagnostics only after failure, avoiding successful multi-gigabyte artifact uploads.github/workflows/ci.ymlEnvironment and security
The validated executors reported Apple M4 Pro (Virtual), 6 logical CPUs, 30,064,771,072 bytes of memory (about 28 GiB), arm64, macOS 26.5.1 build
25F80, Xcode 27 build27A5228h, an iOS 27 runtime, and an iPhone 17 simulator.There are no CircleCI project environment variables, organization contexts, additional SSH keys, or project API tokens. Checkout uses HTTPS, and every live job proved that no credential helper/extraheader remained after checkout. The installed CircleCI GitHub App itself has CircleCI's vendor-required repository write scopes, but those installation permissions are not exposed as a retained Git credential or job secret.
Validation
circleci config validate .circleci/config.ymlcircleci config process .circleci/config.yml./swiftformat --lint— 0/1012 files require formatting./test --snapshots --review --timings: 41 suites / 377 images passed in 7m04s with no reference changesThe attached plan listed 381 snapshot references, but the exact migration head contains 377; the shared
StuffSnapshotTestsscheme runs all 41 suites in one invocation. No references or tolerances changed.Snapshot compatibility review
The sharded CircleCI baseline and unsharded local run both passed all 377 images at the existing requirement: identical dimensions and at least 99.9% of pixels within Delta E 10. A review-mode pass identified 29 non-byte-identical candidates: 22 had only scattered single-digit antialiasing/color-rounding drift; the seven double-digit cases were manually compared as reference/candidate/diff. Their hierarchy, dimensions, content, and layout matched, with differences confined to antialiasing, thin divider edges, and top-chrome rendering. No reference was re-recorded and no tolerance was loosened.
Provider comparison
The completed comparison uses GitHub Actions 31416367884 and the final unified CircleCI workflow. The two CircleCI jobs completed successfully in 6m40s and 10m44s, with an 11m03s workflow critical path.
Cost
CircleCI bills
m4pro.mediumat 200 credits/minute. The unified workflow consumed 17m24s of aggregate executor time, about 3,480 credits / $2.09 at the Performance-plan rate of $15 per 25,000 credits. The latest two-shard workflow consumed 25m57s, about 5,190 credits / $3.11, so reunifying snapshots saves about 1,710 credits / $1.03 (33%) per workflow. Its 11m03s critical path is 1m04s longer than the two-shard CircleCI workflow's 9m59s. Sources: CircleCI resource pricing and Performance plan overview.Branch protection
Before cutover, classic branch protection required the GitHub-only format, iOS, Bumper, and two snapshot contexts. It now requires exactly the three retained GitHub checks and two CircleCI checks above, pinned to their respective GitHub Apps. The renamed required contexts are exercised by the current PR head. The separate pre-existing
Require PR for Mainrepository ruleset remains disabled and unchanged.Review focus
mainRollback
Before merge, close this PR and disable the CircleCI project/repository access, then restore the old classic required contexts. After merge, restore the deleted GitHub jobs from the migration parent, switch classic branch protection back to their GitHub contexts, and disable the CircleCI project. Snapshot references require no rollback.