Conversation
) * fix: stop using globals for patch data * chore: run et format * chore: add missing files * test: add unittest * chore: run et format * chore: move elf_cache down into runtime * chore: rename elf* to patch* * chore: clean up logs * chore: clean up comments * chore: use Shorebird dart * chore: small cleanup
…ASE_URL (#97) * fix: make dart/flutter work without FLUTTER_STORAGE_BASE_URL * feat: shorebird flutter should work without setting FLUTTER_STORAGE_BASE_URL * fix: flutter_tools test fixes * fix: enable running flutter_tools tests * chore: remove unnecessary workflow
* chore: move build_engine scripts into this repo * chore: fix path of content_aware_hash.sh
* chore: roll dart to 6a78a2deaee05bc74775fcfa2ff27aa53c96efca * wip * chore: run et format * chore: attempt to clean up shorebird.cc * chore: fix build * chore: remove FLUTTER_STORAGE_BASE_URL override
* feat: allow patch_verification_mode * test: update tests * chore: rename to patch_verification
…clude patch_verification option
* es/report_start_fix * fix: second callsite
* chore: add a C++ interface onto the updater * chore: centralize SHOREBIRD_PLATFORM_SUPPORTED * test: fix tests
Previously we stopped reporting start on android by accident. This fixes that. I also removed the once-per-process guard since it's not necessary. This should be correctly reporting once-per-shell and let the rust code only handle the first of the calls. Fixes shorebirdtech/shorebird#3488
As part of our previous fix for FlutterEngineGroup, we introduced a new bug whereby report_launch_start could be called more than once in a multi-engine scenerio. That would cause confusion about what the current boot patch is, since the current patch is updated as part of report_launch_start. report_launch_start should only be called once per processs, which this change fixes. We still need more end-to-end testing at this layer to prevent bugs like this from sneaking in.
…#108) - Create a template Flutter project once in setUpAll and copy it per test, avoiding repeated `flutter create` calls - Run a warm-up `flutter build apk` in setUpAll (outside per-test timeout) to prime the Gradle cache - Add actions/cache for ~/.gradle so subsequent CI runs start warm - Add VERBOSE env var and failure output logging from #107
* chore: split CI into parallel jobs Split the single CI job into three parallel jobs: 1. flutter-tools-tests: Runs on ubuntu + macOS (unchanged) 2. shorebird-android-tests: Runs on Ubuntu only (faster runners) 3. shorebird-ios-tests: Runs on macOS only (requires Xcode) This improves CI performance by: - Running all jobs in parallel instead of sequentially - Moving Android tests off macOS to faster Ubuntu runners - Removing Windows from the matrix (nothing was running there anyway) Expected speedup on macOS: ~5 minutes (no longer runs Android tests) * Add Android smoke test on macOS Run a single "can build an apk" test on macOS to catch any platform-specific issues with Android builds on macOS. * Add comment explaining why Windows is excluded
* feat: add sharded CI build runner Adds a Dart-based shard runner for parallel engine builds: - JSON configs for Linux, macOS, Windows shards - run_shard.dart: executes gn/ninja/rust builds - compose.dart: assembles iOS/macOS frameworks - GCS upload/download for artifact staging * refactor: parse compose.json into typed objects Move from reading compose.json directly as Map<String, dynamic> to using proper ComposeConfig and ComposeDef model classes. This follows a more idiomatic Dart pattern. * feat: add finalize.dart for manifest generation and uploads Implements the finalize job logic that: - Downloads artifacts from GCS staging - Generates artifacts_manifest.yaml - Uploads to production GCS bucket (download.shorebird.dev) Ports upload logic from linux_upload.sh, mac_upload.sh, and generate_manifest.sh into a single Dart script. * fix: correct cargo-ndk invocation for Android Rust builds - Use --target flag (not -t) with Rust target triples - Set ANDROID_NDK_HOME to engine's hermetic NDK - Build all Android targets in a single cargo ndk command - Remove incorrect _androidArch helper function This matches the behavior of linux_build.sh. * test: add unit tests for config parsing Tests cover: - PlatformConfig: single-step shards, multi-step shards, compose_input - BuildStep: gn_ninja and rust step types - ComposeConfig: compose definitions, requires, script, args - Error handling for unknown shards/compose names * feat: add artifacts field to shard configs Define artifacts declaratively in JSON configs instead of hardcoding upload paths in Dart. Each artifact specifies: - src: source path relative to out/ - dst: destination path with $engine placeholder - zip: whether to zip before upload (for directories like dart-sdk) - content_hash: whether to also upload to content-hash path (for Dart SDK) This makes the config self-describing and aligns with Flutter's ci/builders/*.json pattern of explicit artifact declarations. * refactor: read shard names from JSON configs instead of hardcoding Load PlatformConfig for each platform to get shard names dynamically, rather than maintaining a duplicate list in finalize.dart. * feat(ci): add manifest generation and bucket configuration - Extract generateManifest to lib/manifest.dart with tests - Refactor finalize.dart to use artifacts from JSON configs - Add --bucket flag for test uploads to alternate buckets - Add compare_buckets.dart for validating uploads against production * chore: add pubspec.lock for shard_runner * chore: allow shard_runner pubspec.lock in gitignore * chore: use local .gitignore for shard_runner pubspec.lock * refactor: load manifest from template file Move manifest content to artifacts_manifest.template.yaml and update generateManifest to load from template file with sync IO. * fix: fail finalize on download errors instead of continuing A missing shard download means incomplete artifacts. Better to fail loudly than silently upload an incomplete build. * fix: fail on gsutil/zip errors instead of warning Upload and zip failures should halt the build, not silently continue with missing artifacts. * refactor: clean up shard_runner CLI and config parsing - Add shared runChecked() helper to eliminate duplicated Process.run + exit code check patterns across config.dart, gcs.dart, finalize.dart, and compose.dart - Add @immutable annotations to all data classes (via package:meta) - Remove implicit single-step shard shorthand; all shards now use explicit steps arrays in JSON configs - Convert all async file IO to sync equivalents (existsSync, etc.) - Make --engine-src and --run-id mandatory CLI args, removing hidden defaults and GITHUB_RUN_ID env var fallback - Restructure compose.json to use explicit flags/path_args instead of a single args list that guessed flag vs path semantics - Collect outDirs from config upfront rather than accumulating during execution * ci: add shard_runner tests to shorebird_ci workflow - Add analysis_options.yaml (package:lints/recommended with strict mode) - Add shard-runner-tests job with format, analyze, and test steps - Fix stale await on sync PlatformConfig.load in compare_buckets.dart - Reformat all files to Dart standard (80 char width)
Each shard runs on a separate machine, so it needs its own Rust build step for the updater library. Previously only the host/android shards had Rust steps, but all shards that build libflutter need libupdater.a for their specific target triple.
On Windows, gcloud SDK tools like gsutil are installed as .cmd files. When Dart's Process.run is called without runInShell, it doesn't resolve these .cmd extensions. This adds a helper that explicitly checks for .cmd versions in PATH on Windows.
* Bump flutter_flavorizr to published 2.5.0 (fix flavor APK build)
The Shorebird Android Tests' flavor builds fail on 3.44.1:
Product Flavor playStore contains custom resource values,
but the feature is disabled.
The test scaffolds flavors with flutter_flavorizr pinned to a fork at a
Flutter-3.29-era ref (AngeloAvv/flutter_flavorizr#291, which was never
merged). The AGP that ships with 3.44.1 disables resValues by default, and
the 3.29-era generator doesn't enable the build feature, so the generated
project won't configure.
Move to the published flutter_flavorizr ^2.5.0, which has moved well past
3.29 and generates AGP-8-compatible flavor config. Removes the stale fork
pin + felangel TODO.
* Pass -f to flutter_flavorizr (skip interactive prompt under CI)
flutter_flavorizr 2.5.0 added an interactive 'Do you want to proceed?
(Y/n)' confirmation, which throws 'No terminal attached to stdout' in CI
(both Android Tests and the Smoke build). The -f/--force flag runs it
non-interactively.
* Pin flutter_flavorizr to 2.4.2 (Ruby xcodeproj, fixes iOS flavor archive)
2.5.0 replaced Ruby xcodeproj with dart_xcodeproj; its generated .pbxproj
breaks 'flutter build ipa --no-codesign --flavor' (unsigned flavor archive
demands a Development Team). All versions emit identical visible signing
settings, so it's the dart_xcodeproj generation, not config — and the old
fork worked precisely because it used Ruby xcodeproj.
2.4.2 is the last Ruby-xcodeproj release and still carries the 3.29/AGP-8
resValues fix that the 3.29-era fork lacked, so it should green both the
Android (resValues) and Smoke (iOS flavor) builds.
* Enable resValues build feature after flavorizr (fix flavored APK)
There is no single flutter_flavorizr version that greens both checks:
2.5.0 fixes the AGP-8 resValues issue but its dart_xcodeproj rewrite breaks
the unsigned iOS flavor archive; 2.4.2 (last Ruby-xcodeproj release) fixes
iOS but predates the resValues fix.
So pin 2.4.2 (iOS works) and post-fix the one thing it misses: append
android.defaults.buildfeatures.resvalues=true to the generated
gradle.properties so the flavored 'flutter build apk' configures under AGP 8.
Wraps flavorizr rather than forking it.
createForSnapshots took a VM snapshot and an isolate snapshot and concatenated data, data, text, text. Dart 3.13 folded the VM isolate into the isolate group, so kVMDataSymbol and kIsolateDataSymbol are the same symbol and resolve to the same buffer. The base stream the updater diffs against was therefore twice the bytes analyze_snapshot --dump_blobs writes, misaligned against the host extraction, with nothing on either end validating the length. Now takes one snapshot and pushes two mappings, matching HandleDumpBlobs. The surviving source is the VM resolve path deliberately. ResolveIsolateData fires ReportLaunchStart and returns the patch when TryLoadFromPatch finds one. ResolveVMData is patch-blind, and this stream has to be the unpatched base. That already held, but only because SetBaseSnapshot runs before the patch is front-inserted. It now holds by construction. Also rewrites the front-insert comment, which credited the VM isolate for behavior that is required because linked patch code executes out of the base image.
dart_sdk_revision moves to 1c05054e0833f9fc163e81e294a7ef1562e9f930 and the sidecar values come from the publish-darwin-arm64 run that produced that artifact (run 31415119236). Re-enables the macos-arm64 gcs hook. It was pinned to 'False' during the dry run because nothing had been published for the offsets-regen tip and gclient sync 404'd on it. Both halves of that are gone: the artifact exists at this revision, and the hook is not vestigial. Only the --no-prebuilt-dart-sdk shards build Dart from source; the plain --mac arm64/x64 shards and the ios-release shard all consume the prebuilt.
Every engine shard that compiles harfbuzz failed: ninja: error: '../../flutter/third_party/harfbuzz/src/hb-paint-bounded.cc', needed by 'obj/.../harfbuzz_sources.hb-paint-bounded.o', missing and no known rule to make it 3b06709 ("split shorebird C API consumption") took DEPS wholesale from a stale base and moved 13 third-party pins backward, undoing every upstream roll that had landed in between. It touched no build files, so each roll's build-file half survived while its DEPS half regressed, and the two halves have disagreed ever since. No later roll re-bumped any of them, so these are the newest values the branch has ever carried. harfbuzz is the case that surfaced. Roll a006091 added "src/hb-paint-bounded.cc" to build/secondary/flutter/third_party/harfbuzz/ BUILD.gn and bumped the pin in the same commit. That BUILD.gn at HEAD is byte-identical to the post-roll version. The file is present at 49844c3 and absent at ea6a172, confirmed against the mirror. Restored: clang, rapidjson, harfbuzz, glfw, shaderc, googletest, perfetto (which also moved back to chromium_git), freetype2, libpng, zlib, angle, imgui, and the rbe reclient_cfgs cipd version. Left alone: updater_rev, which that commit also changed but which has been deliberately re-set since, and the dart-sdk pin and its gcs hook.
All three iOS shards failed to compile: patch_cache.cc:51:25: error: no matching function for call to 'Dart_LoadELF' note: candidate function not viable: requires at most 6 arguments, but 8 were provided Dart_LoadELF used to hand back a VM pair and an isolate pair. The VM isolate folded into the isolate group, so the VM and isolate symbols name the same buffers and the loader now resolves a single data/text pair from kSnapshotDataAsmSymbol and kSnapshotTextAsmSymbol. patch_cache.cc was the last caller still passing the split form; embedder.cc had already moved. Same root cause as the snapshots_data_handle fix: code written when the two snapshots were distinct, against a Dart where they no longer are. Verified by compiling the file against the 3.47 dart-sdk headers, and by confirming the pre-fix version reproduces the exact CI error under the same command. patch_mapping.cc, snapshots_data_handle.cc and updater.cc also compile clean against those headers.
TryLoadFromPatch compared the incoming symbol against its own copies of the snapshot symbol names. Folding the VM isolate into the isolate group renamed those symbols to kDartSnapshotData and kDartSnapshotText, and the copies in patch_cache.cc kept the old names, so every call fell through the filter and returned nullptr. iOS resolved both isolate snapshots from the base App.framework and ran base code on every launch while the updater reported a patch as running. Callers now pass a PatchSymbol enum naming which half of the isolate pair they want, so no symbol name crosses the boundary and there is nothing left to keep in sync. Correcting the strings alone would not have held, because the VM and isolate symbols share the same names after the fold and a string can no longer tell them apart. A patch that fails to load now falls back to the base image instead of aborting. That path was unreachable while the filter rejected every symbol.
…g base A patch that installs but fails to load fell back to the base image without telling the updater. ReportLaunchStart has already promoted the patch to current_boot by that point, so the updater kept reporting it as running while base code executed, and the next launch retried the same broken patch. Report the failure, which marks it bad and activates the next best patch. The call is guarded once per process, so the second symbol lookup is a no-op. This is the reporting half of the gap that let the iOS symbol mismatch look like a working patch for a whole session. Also correct two comments claiming shorebird_report_launch_start() runs from TryLoadFromPatch(). It runs from ResolveIsolateData in runtime/dart_snapshot.cc, before any patch load is attempted.
Picks up the 6-way debug language shard split plus the two Linux build guards. Prebuilt pin (sha256sum, size_bytes, generation) matches the published darwin-arm64 sidecar for this revision. Drops canvaskit_cipd_instance, which has no readers left in the tree.
The existing tests build handles through the public constructor from a hand-made vector, so they cover the Read and Seek arithmetic and never touch blob count or order. That is the surface that was wrong: with kVMDataSymbol and kIsolateDataSymbol resolving to the same buffer, taking two snapshots appended every byte twice and misaligned the stream against the host's dump_blobs extraction, with no length validated at either end.
Six tests carried 3.44-era expectations through the rebase while the code under test tracks upstream. All fixes are test-side; no source changed. The iOS deployment target moved to 15.0 upstream. build_test and common_test hardcode the literal where base/build.dart reads it from FlutterDarwinPlatform.ios.deploymentTarget(), so the expectation drifted; upstream's own ios_test.dart already reads 15.0. macos_test expected lipo -verify_arch to receive both archs. darwin.dart is identical to upstream and passes one, so the fake never matched and the expected throw never fired. version_test's clean-environment check is upstream-authored and does not know about the flutter_release branch lookup GitTagVersion.determine runs ahead of the tag lookup. Added that call to the fake sequence, keeping the clean-environment assertion on it so the test still covers what it names.
The guard resolved symlinks on the allowed root but not on the path being checked, so the two never matched on macOS, where Directory.systemTemp reports /var/folders/... and resolving it yields /private/var/folders/.... path.canonicalize normalizes without touching symlinks, so every createTempSync against the unresolved form was rejected. Upstream flutter#187320 added the resolved root to fix callers that resolve the path themselves. Keeping both forms serves those callers and the unresolved ones together, and preserves the cached lookup that commit introduced to keep the check off the I/O path. Linux has no /var symlink and upstream runs test/general.shard on Linux alone, so nothing exercised this. The Shorebird matrix covers macOS too, where it accounted for 162 failures against zero on ubuntu. Applied to the devicelab copy as well to keep the two identical, matching the scope of flutter#187320.
Matching both spellings of the temp root covers a symlinked head such as macOS's /var -> /private/var, but not a symlink further down. When temp itself is a link, the resolved root and the path being checked diverge mid-path and no string comparison of the two can succeed. Resolve the path under test as well, falling back to the nearest existing ancestor since paths are routinely checked before they exist. The comparison against the cached roots still runs first, so the added I/O happens only on paths that would otherwise be rejected. This is the case flutter#187320 added its regression test for, which fails whenever it runs on its own. A full-file run happens to leave state that lets it pass, which is why the shard reported it inconsistently.
Points DEPS at the 3.47.1 dart-sdk RC. 460d9f8 is flutter_release_ex/3.47.1-rc0, the 158 fork commits from the 3.47.0 release replayed onto Dart 3.13.1. Upstream's 3.13.0 to 3.13.1 delta is 7 commits over 22 files. Only 0456d8c touches VM code the fork patches, and its hunks in il.cc/il.h are thousands of lines away from ours, so the rebase replayed with no conflicts and no dropped hunks. Dart CI 32501604550 green on all 39 jobs. The three gcs pins come from the sidecar JSON published by run 32510313294 and are cross-checked against the object itself. They must move with dart_sdk_revision or gclient sync fails on the mac shard. updater_rev is unchanged. Updater main has not moved since 3.47.0.
Rebased onto Flutter 3.47.2 / Dart 3.13.2. Updater unchanged at 1f85c4a. Carries the dart_dynamic_modules build repair (flutter#836), which had landed only on flutter_release_ex/3.47.0-rc0, plus three CI fixes for setup-dart's problem matcher.
Built on dart-sdk 52183de from flutter_release_ex/3.47.2-rc0. Base moved Flutter 3.47.1 -> 3.47.2 and Dart 3.13.1 -> 3.13.2; updater unchanged at 1f85c4a. Dart side carries the dart_dynamic_modules build repair (flutter#836), which had landed only on flutter_release_ex/3.47.0-rc0 and would otherwise have been dropped by the rebase, plus three CI fixes disabling setup-dart's problem matcher across all three composite wrappers. Kernel SDK-hash audit clean: shipped dart-sdk vm_platform*.dill all hash-zeroed, const_finder carries upstream 60a57cd42d, both patched_sdk sets zeroed.
3 tasks
) Wires --shorebird-trace into: - BuildAarCommand and BuildFrameworkCommand (iOS frameworks) - addCommonDesktopBuildOptions in FlutterCommand, which covers desktop builds (linux, macos, windows) - AAR Gradle build path in AndroidGradleBuilder, mirroring the existing trace session usage in the assemble path The AAR path runs Gradle through _processUtils.run rather than _runGradleTask, so there is no onStart hook to feed onGradleSpawn; the trace still carries the outer gradle span and the per-task events from the init script, just without the Perfetto spawn arrow. Note: when 'flutter build aar' is invoked with multiple modes in a single run, each iteration's trace overwrites the previous one. Shorebird's release flow only builds release mode, so this is not a concern in practice. Claude-Session: https://claude.ai/code/session_01X2aY3kQvT2tgM1mYbN9c7W
`flutter build macos` runs xcodebuild with the same xcode_backend.dart script phase as `flutter build ios`, so the iOS trace session applies unchanged: pod install span, SHOREBIRD_TRACE_FILE forwarded into the build phase so `flutter assemble` writes its per-target trace, outer `xcode build` span, per-target subsections from xcresulttool, and the merged trace written to the path the CLI passed. The only thing the macOS path lacked was an xcresult bundle: the iOS build requests one for error reporting, macOS never did. The session now owns a temp bundle path for callers that don't bring their own (`ownResultBundleArgs`) and deletes it after parsing. `finish()` takes the build target so the outer span is `flutter build macos` rather than a hardcoded `ios`. Also fixes the compile error in build_macos_test.dart (the skipped shorebird.yaml test constructed BuildCommand without the arguments upstream has since made required), which was preventing the whole file from running. Claude-Session: https://claude.ai/code/session_01X2aY3kQvT2tgM1mYbN9c7W
eseidel
force-pushed
the
es/trace-macos
branch
from
September 11, 2026 15:51
91b2142 to
169fea4
Compare
2 tasks
eseidel
enabled auto-merge (squash)
September 12, 2026 05:38
nickshorebird
force-pushed
the
shorebird/dev
branch
from
September 14, 2026 14:17
e137e18 to
a9314be
Compare
nickshorebird
force-pushed
the
shorebird/dev
branch
from
September 15, 2026 13:22
19660dc to
65e9873
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #136 (which registers the flag on desktop build commands). Follow-up to the "flag accepted but not wired" row in that PR's table.
Summary
flutter build macosruns xcodebuild with the samexcode_backend.dartscript phase asflutter build ios, soIosBuildTraceSessionapplies unchanged — this PR drops it intobuild_macos.dartat the same seven pointsmac.dartuses it:pod installspan aroundprocessPodsIfNeeded(phase sub-spans come from cocoapods.dart viaBuildTracer.current, as today).SHOREBIRD_TRACE_FILE=…on the xcodebuild command line, so the script phase'sflutter assemblewrites its per-target trace and the session merges it.xcode buildspan, plus per-target subsections parsed from the xcresult bundle.flutter build macosspan; trace written on success, tracer cleared on failure /--config-only.The one thing macOS lacked was an xcresult bundle — iOS requests one for error reporting, macOS never did.
IosBuildTraceSession.ownResultBundleArgs()now hands out a temp bundle path for callers that don't bring their own and cleans it up inonXcodeFinished. Only requested when tracing is on, so untraced macOS builds are byte-for-byte the same xcodebuild invocation as before.finish()takes abuildTargetso the outer span isn't hardcoded toios.Also
build_macos_test.dartdidn't compile onshorebird/dev: the (skipped) shorebird.yaml test constructsBuildCommandwithout the arguments upstream has since made required, which took the whole file out of the test run. Fixed here since this PR adds a test to that file.CLI side
Nothing on the shorebird_cli side needs to change to collect this —
prepareBuildTracealready runs for macOS, and the merged summary's generic buckets (flutterBuild,flutterAssemble,native,dart,network,flutterTool) populate from these events. Two things don't yet:BuildTraceSummaryonly fillsiosplatform stats whenplatform == 'ios', so the pod-install phase breakdown and xcode subsection distribution are dropped for macOS. Trivial to widen once this lands.'android' | 'ios'.Test plan
Against the
e16cf749tool checkout:dart analyzeonbuild_macos.dart,mac.dart,ios_build_trace_session.dart,build_macos_test.dart— no issues.dart test test/commands.shard/hermetic/build_macos_test.dart test/commands.shard/hermetic/build_ios_test.dart— 61 pass, 1 skipped. Includes a new test asserting the xcodebuild invocation carriesSHOREBIRD_TRACE_FILEand-resultBundlePath, the trace is written withflutter build macos/xcode buildspans, and the temp xcresult directory is removed.Not yet run end-to-end against a real
shorebird release macos— that needs a pin bump to a revision containing this, same as #136.https://claude.ai/code/session_01X2aY3kQvT2tgM1mYbN9c7W