Skip to content

fix(pushapk): Actually check apk signatures - #1514

Draft
hneiva wants to merge 5 commits into
mozilla-releng:masterfrom
hneiva:hneiva/fix-skip-check-signature
Draft

fix(pushapk): Actually check apk signatures#1514
hneiva wants to merge 5 commits into
mozilla-releng:masterfrom
hneiva:hneiva/fix-skip-check-signature

Conversation

@hneiva

@hneiva hneiva commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

pushapkscript has never verified an APK or AAB signature, for any product, in any environment. skip_check_signature was read from the wrong config dict, so the jarsigner and MANIFEST.MF checks in script.py were dead code. This predates the initial monorepo import.

Switching them on needed four more fixes, one of which would otherwise have broken production.

The bug

script.py:

if not publish_config.get("skip_check_signature", True):

skip_check_signature is declared at the product level of the config schema, but this reads it from publish_config, the per-app/channel/store dict built by get_publish_config(). That function never emits the key, so the lookup always missed and always fell back to the default.

Two bugs, not one:

  1. Wrong dict. The option is unreachable.
  2. Wrong default. True means skip, and the else branch claims the product is configured with the option, but that branch is what ran when the key was absent, which is every product in worker.yml.

Fixing only (1) changes nothing, since nothing sets the option. Both had to move together. No test covered either branch, which is why this survived.

What else was blocking it

jarsigner was not installed anywhere. Both the deployed image and the test image installed default-jre-headless, which ships keytool but not jarsigner; jarsigner is JDK only. Verified on debian:bookworm, where the JRE package provides just /usr/bin/keytool while default-jdk-headless provides both. Turning verification on without this would have failed every task in production.

Samsung and Huawei had no certificate_alias. The non-Google branch of _get_channel_publish_config never copied it into its result, so jarsigner._pluck_configuration raised KeyError on it.

mozillavpn dev/fake-prod has no alias at all, and init_worker.sh imports no certificate for it ("no dep signing for mozillavpn yet"). A None alias reached the jarsigner argv as a TypeError from subprocess.

The test fixtures cannot be verified. The APKs date from 2017 and use SHA1withDSA with a 1024-bit key. Modern JDKs disable both, so jarsigner reports them as unsigned and -verify -strict exits 16. Their manifests also list entries that were stripped out of the archives, which -strict rejects on its own.

Commits

  • set certificate_alias for non-Google target stores One line, matching what the Google paths already did. The alias identifies the certificate the incoming artifact was signed with, which the upstream signing task decides, so it is the same whichever store the artifact goes to. Huawei support, currently in flight, needs no further change.
  • fail loudly when no certificate_alias is configured ConfigValidationError naming the store and pointing at skip_check_signature, instead of a KeyError or a TypeError from inside subprocess.
  • install the JDK, so jarsigner exists default-jdk-headless in both the deployed and the test image.
  • sign the integration fixtures so they can be verified Generate a throwaway RSA-2048 keypair per test and re-sign each fixture under the alias its config expects, after dropping the stale META-INF so jarsigner writes a fresh manifest. The generated configs now declare SHA-256, which is what worker.yml uses in production. Also fixes test_main_with_samsung_store, which signed for nightly while the fenix release app it targets declares fenix-production.
  • verify APK signatures, which never actually ran Read product_config, default to verifying. mozillavpn non-prod opts out explicitly, as the one config that genuinely cannot verify.

Testing

  • 91 pass in a default-jdk-headless container, with the integration tests genuinely running jarsigner -verify -strict against freshly signed APKs.
  • 84 pass and 7 skip locally without a JDK. Detection runs the binaries rather than looking them up, because macOS ships stubs on PATH that exist but exit non-zero.
  • 264 repo-level init tests pass. ruff, ruff-format and yamllint are clean.
  • The new test_script.py test fails against the old code with assert 0 == 2, so it is a real regression test.
  • Audited every rendered worker.yml combination for a missing alias. mozillavpn dev/fake-prod is the only one, and it opts out.

Risk

This enables a check that has been dead since before the monorepo import, so production artifacts have never been through jarsigner -verify -strict on a modern JDK. Autograph signs with PKCS#7 and SHA-256, so it should hold, but the fixtures show that weak-algorithm rejection is not hypothetical.

Worth watching the first dev/fake-prod task before this reaches prod. skip_check_signature: true on a product is the escape hatch if some artifact turns out not to verify.

The non-Google branch of _get_channel_publish_config never copied
`certificate_alias` into its result, while all three Google paths did, so a
samsung publish config had no alias at all and jarsigner._pluck_configuration
would raise KeyError on it. Nothing noticed, because its only caller is
jarsigner.verify(), which never runs.

The alias identifies the certificate the incoming artifact was signed with,
which the upstream signing task decides, so it is a property of the artifact
rather than of the destination store and belongs at the app level for every
store alike. Huawei support, currently in flight, needs no further change.
_pluck_configuration indexed `publish_config["certificate_alias"]`, which
raises KeyError when the key is absent, and passed the value through
untouched when it was None. A None alias ends up in the jarsigner argv,
where subprocess reports it as `TypeError: expected str, bytes or
os.PathLike object, not NoneType`.

Raise ConfigValidationError instead, naming the store and pointing at
`skip_check_signature`, so a product configured without an alias says so
rather than failing inside subprocess.
Both the deployed image and the test image installed
`default-jre-headless`, which ships keytool but not jarsigner: jarsigner
lives in the JDK. Verified against debian:bookworm, where
default-jre-headless provides only /usr/bin/keytool while
default-jdk-headless provides both.

This did not matter while signature verification was dead code, but the
script does shell out to jarsigner, and the README already says a JDK is
required. Install default-jdk-headless in both images so the binary is
actually present before verification is switched on.
…ified

The fixture APKs date from 2017 and are signed with SHA1withDSA and a
1024-bit DSA key. Modern JDKs disable both, so jarsigner treats them as
unsigned and `-verify -strict` exits 16. Their manifests also list entries
that were stripped out of the archives, which `-strict` rejects on its own.
No alias juggling can make them verify.

Generate a throwaway RSA-2048 keypair per test instead and re-sign the
fixtures with it under the alias the product config expects, after dropping
the stale META-INF so jarsigner writes a fresh manifest. The generated
configs now declare SHA-256 to match, which is what worker.yml uses in
production anyway.

test_main_with_samsung_store signed for "nightly" while the fenix release
app it targets declares "fenix-production"; it now uses the right alias.

Signing needs a real keytool and jarsigner, so the class is skipped when
they are missing. Detection runs the binaries rather than looking them up,
because macOS ships stubs on PATH that exit non-zero with "Unable to
locate a Java Runtime".
`skip_check_signature` is a product-level option, but async_main read it
from `publish_config`, the per-app/channel/store dict that
get_publish_config() builds. That function has never emitted the key on any
of its paths, so the lookup always missed and always fell back to the
default. jarsigner.verify() and manifest.verify() were dead code for every
product in every environment. The bug predates the initial monorepo import.

The default was wrong too. `True` means skip, and the log line in the
skipped branch claims the product is configured with the option, but that
branch is what ran when the key was absent, which is every product in
worker.yml. Reading the right dict alone changes nothing, since nothing
sets the option, so both have to move together: read `product_config`, and
default to verifying.

mozillavpn dev/fake-prod is the one config that cannot verify, because
there is no dep signing for it yet and init_worker.sh imports no
certificate, so it opts out explicitly.

No test covered either branch of the condition, which is why this survived.
There is one now; it fails against the old code with `assert 0 == 2`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant