Decide whether `codecov/project` should report on PRs

[mtrezza]

Issue

Part of #10681.

The codecov/project check fails if the total coverage of a pull request is lower than that of its base commit, even by a single line, unless the decrease comes from lines that the pull request removes. The repository has no Codecov YAML, so Codecov uses its default project status, which has no threshold. Coverage changes below are given as Codecov shows them, in percentage points.

codecov/project got a result on 374 of the 519 pull requests into alpha created from January 1 to September 29, 2026. It ended as failed on 88 of them (23.5%), and 72 of these 88 were merged. For pull requests created from July 1, it ended as failed on 23 of 116. codecov/patch, the required coverage check on alpha, passed on 69 of the 88. On 66 of these 69, the decrease came only from lines that are covered in some CI runs but not in others, or from missing coverage uploads.

• 360 of the 374 pull requests had at least one failed result, mostly at -6% to -9%, because Codecov reports as soon as the first coverage uploads arrive, while other test jobs are still running. On the 272 of them that ended as passed, the check turned to passed a median of 3 minutes after the first failed result. On 12 of these, the failed check run remained visible next to a later passed commit status. Codecov waits for CI only based on the commit statuses of CI services it recognizes, such as Jenkins or Buildkite. GitHub Actions jobs report check runs instead, and Codecov doesn't count the commit statuses of CodeRabbit, Snyk or the commit message check as CI, so it considers CI as passed from the start.
• 57 of the 88 failures are drops of 0.01% or 0.02%, which is about 1 to 3 lines. Of the 58 pull requests that changed neither src/ nor spec/ and had the same 8 or 9 coverage uploads and the same number of lines as the commit Codecov compared them to, the number of covered lines differed by 1 to 3 in 20, and all 12 with fewer covered lines failed.
• Since April, the lines that vary between CI runs are, on current alpha, lines 1868 and 1887 in the error handler of updateObjectsByQuery in src/Adapters/Storage/Postgres/PostgresStorageAdapter.js, and line 690 in the error handler of findOneAndUpdate in src/Adapters/Storage/Mongo/MongoStorageAdapter.js. On Postgres, some push specs end before the background update of their _PushStatus object. If the update runs after afterEach has dropped all tables, it fails and enters the error handler. This happened in 1 of 31 local runs of the push specs; on CI, the two lines were covered in 17 of 77 alpha reports with all uploads. The MongoDB line was covered in 4 of 74; its cause wasn't reproduced. Earlier in 2026, other lines varied in the same way, for example line 110 of src/AccountLockout.js.
• 26 of the 27 failures of more than 0.10% occurred together with a failed or cancelled test job that is a required check on alpha. A test job uploads its coverage only if its tests pass, so the coverage lacks the lines that only this job covers. A missing upload of the Redis Cache job lowers the coverage by about 67 lines, or 0.4%. In the remaining pull request, fix: File upload Content-Type override via extension mismatch (GHSA-vr5f-2r24-w5hc) #10383, 2 of 8 uploads were never processed by Codecov.
• On 3 of the 88 pull requests, codecov/project failed only because of a coverage loss that codecov/patch doesn't report. fix: Cloud Function multipart requests bypass the maxUploadSize limit #10498 added size checks that reject requests earlier, so the existing tests no longer reach 2 lines in src/Routers/FunctionsRouter.js (shown as -0.01%). fix: MFA SMS token request fails when maxPasswordHistory is set #10543 replaced a save, so that 1 line in src/Adapters/Auth/mfa.js is no longer reached (-0.01%). feat: Add SDK Adapter #10256 replaced code with less covered code (-0.05%). The losses in fix: Cloud Function multipart requests bypass the maxUploadSize limit #10498 and fix: MFA SMS token request fails when maxPasswordHistory is set #10543 are as small as the variation of 1 to 3 lines, so a threshold that ignores the variation would also ignore them. The variation can also hide such a loss: on the last commit of test: Tests fail when third-party servers don't respond in time #10723, 2 varying Postgres lines offset the loss of 1 line in src/Adapters/Auth/utils.js, and codecov/project passed.
• On the 123 pull requests on which codecov/project ended as failed on the last or an earlier commit, no comment or review by a person refers to codecov/project. The comments that mention Codecov refer to codecov/patch by name, or, in fix: Preserve UTF-8 GraphQL upload filenames with latest graphql-upload #10478, were made while codecov/patch had failed for the same lines.
• codecov/project is not a required check on alpha, but it is on release and release-8.x.x. Of the pull requests into these branches created in 2026 that got a Codecov result, it ended as failed on 29 of 94 into release-8.x.x and 5 of 23 into release, and 33 of these 34 were merged regardless.

[parse-github-assistant]

🚀 Thanks for opening this issue!