Skip to content

Investigate cancelled PostgreSQL matrix jobs #10711

Description

@mtrezza

Issue

Part of #10681.

Test jobs that end as cancelled ran into the job timeout of 20 minutes (timeout-minutes: 20 in .github/workflows/ci.yml). GitHub Actions reports a job that exceeds its timeout as cancelled instead of failed, and stops the test run before Jasmine prints its summary.

  • From August 13, 2025 to September 27, 2026, 512 of 32,501 test jobs were cancelled by the job timeout: 2.4% of the PostgreSQL jobs and 1.0% of the MongoDB, Redis Cache and Node jobs. 37 other test jobs were cancelled individually for other reasons. All other cancelled test jobs belonged to runs that were cancelled for a newer commit.
  • Logs are still available for the 55 test jobs that timed out since July 1, 2026:
    • 36 jobs, in all test job types, timed out because the default LiveQuery client connected to a Parse Server that was already shut down. The Parse JS SDK derives the LiveQuery URL from Parse.serverURL when the first default LiveQuery client is created, and keeps it for the rest of the test run. The specs does shutdown liveQuery server and does shutdown separate liveQuery server start a second Parse Server on port 1345, which sets Parse.serverURL to that server. If one of these specs is the first in the test run to create a default LiveQuery client, every later subscription with the default LiveQuery client connects to port 1345 after that server has been shut down, and waits until the spec times out after 20 s. The LiveQuery specs that follow add about 15 min to the test run. Running one of these specs followed by can subscribe to query reproduces the timeout locally with MongoDB and PostgreSQL.
    • 8 PostgreSQL jobs timed out on a slow runner, without any spec timing out. By then they had run 86% to 100% of the specs. Their median spec took 151 to 221 ms, compared with 43 to 80 ms in the other PostgreSQL jobs of the same commit, and Postgres took about 10 times as long to sync its checkpoints to disk, which points to a slow disk. Since July 1, 31 successful test jobs took between 15 and 19.6 min, 27 of them PostgreSQL jobs.
    • 7 PostgreSQL jobs have a different cause, which is addressed separately.
    • 4 jobs timed out after other specs had failed. Other test jobs of the same commit failed as well, which points to the change under test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions