Skip to content

ci: Benchmarks check fails on CI runner noise #10704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mtrezza merged 2 commits into from
Sep 27, 2026
+721 −136
Merged

ci: Benchmarks check fails on CI runner noise #10704

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2c13036
ci: Benchmarks check fails on CI runner noise
mtrezza Sep 26, 2026
78dddb7
ci: Require all retest measurements to clear a flagged benchmark
mtrezza Sep 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
171 changes: 42 additions & 129 deletions .github/workflows/ci-performance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,49 +22,58 @@ jobs:
performance-check:
name: Benchmarks
runs-on: ubuntu-latest
timeout-minutes: 30
# Allows flagged benchmarks to be measured again, see benchmark/compare.js
timeout-minutes: 45

steps:
- name: Checkout PR branch (for benchmark script)
- name: Checkout PR branch
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 1

- name: Save PR benchmark script
run: |
mkdir -p /tmp/pr-benchmark
cp -r benchmark /tmp/pr-benchmark/ || echo "No benchmark directory"
cp package.json /tmp/pr-benchmark/ || true

# The base branch is kept in a separate directory, so that both builds are available to
# measure flagged benchmarks again
- name: Checkout base branch
uses: actions/checkout@v4
with:
ref: ${{ github.base_ref }}
fetch-depth: 1
clean: true
path: base

- name: Restore PR benchmark script
- name: Use PR benchmark script for base branch
run: |
if [ -d "/tmp/pr-benchmark/benchmark" ]; then
rm -rf benchmark
cp -r /tmp/pr-benchmark/benchmark .
if [ -d benchmark ]; then
rm -rf base/benchmark
cp -r benchmark base/
fi

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
cache-dependency-path: |
package-lock.json
base/package-lock.json

- name: Install dependencies (base)
working-directory: base
run: npm ci

- name: Build Parse Server (base)
working-directory: base
run: npm run build

- name: Install dependencies (PR)
run: npm ci

- name: Build Parse Server (PR)
run: npm run build

- name: Run baseline benchmarks
id: baseline
working-directory: base
env:
NODE_ENV: production
run: |
Expand All @@ -73,54 +82,24 @@ jobs:
if [ ! -f "benchmark/performance.js" ]; then
echo "⚠️ Benchmark script not found - this is expected for new features"
echo "Skipping baseline benchmark"
echo '[]' > baseline.json
echo "Baseline: N/A (no benchmark script)" > baseline-output.txt
echo '[]' > ../baseline.json
echo "Baseline: N/A (no benchmark script)" > ../baseline-output.txt
exit 0
fi
taskset -c 0 npm run benchmark 2>&1 | tee baseline-output.txt || npm run benchmark 2>&1 | tee baseline-output.txt || true
taskset -c 0 npm run benchmark 2>&1 | tee ../baseline-output.txt || npm run benchmark 2>&1 | tee ../baseline-output.txt || true
# Extract JSON from output (everything between first [ and last ])
sed -n '/^\[/,/^\]/p' baseline-output.txt > baseline.json || echo '[]' > baseline.json
sed -n '/^\[/,/^\]/p' ../baseline-output.txt > ../baseline.json || echo '[]' > ../baseline.json
continue-on-error: true

- name: Save baseline results to temp location
run: |
mkdir -p /tmp/benchmark-results
cp baseline.json /tmp/benchmark-results/ || echo '[]' > /tmp/benchmark-results/baseline.json
cp baseline-output.txt /tmp/benchmark-results/ || echo 'No baseline output' > /tmp/benchmark-results/baseline-output.txt

- name: Upload baseline results
uses: actions/upload-artifact@v4
with:
name: baseline-benchmark
path: |
/tmp/benchmark-results/baseline.json
/tmp/benchmark-results/baseline-output.txt
baseline.json
baseline-output.txt
retention-days: 7

- name: Checkout PR branch
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 1
clean: true

- name: Restore baseline results
run: |
cp /tmp/benchmark-results/baseline.json ./ || echo '[]' > baseline.json
cp /tmp/benchmark-results/baseline-output.txt ./ || echo 'No baseline output' > baseline-output.txt

- name: Setup Node.js (PR)
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'

- name: Install dependencies (PR)
run: npm ci

- name: Build Parse Server (PR)
run: npm run build

- name: Run PR benchmarks
id: pr-bench
env:
Expand Down Expand Up @@ -158,93 +137,27 @@ jobs:

- name: Compare benchmark results
id: compare
env:
NODE_ENV: production
BENCHMARK_BASE_DIR: base
BENCHMARK_COMMAND: taskset -c 0 npm run benchmark
run: |
set -o pipefail
node -e "
const fs = require('fs');

let baseline, pr;
try {
baseline = JSON.parse(fs.readFileSync('baseline.json', 'utf8'));
pr = JSON.parse(fs.readFileSync('pr.json', 'utf8'));
} catch (e) {
console.log('⚠️ Could not parse benchmark results');
process.exit(0);
}

// Handle case where baseline doesn't exist (new feature)
if (!Array.isArray(baseline) || baseline.length === 0) {
if (!Array.isArray(pr) || pr.length === 0) {
console.log('⚠️ Benchmark results are empty or invalid');
process.exit(0);
}
console.log('# Performance Benchmark Results\n');
console.log('> ℹ️ Baseline not available - this appears to be a new feature\n');
console.log('| Benchmark | Value | Details |');
console.log('|-----------|-------|---------|');
pr.forEach(result => {
console.log(\`| \${result.name} | \${result.value.toFixed(2)} ms | \${result.extra} |\`);
});
console.log('');
console.log('✅ **New benchmarks established for this feature.**');
process.exit(0);
}

if (!Array.isArray(pr) || pr.length === 0) {
console.log('⚠️ PR benchmark results are empty or invalid');
process.exit(0);
}

console.log('# Performance Comparison\n');
console.log('| Benchmark | Baseline | PR | Change | Status |');
console.log('|-----------|----------|----|---------| ------ |');

let hasRegression = false;
let hasImprovement = false;

baseline.forEach(baseResult => {
const prResult = pr.find(p => p.name === baseResult.name);
if (!prResult) {
console.log(\`| \${baseResult.name} | \${baseResult.value.toFixed(2)} ms | N/A | - | ⚠️ Missing |\`);
return;
}

const baseValue = parseFloat(baseResult.value);
const prValue = parseFloat(prResult.value);
const change = ((prValue - baseValue) / baseValue * 100);
const changeStr = change > 0 ? \`+\${change.toFixed(1)}%\` : \`\${change.toFixed(1)}%\`;

let status = '✅';
if (change > 50) {
status = '❌ Much Slower';
hasRegression = true;
} else if (change > 25) {
status = '⚠️ Slower';
hasRegression = true;
} else if (change < -25) {
status = '🚀 Faster';
hasImprovement = true;
}

console.log(\`| \${baseResult.name} | \${baseValue.toFixed(2)} ms | \${prValue.toFixed(2)} ms | \${changeStr} | \${status} |\`);
});

console.log('');
if (hasRegression) {
console.log('⚠️ **Performance regressions detected.** Please review the changes.');
process.exitCode = 1;
} else if (hasImprovement) {
console.log('🚀 **Performance improvements detected!** Great work!');
} else {
console.log('✅ **No significant performance changes.**');
}
" | tee comparison.md
# A PR branch that predates the comparison script has to be updated to be compared
if [ ! -f benchmark/compare.js ]; then
echo "⚠️ Benchmarks cannot be compared because this branch does not contain \`benchmark/compare.js\`. Update the branch with the base branch." | tee comparison.md
exit 1
fi
node benchmark/compare.js baseline.json pr.json | tee comparison.md

- name: Upload comparison
if: always()
uses: actions/upload-artifact@v4
with:
name: benchmark-comparison
path: comparison.md
path: |
comparison.md
retest-output.txt
retention-days: 30

- name: Generate job summary
Expand Down
3 changes: 3 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,8 @@ Note that the script above will ONLY be executed during initialization of the co

Parse Server includes an automated performance benchmarking system that runs on every pull request to detect performance regressions and track improvements over time.

A benchmark is flagged as a regression if it is more than 25% slower than on the base branch. To rule out noise of the CI runner, a flagged benchmark is measured again 3 times for the base branch and 3 times for the pull request, alternating between them. The check only fails if the regression persists in the medians of these measurements, or if any of these measurements fails.

#### Adding Tests

You should consider adding performance benchmarks if your contribution:
Expand Down Expand Up @@ -342,6 +344,7 @@ Performance benchmarks are located in [`benchmark/performance.js`](benchmark/per
```bash
npm run benchmark:quick # Quick test with 10 iterations
npm run benchmark # Full test with 10,000 iterations
BENCHMARK_NAMES='["Feature Name"]' npm run benchmark # Run only the listed benchmarks
```

For new features where no baseline exists, the CI will establish new benchmarks that future PRs will be compared against.
Expand Down
Loading
Loading