Skip to content

Improve performance by fusing kernels#57

Open
au2001 wants to merge 5 commits into
1inch:masterfrom
codesensus-org:perf/fuse-kernels-and-align
Open

Improve performance by fusing kernels#57
au2001 wants to merge 5 commits into
1inch:masterfrom
codesensus-org:perf/fuse-kernels-and-align

Conversation

@au2001

@au2001 au2001 commented Jul 26, 2026

Copy link
Copy Markdown

Change Summary

What does this PR change?

It improves performance by 15-98% through 2 compounding changes:

  • aligning mp_number to 2x 16 bytes instead of 8x 4 bytes; and
  • merging kernels for iterating/transforming/scoring into a single one

(profanity_inverse is untouched in this PR, so there are still 2 launches total, down from 3-4)

Testing & Verification

How was this tested?

  • Unit tests
  • Integration tests
  • Manual testing (describe steps)
  • Verified on staging

I ran each scoring method on different GPUs with the previous and new codebase.
For each resulting address, I ran an external verification script which found no mistake.
Here are the median results for the ~most popular variant of each NVIDIA generation:

GPU VRAM base head inverse-multiple gain
L40S 45 GB 1,228 MH/s 1,413 MH/s -I 1572864 +15.1%
RTX 5070 12 GB 797 MH/s 1,397 MH/s  -I 393216 +75.2%
RTX 4060 8 GB 397 MH/s 623 MH/s  -I 229376 +56.7%
RTX 3060 12 GB 342 MH/s 676 MH/s  -I 458752 +97.7%
RTX 2060 12 GB 263 MH/s 476 MH/s  -I 8192 +81.1%
GTX 1060 3 GB 141 MH/s 263 MH/s  -I 98304 +86.2%

Notes:
GPUs which were bound by memory latency/bandwidth see the most performance increase.
That's consumer-grade GPUs, while the L40S sees the least gain, as expected.

During benchmarks, when reaching speeds of 2-3ms per round, the millisecond clock precision was insufficient.
I thus bumped it to microseconds for increased accuracy.

A bug also prevented from enabling profiling: clCreateCommandQueueWithProperties on OpenCL 2.0+ expects a zero terminated list of name/value pairs, not a bitfield like 1.2 – 0 just so happened to be interpreted as an empty list.

Risk Assessment

Risk Level:

  • Low - Minor changes, no operational impact
  • Medium - Moderate changes, limited impact, standard rollback available
  • High - Significant changes, potential operational impact, complex rollback

Risks & Impact

Fully backwards-compatible with no user-facing changes.
But changes how the kernels work in a non-trivial way.
Requires clearing the cache to see changes (and to rollback).

Disclaimer: this PR was assisted by Claude Opus 5 for code exploration & benchmark scripts.
However, all code being submitted has been hand-written – and I'm a human :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant