Run the hybrid solver on hypre's threads-as-ranks backend (no mpirun) - #114
Run the hybrid solver on hypre's threads-as-ranks backend (no mpirun)#114danielepanozzo wants to merge 2 commits into
Conversation
The hybrid solver needed MPI, which meant the whole application had to be launched under mpirun. That is fine for a driver but not for a library: anything embedding polysolve inherited the requirement. hypre can now build its MPI surface on threads of one process (HYPRE_ENABLE_THREAD_MPI), so the ranks become threads and mpirun goes away. The resulting binary links no libmpi at all. What this needed: - tmpi_mpi_compat.hpp: the few MPI facilities hypre's backend does not already provide. hypre maps MPI_Bcast/Allreduce/Scatterv/... onto its own surface already; missing were the MPI-3 shared-memory windows (with threads, "shared memory" is just a pointer broadcast), MPI_IN_PLACE, and MPI_Init/Initialized/Finalized. - The ranks are started by the solver rather than by mpirun. The first hybrid solver constructed calls hypre_tmpi_team_start(), the calling thread becomes rank 0 and keeps driving, and the workers sit in the existing command loop. They now return from their thread function instead of std::exit(0), and the last solver destroyed shuts the team down so the rest of the process sees a one-rank world again. - is_running_worker_loop and worker_registry are thread_local. They were per-rank only because each rank used to be a process; as threads they were shared and the workers raced on them. MPI remains available: POLYSOLVE_WITH_MPI still selects a rank-parallel hybrid solver, it just no longer implies OpenMPI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## hybrid-solver #114 +/- ##
==============================================
Coverage 80.07% 80.07%
==============================================
Files 52 52
Lines 2138 2138
Branches 284 284
==============================================
Hits 1712 1712
Misses 426 426
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The Catch2 tests cannot time the multi-rank path: they drive the solver from rank 0 while the other ranks sit in the worker loop, and under mpirun at more than one rank the suite is torn down when a worker leaves the loop. Timing that measures the teardown, not the solve. tests/bench_spmd.cpp is the same source built into both an OpenMPI and a thread-MPI build, so the two backends are compared on one driver rather than on two that happen to look similar. It builds a 7-point Laplacian of a given grid size, solves it, and prints setup/solve/total plus the relative residual so a run that converged differently cannot be mistaken for a run that was merely faster. scripts/bench_hybrid.sh sweeps rank counts on both backends. It takes an flock so two sweeps cannot halve each other's scores, refuses to start against a loaded machine rather than quietly reporting contended numbers, and interleaves the backends inside each repetition so drift hits both equally. scripts/bench_report.py renders the CSV as Markdown: best-of-N rather than mean, the worst run-to-run spread so the reader can judge whether a gap is real, and a check that both backends reached the same residual at each rank count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Benchmark: OpenMPI processes vs thread-MPI threadsThreadripper PRO 3995WX, 64 cores, single NUMA node, idle. 7-point Laplacian, 120³ = 1.73M unknowns, Grid 120^3, best of 3 runs; worst run-to-run spread 10.7%.
setup vs solve
Reading thisCorrectness first: both backends reach an identical relative residual at every rank count. The residual changes with the rank count — the partitioning changes, so AMG builds a different hierarchy — but never between backends at the same count. Threads win at 1–2 ranks, OpenMPI wins from 4 up, ending 27% ahead at 64. The loss is not where I expected: look at the setup/solve split. thread-MPI's solve is competitive and actually faster at 16 and 32 ranks (0.50s vs 0.54s). The entire deficit is in setup, which plateaus around 0.51s for threads while OpenMPI keeps falling to 0.28s. AMG setup is the communication-heavy phase, so this points at the thread-MPI collective/matching path rather than anything about the solve. Caveat on precision: worst run-to-run spread is 10.7%, which is larger than I would like. The 1–2 rank and 32–64 rank conclusions are safely outside that; the 4-rank crossover (0.96x) is inside it and should be read as "roughly equal", not as a win for either. What this does and does not justifyIf you need Still to comeGPU numbers. |
GPU resultsSame driver, same 120³ problem, RTX 3080 Ti. Best of 3, load 2.1–2.2, spread under 2%.
One GPU is 2.5× faster than all 64 CPU cores at their best, and 18× faster than a single rank. The two builds are 1.2% apart, which is the point of the table. A correction worth recordingMy first GPU run reported times identical to CPU. That was not the GPU being slow: Note also that Summary across both comments
|
| find_package(MPI QUIET) | ||
| if (NOT MPI_CXX_FOUND) | ||
| message(WARNING "POLYSOLVE_WITH_MPI was requested but MPI was not found, proceeding without MPI dependent solvers.") | ||
| if (FALSE) |
There was a problem hiding this comment.
Pull request overview
This PR updates the hybrid linear solver’s “rank” acquisition to use hypre’s threads-as-ranks backend so embedding applications no longer need to launch under mpirun, while keeping POLYSOLVE_WITH_MPI=ON as the selector for rank-parallel hybrid solving.
Changes:
- Introduces a thread-MPI compatibility shim (
tmpi_mpi_compat.hpp) to cover missing MPI facilities (shared windows,MPI_IN_PLACE, init-state calls). - Refactors
CPUHybridSolverto start/join hypre thread ranks internally, and makes worker-loop state/registrythread_localto avoid cross-thread races. - Adds an SPMD benchmark driver plus scripts to sweep rank counts and report results.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| CMakeLists.txt | Adjusts MPI discovery behavior when POLYSOLVE_WITH_MPI is enabled. |
| cmake/recipes/hypre.cmake | Switches hypre recipe to thread-MPI backend (fork/branch) and toggles hypre MPI options. |
| src/polysolve/linear/tmpi_mpi_compat.hpp | New MPI-compat layer for hypre thread-MPI builds (windows, in-place reductions, init-state). |
| src/polysolve/linear/CPUHybridSolver.hpp | Makes worker state thread_local; adds rank-team lifetime management helpers. |
| src/polysolve/linear/CPUHybridSolver.cpp | Starts/stops thread ranks via hypre tmpi team; worker threads return instead of exiting. |
| tests/CMakeLists.txt | Adds bench_spmd executable for benchmarking. |
| tests/bench_spmd.cpp | New SPMD benchmark program that drives the hybrid solver from all ranks. |
| scripts/bench_hybrid.sh | New sweep script to benchmark OpenMPI vs thread-MPI backends. |
| scripts/bench_report.py | New CSV→Markdown report generator for benchmark output. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if(POLYSOLVE_WITH_MPI) | ||
| # hypre's thread-MPI backend supplies the ranks, so no MPI installation is | ||
| # needed. Keep looking for one only to satisfy anything that still wants it. | ||
| find_package(MPI QUIET) | ||
| if (NOT MPI_CXX_FOUND) | ||
| message(WARNING "POLYSOLVE_WITH_MPI was requested but MPI was not found, proceeding without MPI dependent solvers.") | ||
| if (FALSE) |
| include(CPM) | ||
| CPMAddPackage( | ||
| NAME hypre | ||
| GITHUB_REPOSITORY hypre-space/hypre | ||
| GIT_TAG 7e247a231ebdeb44b06c7c9d3b5bee3bac21123f | ||
| GITHUB_REPOSITORY danielepanozzo/hypre | ||
| GIT_TAG thread-mpi-backend | ||
| SOURCE_SUBDIR src | ||
| ) |
| Eigen::setNbThreads(1); | ||
| HYPRE_SetMemoryLocation(HYPRE_MEMORY_HOST); | ||
| HYPRE_SetExecutionPolicy(HYPRE_EXEC_HOST); | ||
| spdlog::set_level(spdlog::level::off); |
| if [ "$backend" = openmpi ]; then | ||
| line=$(mpirun -quiet --oversubscribe -np "$n" "$MPI_BUILD/tests/bench_spmd" "$GRID" 2>/dev/null | tail -1) | ||
| else | ||
| line=$(HYPRE_TMPI_NUM_THREADS=$n "$TMPI_BUILD/tests/bench_spmd" "$GRID" 2>/dev/null | tail -1) | ||
| fi |
| runs = defaultdict(list) | ||
| grids = set() | ||
| res = defaultdict(set) # residual per rank count, not overall | ||
| with open(path) as fh: |
| if (sendbuf == MPI_IN_PLACE) | ||
| { | ||
| const size_t nb = polysolve::tmpi_compat::datatype_size(dt) * (size_t) count; | ||
| void *tmp = std::malloc(nb ? nb : 1); | ||
| std::memcpy(tmp, recvbuf, nb); | ||
| const HYPRE_Int rc = hypre_MPI_Allreduce(tmp, recvbuf, count, dt, op, comm); | ||
| std::free(tmp); | ||
| return rc; | ||
| } |
Stacked on #112 — please read that one first; this PR only changes how the hybrid solver gets its ranks.
What this does
#112's hybrid solver needs MPI, so the whole application has to be launched under
mpirun. That is fine for a driver but not for a library: anything embedding polysolve inherits the requirement.hypre can now build its MPI surface on threads of one process instead of processes, so the ranks become threads and
mpirungoes away. The resulting binary links nolibmpiat all:MPI is not removed as a concept —
POLYSOLVE_WITH_MPI=ONstill selects a rank-parallel hybrid solver. It just no longer means OpenMPI.How
tmpi_mpi_compat.hpp(new) supplies the few MPI facilities hypre's backend does not already provide. hypre mapsMPI_Bcast/Allreduce/Scatterv/… onto its own surface already, so what was missing is: the MPI-3 shared-memory windows (when ranks are threads, a "shared" window is just a pointer broadcast),MPI_IN_PLACE, andMPI_Init/Initialized/Finalized.The solver starts its own ranks. The first hybrid solver constructed calls
hypre_tmpi_team_start(); the calling thread becomes rank 0 and keeps driving, and the workers land in the command loop this PR's parent already has. They now return from their thread function rather thanstd::exit(0), and the team is refcounted so the last solver destroyed shuts it down and the rest of the process sees a one-rank world again.is_running_worker_loopandworker_registrybecamethread_local. They were per-rank only because each rank used to be a process; as threads they are shared, and the workers raced on them. This is the main hazard when moving rank-parallel code onto threads, and worth knowing about for any other statics added to this solver later.Dependency note
cmake/recipes/hypre.cmakecurrently points atdanielepanozzo/hypre@thread-mpi-backend, where the backend lives as an open PR. That pointer should move to upstream hypre before this merges.Status
Full suite passes with the ranks as threads:
Rank count comes from
HYPRE_TMPI_NUM_THREADS, defaulting to the cores online.Performance numbers are still to come — I am benchmarking the hybrid solver as OpenMPI vs threads-as-ranks, CPU and GPU, on an idle 64-core machine, and will post the comparison here. Treat the PR as functionally complete but not yet performance-justified.
🤖 Generated with Claude Code