cuSolverMp 0.7.2: cusolverMpSygvd internal communication failure on 16-rank 4x4 grid

[Stardust0831]

Describe the bug

ABACUS beta6 can hang during the first SCF/RT step when ks_solver cusolvermp is used with 16 MPI ranks and 16 GPUs arranged as a 4x4 cuSolverMp grid. The same workload completes with cuSolverMp 0.9.0 and cuBLASMp 0.9.1.

The failure is not an ABACUS-side NCCL initialization failure. Instrumentation showed that all ranks successfully create their NCCL communicator, CUDA stream, cuSolverMp handle, and cuSolverMp device grid. The hang starts inside the cusolverMpSygvd() call used by Diag_CusolverMP_gvd.

Expected behavior

cusolverMpSygvd() should either complete the distributed generalized eigensolve or return a documented cuSolverMp error. It should not leave a subset of ranks waiting indefinitely with an internal NULL communicator.

To reproduce

The failure is reproducible with NVIDIA's official cuSOLVERMp/mp_sygvd.c sample, independently of ABACUS:

mpirun -np 16 ./mp_sygvd -grid_layout R -p 4 -q 4 -m 39 -mbA 10 -nbA 10

The same sample also reproduced the failure with -grid_layout C. The output contains twelve warnings from ranks 4--15:

NCCL WARN CommUserRank : comm argument is NULL

In the ABACUS RT-TDDFT case, the workload is the small O3 test with 3 MD/RT steps, device gpu, ks_solver cusolvermp, 16 MPI ranks, and 16 V100 GPUs. cusolverMpSygvd_bufferSize() returns success, but the first cusolverMpSygvd() call stalls.

Environment

ABACUS: v3.11.0-beta6, commit 31c899d
GPU: 16 x Tesla V100-SXM2-16GB (SM70)
CUDA: 12.9.1
NVHPC: 26.3
cuSolverMp: 0.7.2 (NVHPC math_libs/12.9)
cuBLASMp: 0.8.0 (NVHPC math_libs/12.9)
NCCL: 2.29.3
MPI: OpenMPI 5.0.10
OS: Linux x86_64

Additional context

NVIDIA's cuSolverMp release notes state that versions 0.4.2 through 0.8.0 contain STEDC-related defects affecting Syevd/Sygvd in certain 2D configurations. The installed 0.7.2 is in that range.

As an A/B check, the same ABACUS executable rebuilt against project-local cuSolverMp 0.9.0 and cuBLASMp 0.9.1 completed the 16-GPU test:

Slurm job: 682234
Exit code: 0
MD/RT steps: 3
NCCL WARN count: 0
!FINAL_ETOT_IS -1336.999498209972 eV

Suggested maintainer action:

1. Confirm the affected cuSolverMp version range for the ABACUS Sygvd integration.
2. Document a minimum supported version for NCCL-backed 2D Sygvd.
3. Add a configure-time warning or guard for known-bad versions.
4. Add a small multi-rank regression case where CUDA CI resources permit.

ABACUS GPU CI evidence (2026-07-20)

The SAI self-hosted runner workflow was exercised on 4 V100 GPUs with explicit library selection. The project prefix selected cuSolverMp 0.9.x and cuBLASMp 0.9.x; the NVHPC system prefix is selectable separately as cuSolverMp 0.7.2 and cuBLASMp 0.8.0. Run 29707130597 used Slurm job 685147 and completed all 7 PW layers, 16 NAO-Gamma cases, 3 NAO-multik cases, and 16 RT-TDDFT cases. The only failure was the existing stochastic SDFT BPCG case 005_PW_SDFT_MALL_BPCG_GPU, whose reference drift was 1.66e-6 eV in total energy, 1.24e-4 in total force, and 7.35e-2 in total stress.

An isolated project-library RT-TDDFT smoke run with ks_solver cusolvermp passed all four checks on 4 V100 GPUs (Slurm job 685177, 109.447 s). This confirms that the test can be used as a small supported-version regression without depending on the full suite result.

Attached artifacts should include the official sample source/log, ABACUS input files, the full runtime log, ldd output, and the old/new cuSolverMp version headers. No proprietary NVIDIA library binaries are required.