[BUG]: McXtrace and GPU: Indications of thread collisions / undefined behaviour

[willend]

Describe the bug (update on July 8th, 2026)
Looking at today’s output ’nightlies’ output a few tests are off betwen CPU and GPU with indication of thread race conditions:

DONE:

• Issues relating to Multilayer_elliptic were fixed in connection with McXtrace: Work toward fixing #2514 (GPU) #2556
  • (MAXIV_FemtoMAX, Test_ML_elliptic, Test_Mirrors_7)
• Test_Absorption_3 - Abs_objects - fixed via Likely fix for McXtrace / Abs_objects test on GPU #2557
• Test_SX(_1 - i.e. test 1) - Single_crystal pure FIXED: Was using hkl_info.type which is unset on GPU (global mem element)

WORKAROUNDS: (#2558)

• Test_PowderN_2 - Single_crystal in Powder mode
  • Observation: Intensity diverges if powder=1 is used. Worse for long vs. short reflection lists. Disable test-element and write new, specific issue for McXtrace Single_crystal powder=1.
• Test_FluoPowder_5 - Fluorescence + PowderN in GROUP. Output unstable on GPU for index=5 test. idsable test-element and write new, specific issue
• Test_SX_3 - Single_crystal + Fluorescence in GROUP Output unstable on GPU for index=3 test. idsable test-element and write new, specific issue

[willend]

Status July 7th post PR on Multilayer_elliptic

Finding instruments in: /u/data/pkwi/WORK/micromamba/envs/mcxtrace-dev/share/mcxtrace/resources/examples
Copying instruments to: ./mcxtrace-3.99.99_openacc_mpi_x_7_Test_Absorption_Test_FluoPowder_Test_PowderN_Test_SX_1e7_Linux_20260707_2125_56

Compiling instruments [seconds]…
Test_Absorption :  39.65
Test_FluoPowder :  72.56
Test_PowderN    :  59.53
Test_SX         :  29.82

Running tests / getting status…
Test_Absorption  :   Display OK (1s)
Test_Absorption  :   2.80    [val: 8.21872e-12 / 8.25376e-12 = 100 %]
 - Test 1 plots generated OK
Test_Absorption_2:   2.54    [val: 8.21872e-12 / 8.24127e-12 = 100 %]
 - Test 2 plots generated OK
Test_Absorption_3:   2.54    [val: 2.07039e-09 / 8.28866e-12 = 24979 %] <--- BIG DISCREPANCY??
 - Test 3 plots generated OK
Test_Absorption_4:  10.48    [val: 9.28347e-10 / 9.36962e-10 = 99 %]
 - Test 4 plots generated OK
Test_FluoPowder  :   Display OK (1s)
Test_FluoPowder  :   3.66    [val: 2.26053e-18 / 2.23591e-18 = 101 %]
 - Test 1 plots generated OK
Test_FluoPowder_2:   5.52    [val: 1.76752e-18 / 1.89664e-18 = 93 %]
 - Test 2 plots generated OK
Test_FluoPowder_3:  16.31    [val: 2.90443e-18 / 2.8904e-18 = 100 %]
 - Test 3 plots generated OK
Test_FluoPowder_4:  16.34    [val: 6.38342e-19 / 6.18074e-19 = 103 %]
 - Test 4 plots generated OK
Test_FluoPowder_5:  24.92    [val: 6.67297e-19 / 1.76094e-18 = 38 %] <--- BIG DISCREPANCY??
 - Test 5 plots generated OK
Test_PowderN     :   Display OK (1s)
Test_PowderN     :   3.02    [val: 1.26605e-11 / 1.26755e-11 = 100 %]
 - Test 1 plots generated OK
Test_PowderN_2   :   4.37    [val: 6.98986e+62 / 4.44793e-13 = 157148606205583277885437490288713988939454161355953091608284192604256150224896 %] <--- BIG DISCREPANCY??
 - Test 2 plots generated OK
Test_PowderN_3   :   5.79    [val: 1.98296e-11 / 1.98093e-11 = 100 %]
 - Test 3 plots generated OK
Test_SX          :   Display OK (1s)
Test_SX          :   2.79    [val: 0.0 / 2.29638e-13 = 0 %] <--- BIG DISCREPANCY??
 - Test 1 plots generated OK
Test_SX_2        :   4.33    [val: 5.90974e-14 / 6.1e-14 = 97 %]
 - Test 2 plots generated OK
Test_SX_3        :   4.24    [val: 0.0 / 1.19386e-13 = 0 %] <--- BIG DISCREPANCY??
 - Test 3 plots generated OK
======================================
Overall test result:
FAILED! One or more tests errored (0 compile errs / 0 runtime errs / 5 values off)

[willend]

Archive of the above test http://new-nightly.mcxtrace.org/GPUtest-07072026.tgz

[willend]

Incoming fix for Test_Absorption via #2557

(mcxtrace-dev) [pkwi@hypatia tmp]$ mxtest --openacc --instr=Test_Absorption -n1e7
loading system configuration
Output of test will be placed in: .
ncount is: 1e7
openacc is enabled
Testing: 3.99.99

Finding instruments in: /u/data/pkwi/WORK/micromamba/envs/mcxtrace-dev/share/mcxtrace/resources/examples
Copying instruments to: ./mcxtrace-3.99.99_openacc_Test_Absorption_1e7_Linux_20260707_2310_28

Compiling instruments [seconds]…
Test_Absorption :  39.80

Running tests / getting status…
Test_Absorption  :   Display OK (1s)
Test_Absorption  :   3.38    [val: 8.20333e-12 / 8.25376e-12 = 99 %]
 - Test 1 plots generated OK
Test_Absorption_2:   2.54    [val: 8.20333e-12 / 8.24127e-12 = 100 %]
 - Test 2 plots generated OK
Test_Absorption_3:   3.61    [val: 8.20333e-12 / 8.28866e-12 = 99 %]
 - Test 3 plots generated OK
Test_Absorption_4:  56.65    [val: 9.32409e-10 / 9.36962e-10 = 100 %]
 - Test 4 plots generated OK
======================================
Overall test result:
SUCCESS