[Feature Request] Cross-Simulation Test Suite?

[emstruong]

Hi Phil,

I think this would be a lot of work, but I'd like to request that perhaps the test suite should have multiple simulation studies that run through: (1) single core runSimulation; (2) parallelrunSimulation; (3) single core runArraySimulation; and (4) parallel runArraySimulation.

I think this is help to make sure that the simulation results are bit-identical regardless of which path you take. Perhaps there is a clever way to write this test code by use of function factories

I mention this because:

1. In my fork, I asked Claude to leverage the latest future and mirai features to enable load-balancing. Such that every parallel core is always doing something within a Design condition, instead of running it in waves. I believe this should improve simulation completion times substantially.
2. It's just helpful to have a strong test suite because it helps block out potential bugs and ensures correctness. The upfront investment makes aggressive downstream refactorings and changes easier to accept.

[philchalmers]

I'm not quite sure what you're asking here, at least relative to the current state of testing. The current test suite evaluates each of the 4 conditions to ensure that things don't break downstream (or if they do, I'm aware and patch them). You could certainly make an offline simulation to do what you're asking, but I'm not quite sure how this would add to the current quality control. Though maybe I'm just missing something in your request.

Regarding additional load balancing on the design object, sure, this could be improved. If you took the entire design and expanded each row based on the number of replications and then parallelized directly across the rows, then you could load balance the whole thing. The issue that I find with this approach is that clusters can't manage the entire simulation all at once, hence the need to break up the jobs into batches, which front-end users have prior insight into. Batches always incur overhead, but it's overhead that's easier to track and manually distribute.

[emstruong]

I'm not quite sure what you're asking here, at least relative to the current state of testing. The current test suite evaluates each of the 4 conditions to ensure that things don't break downstream (or if they do, I'm aware and patch them). You could certainly make an offline simulation to do what you're asking, but I'm not quite sure how this would add to the current quality control. Though maybe I'm just missing something in your request.

I think what I'm wondering is whether the test suite checks that the final output is bit-identical regardless of what approach [runArraySimulation, runSimulation] x [single core, parallel] is taken. That might help things from a correctness point-of-view not just to prevent the prior Jensen inequality issue, but also potential future problems.

The issue that I find with this approach is that clusters can't manage the entire simulation all at once, hence the need to break up the jobs into batches

Ah, I think I may have mis-communicated here. So what I've found is that even if you break up 10000 replications into batches of 200. Let's say that each batch has a 11 core node assigned to it. I've found that it seems like the batch of 200 replications is completed in waves of (11 - 1 = ) 10 replications. However, there are cases in which each replication takes a different amount of time within this batch of 200, either because of some unluckiness in convergence or etc. My understanding is that both future and mirai currently allow load balancing such that within this batch of 200, there is always a core working on something and that this wave behavior within batches is inhibited.

[emstruong]

This may or may not be related to the load-balancing discussion: https://futurize.futureverse.org/articles/futurize-81-SimDesign.html. Perhaps there's a way to remove the complexity of parallel-processing by offloading the code to a different package completely?

[philchalmers]

future is certainly one way to do load balancing in the package. Alternatively, if using the package's default parallel approach with pbapply, you can set the load balancing option globally via

library(pbapply)

# Enable load balancing
pboptions(use_lb = TRUE)

# run either
runSimulation(..., parallel=TRUE)
runArraySimulation(..., parallel=TRUE)

From what I recall this will affect the order with which the replicates are evaluated, but they should be the same replicates (just permuted).

[philchalmers]

Confirmed, at least locally.

library(SimDesign)

Design <- createDesign(N = c(10, 20, 30))


#~~~~~~~~~~~~~~~~~~~~~~~~
#### Step 2 --- Define generate, analyse, and summarise functions

# help(Generate)
Generate <- function(condition, fixed_objects) {
    dat <- with(condition, rnorm(N, 10, 5)) # distributed N(10, 5)
    dat
}

# help(Analyse)
Analyse <- function(condition, dat, fixed_objects) {
    ret <- c(mean=mean(dat)) # mean of the sample data vector
    ret
}

# help(Summarise)
Summarise <- function(condition, results, fixed_objects) {
    # mean and SD summary of the sample means
    ret <- c(mu=mean(results$mean), SE=sd(results$mean))
    ret
}


#~~~~~~~~~~~~~~~~~~~~~~~~
#### Step 3 --- Collect results by looping over the rows in design

seeds <- c(1,2,3)

# run the simulation in testing mode (replications = 2)
Final <- runSimulation(design=Design, replications=100, parallel=TRUE, seed=seeds,
                       generate=Generate, analyse=Analyse, summarise=Summarise)


# Enable load balancing
pbapply::pboptions(use_lb = TRUE)

Final2 <- runSimulation(design=Design, replications=100, parallel=TRUE, seed=seeds,
                       generate=Generate, analyse=Analyse, summarise=Summarise)


Final3 <- runSimulation(design=Design, replications=100, parallel=TRUE, seed=seeds,
                        generate=Generate, analyse=Analyse, summarise=Summarise)


Final
#> # A tibble: 3 × 7
#>       N      mu      SE REPLICATIONS SIM_TIME  SEED COMPLETED               
#>   <dbl>   <dbl>   <dbl>        <dbl> <chr>    <dbl> <chr>                   
#> 1    10 10.029  1.5581           100 0.02s        1 Thu Aug 27 09:23:48 2026
#> 2    20  9.8183 1.0553           100 0.14s        2 Thu Aug 27 09:23:48 2026
#> 3    30 10.079  0.99344          100 0.02s        3 Thu Aug 27 09:23:48 2026
Final2
#> # A tibble: 3 × 7
#>       N      mu      SE REPLICATIONS SIM_TIME  SEED COMPLETED               
#>   <dbl>   <dbl>   <dbl>        <dbl> <chr>    <dbl> <chr>                   
#> 1    10 10.029  1.5581           100 0.02s        1 Thu Aug 27 09:23:51 2026
#> 2    20  9.8183 1.0553           100 0.02s        2 Thu Aug 27 09:23:51 2026
#> 3    30 10.079  0.99344          100 0.02s        3 Thu Aug 27 09:23:51 2026
Final3
#> # A tibble: 3 × 7
#>       N      mu      SE REPLICATIONS SIM_TIME  SEED COMPLETED               
#>   <dbl>   <dbl>   <dbl>        <dbl> <chr>    <dbl> <chr>                   
#> 1    10 10.029  1.5581           100 0.02s        1 Thu Aug 27 09:23:54 2026
#> 2    20  9.8183 1.0553           100 0.02s        2 Thu Aug 27 09:23:54 2026
#> 3    30 10.079  0.99344          100 0.02s        3 Thu Aug 27 09:23:54 2026

^{Created on 2026-08-27 with reprex v2.1.1}

[emstruong]

Well, I was hoping to actually review Claude's work in November then file a pull request to get my contributor status, but seeing as how you're so keen, perhaps I could save you some time by giving you Claude's work? https://github.com/emstruong/SimDesign/tree/claude/simulation-load-balancing-06jgq4

It did identify some problems in implementing load balancing. One of them was getting the halting to respect MAX_TIME on runArraySimulation. Another was the proper exporting of the functions to all the workers.

I was also interested in whether the mori package might be used to lower the memory use and improve efficiency, but I don't know if it would be a good idea to introduce yet another dependency, especially on something relatively untested. I still think that the futurize package might be used to reduce the complexity of the parallel computations in SimDesign and reduce the overall complexity of the code base, which will help with correctness.

[philchalmers]

It did identify some problems in implementing load balancing. One of them was getting the halting to respect MAX_TIME on runArraySimulation. Another was the proper exporting of the functions to all the workers.

I'm sure there's a good number of instances that need testing. Let me know when you get closer. Supporting load balancing is certainly something that should be considered in the package, potentially even as a default if it's feasible.

I was also interested in whether the mori package might be used to lower the memory use and improve efficiency, but I don't know if it would be a good idea to introduce yet another dependency, especially on something relatively untested. I still think that the futurize package might be used to reduce the complexity of the parallel computations in SimDesign and reduce the overall complexity of the code base, which will help with correctness.

I think this would only make sense if the fixed_objects input is really large. Otherwise, all data is generated within the distributed jobs, so there's not much data sharing going on in this context.

[emstruong]

Adding a small note that apparently, according to Claude and as evidence by a CPU usage chart I'm looking at, the progress bar given by pbapply package also creates waves in the parallel workload that is independent of whether the different parallel backends use load balancing or not.

I'm sceptical because this seems bizarre, but something to look into for later.

[philchalmers]

That may be true given the messaging interface behavior, but probably an issue to raise in pbapply if it can be isolated.

[emstruong]

Wow, it's bizarre, but I now have empirical reason to suspect that it's true! progress = TRUE can slow things down substantially!

Yes, I might have to raise this up with pbapply, but in my own experience, I've found that when running things in the R-console, calling things print or cat can slow the execution very substantially compared to running the equivalent code without print or cat. Is that what you meant by messaging interface behavior? What surprises me, however, is that the delay could impact parallel processing so severely!

I wonder what parts of this have to do with how R is, how linux works or something specific to pbapply....