Speed up notebook testing with papermill

[martinvonk]

I ran all notebooks for steady and transient. For notebooks with runtimes over 10 seconds I added a parameters tag for some variables. With that parameters tag you can overwrite some of the variables at runtime using papermill. For instance: instead of NGR=101, you can set NGR=2 which significantly speeds up the contour plot. The resulting figure rubbish but that's not the point of the test, the point of the test is to see if code runs and the figure is created.

[martinvonk]

Some notebooks are skipped, e.g. ["benchmarking_besselaes.ipynb", "vertical_anisotropy.ipynb"] for the steady notebooks and "besselaesnew_timing.ipynb" for the transient. Is that because they are too slow @dbrakenhoff ?

[mbakker7]

Thanks for adding papermill!
Any reason why the pumpingtests notebooks are not included in the transient section?

[dbrakenhoff]

Some notebooks are skipped, e.g. ["benchmarking_besselaes.ipynb", "vertical_anisotropy.ipynb"] for the steady notebooks and "besselaesnew_timing.ipynb" for the transient. Is that because they are too slow @dbrakenhoff ?

Yes, these are both slow, the benchmarking notebooks take up quite some time because they run lots of loops, the vertical anisotropy also performs lots of loops to compute some statistics.

[martinvonk]

apperenlty this notebook does not exist anymore: benchmarking_besselaes.ipynb :)

[martinvonk]

Probably some more time to be won here but now at least the notebooks > 10s are much faster

[dbrakenhoff]

So one thing we should add is a little note in the notebooks that use calibration that setting max_nfev is not necessary for users, but only done in the examples to allow fast testing of the notebook? Or could we inject a kwargs dict instead of setting the parameter max_nfev? That way the kwargs dict is empty in the notebook, and papermill sets max_nfev?

[mbakker7]

Howbout we have a kwarg with max_iter=100. That is probably clean anyway to give a max value. Then we set max_iter=1 in papermill. Or do we have to set an explicit value for max_iter in the notebook?

[martinvonk]

There are a few ways to do this:

What is done now in the notebook:

MAX_NFEV = 1000
ml.solve(xtol=1e-4, max_nfev=MAX_NFEV)

for papermill MAX_NFEV = 1.

I can do four things:

Option 1: use a dictionary with all settings

SOLVE_KWARGS = {xtol=1e-4}
ml.solve(**SOLVE_KWARGS)

for papermill let SOLVE_KWARGS = {xtol=1e-4, MAX_NFEV=1}

Option 2: use an empty dictionary that's overwritten

SOLVE_KWARGS = {}
ml.solve(xtol=1e-4, **SOLVE_KWARGS)

for papermill let SOLVE_KWARGS = {MAX_NFEV=1}

Option 3: add a comment

MAX_NFEV = 1000 # this is generally not necessary to parse
ml.solve(xtol=1e-4, max_nfev=MAX_NFEV)

for papermill MAX_NFEV = 1.

Option 4: don't add a comment and let it be dependent on the number of parameters which is good practice anyway

MAX_NFEV = 10*len(ml.parameters) 
ml.solve(xtol=1e-4, max_nfev=MAX_NFEV)

for papermill MAX_NFEV = 1.

[dbrakenhoff]

I would suggest option1, users shouldn't really be confronted with setting max_nfev at all in my opinion, so nice if it can be hidden and papermill is secretly using this to reduce max_nfev.

[martinvonk]

Okay done!