fix(clustalo/align): wait for compression and propagate pigz failures

[dnncha]

Problem

The compressed output expression uses clustalo ... -o >(pigz ... > result.aln.gz). Bash does not wait for that process substitution as part of the foreground command, and the compressor's status is not included in the Clustal Omega exit status, even with set -euo pipefail.

This can report task success before compression finishes, or after a compressor failure when the producer has already successfully written its buffered output. The .aln.gz file may exist but be incomplete or invalid.

Fix

Append && wait $! to the compressed-output command, with the dollar sign escaped in the Groovy string. The producer's failure is preserved by &&; after producer success, Bash waits for the compressor and returns its failure status. This keeps streaming, avoids an intermediate uncompressed file, and preserves the separation of alignment data from verbose stdout. The uncompressed branch and existing nf-test snapshots are unchanged.

Reproduction and tests

bash -euo pipefail -c 'printf alignment > >(cat >/dev/null; exit 42)'
# exit 0 before
bash -euo pipefail -c 'printf alignment > >(cat >/dev/null; exit 42) && wait $!'
# exit 42 after

Added a dependency-free regression test that extracts the actual module output expression and runs it in Bash with stand-in executables:

python modules/nf-core/clustalo/align/tests/test_compression.py -v

Five cases cover delayed compression completion, compressor failure after draining all input (no SIGPIPE assumption), producer failure, uncompressed output, and separation of stdout progress messages from the compressed alignment. Executed against the pinned original and patched source: 2 failures before; all 5 pass after.

These are shell lifecycle tests, not an alignment-engine or full Nextflow validation. Nextflow/nf-test and the native tools are unavailable in this execution environment; the existing native integration tests still need CI. The standalone regression is run with the command above and is not claimed to be automatically wired into nf-test.

Prepared with AI assistance. Based on upstream commit 781f2625386fe0bf2f337c7b99f56e618c59fc99.

[SPPearce]

This feels complicated.
Why does this tool not just make the output file and then gzip it?

[dnncha]

That would be simpler, and seems a reasonable approach. I kept the existing streaming setup to make the correction small and avoid writing an intermediate uncompressed alignment. The problem is that the foreground command can finish successfully while compression is still running, or after the compressor has failed.

Writing the alignment first and then running pigz would also address that, provided either failure makes the task fail. I’m happy to revise in that direction if that is your preference. I would retain the failure checks and verify the revised module through Nextflow/nf-test; the current five checks use shell stand-ins, so they do not establish native workflow integration.

[SPPearce]

@lrauschning , @erikrikarddaniel , do you have any input on this? I don't know why it is being done in this way.

[erikrikarddaniel]

@lrauschning , @erikrikarddaniel , do you have any input on this? I don't know why it is being done in this way.

No idea. I have no recollection of implementing this and have nothing against doing it the simple way, i.e. gzip after creation of the alignment file. That won't leave an intermediate which would be my only concern in general.

[SPPearce]

Ok, let's do that, make the file and then gzip it if required, as we generally do. Much less error prone.
Don't need that python test at that point.

[lrauschning]

Hi @SPPearce,

I added this to the clustalo (and if memory serves, also to other MSA modules that don't support writing to stdout), as uncompressed MSA output can get very large due to gaps -- essentially, they scale with O(n^2) for diverse sequences. Writing uncompressed output to disk was both using a lot of scratch space and becoming a performance bottlenecks in some tests I ran.
I can't comment on the pytest as I've never seen an nfcore module use anything like it, but the && wait $! solution seems alright to me.

Cheers,
Leon