Skip to content

Store combined transcript models in long form to bound peak memory - #586

Open
dariansal wants to merge 1 commit into
GoekeLab:develfrom
dariansal:sparse-combine
Open

Store combined transcript models in long form to bound peak memory#586
dariansal wants to merge 1 commit into
GoekeLab:develfrom
dariansal:sparse-combine

Conversation

@dariansal

@dariansal dariansal commented Aug 13, 2026

Copy link
Copy Markdown

Problem

combineSplicedTranscriptModels() builds one table carrying a start, end and readCount column per sample. A read class is observed in only a few samples, so that table is mostly NA, and the full_join fold materialises a second copy while the first is still live.

Peak memory therefore grows with rows × samples rather than with the data, and large sample sets can exhaust memory before the step completes.

Change

Keep only the populated (chain, sample) cells in long form and reduce them by chain.

The reduction also avoids expanding values by their weights. readCountWeightedMedian() repeats each value readCount times, takes the type-7 median and snaps up to the nearest observed value; with integer weights that is exactly the element at floor(N/2)+1 of the expanded sorted vector, locatable from the cumulative weights alone.

Output is unchanged

The existing implementation is kept as combineSplicedTranscriptModelsDense() and runs whenever the sparse path cannot guarantee an identical result:

  • fewer than three samples, where nGroups is 1, no join happens, and the NSample* columns stay logical
  • any input repeating an intron chain within one sample, which the join would fan out

The RNG draw stays where it was, so both paths see the same grouping for a given seed.

Verification

  • upperMedianByGroup() against readCountWeightedMedian() across 50 randomised configurations
  • both paths identical() on real read classes at n=20 and n=40, RNG seeded the same for each; the combine step ran 42.6× and 44.4× faster
  • both paths against each other on synthetic read classes, plus the duplicate-chain fallback

combineSplicedTranscriptModels() builds one table carrying a start, end and
readCount column per sample. A read class is observed in only a few samples, so
that table is mostly NA, and the full_join fold materialises a second copy of it
while the first is still live. Peak memory therefore grows with rows x samples
rather than with the data, and large sample sets can exhaust memory before the
step completes.

This keeps only the populated (chain, sample) cells in long form and reduces
them by chain, so memory scales with the number of observations.

The reduction also avoids expanding values by their weights.
readCountWeightedMedian() repeats each value readCount times, takes the type 7
median and snaps up to the nearest observed value; because the weights are
integers that is exactly the element at floor(N/2)+1 of the expanded sorted
vector, which is locatable from the cumulative weights alone.

Output is unchanged. The previous implementation is kept as
combineSplicedTranscriptModelsDense() and is used whenever the sparse path
cannot guarantee an identical result: fewer than three samples, where no join
happens and the NSample* columns stay logical, and any input repeating an
intron chain within one sample, which the join would fan out.

Tests cover the weighted median against the original function, the two paths
against each other, and the duplicate-chain fallback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant