Reduce peak memory usage during package extraction

[Punisheroot]

Is your feature request related to a problem? Please describe.

While looking through the package installation path, I noticed the existing # TODO: Optimise/parallelise extract in extract_package().

At the moment, each archive member is extracted using the equivalent of:

with open(dest, "wb") as f:
    f.write(zf.read(member))

ZipFile.read() materializes the entire decompressed member as a bytes object before it is written to disk. As a result, peak memory usage during installation can scale with the size of the largest uncompressed member in the runtime package, even though extraction itself can be performed incrementally.

I did a small preliminary microbenchmark with a synthetic deflated archive containing one 64 MiB member plus 100 small files. Across three isolated runs on CPython 3.13.5, tracemalloc reported roughly 141.5 MiB peak Python allocations with the current zf.read(member) approach, compared with roughly 3.2 MiB when reading each member incrementally and copying it in bounded chunks.

This is only a synthetic non-Windows benchmark, so I would not treat the timing results as representative of real PyManager installs. The main motivation is bounding memory usage; any throughput improvement should be measured separately on Windows with actual runtime packages.

Describe the solution you'd like

Would you be open to changing extract_package() so that regular file members are streamed from ZipFile.open() to their destination in bounded chunks rather than first being materialized in memory?

I would keep the initial change deliberately narrow:

• preserve the existing destination/path validation;
• preserve repair and existing-file behaviour;
• preserve the current progress semantics;
• keep the same warnings for out-of-prefix and overwrite attempts;
• add direct tests for the extraction path;
• benchmark the before/after implementation on Windows with representative Python runtime archives.

Although the existing TODO also mentions parallelisation, I would prefer not to combine that with the first change unless there is a clear benchmark showing that parallel extraction is worthwhile. Streaming alone should remove the memory spike without introducing concurrency, ordering, or file-locking complexity.

Describe alternatives you've considered

The other obvious option is parallel extraction. That may improve wall-clock time on some systems, but it also adds significantly more complexity around disk contention, error handling, antivirus scanning, and Windows file locking.

Another option is increasing/decreasing extraction buffers while keeping ZipFile.read(), but that would not address the main issue because each decompressed member would still be allocated in full.

Additional context

I could take this on and include the focused extraction tests and before/after Windows benchmarks in the PR if this direction sounds useful.

[zooba]

It would be interesting to see benchmarks for any of those ideas. I know from experience that the performance impact on tasks like this are fairly unpredictable, so I wouldn't want to presume anything ahead of time.

However, what I was mainly thinking when I left that TODO was for the case where someone is installing multiple versions of Python simultaneously (most likely to be happening with py install -u, which updates all existing installs). Being able to overlap downloads and extracts of multiple runtimes is likely the most complex (certainly displaying it in the console would have a complexity that I don't particularly care to implement or maintain), but I think it probably has the most potential upside.

What do you think, are you up for trying a few approaches and seeing which ones benchmark well?

[Punisheroot]

Sounds good, I'm up for it.

I hadn't really considered the multi-runtime case when I first looked at that TODO, but overlapping work across installs does sound more promising than just parallelising extraction within a single archive.
I'll try a few approaches and I'll post the numbers here once I have something useful!