memory improvements to changelog and uwsgi reload-on-rss fix - #545
Open
fcollman wants to merge 10 commits into
Open
memory improvements to changelog and uwsgi reload-on-rss fix#545fcollman wants to merge 10 commits into
fcollman wants to merge 10 commits into
Conversation
Collaborator
Author
|
I've also added a memory gaurd for the l2graph, and made an option to prelog requests in order to find failed requests for debugging purposes. |
Collaborator
Author
|
i've added a gaurd option on subgraph as well that can be calibrated differently, and switched the code to avoid using reduce and make use of np.concentenate in constructing subgraph return |
/subgraph memory scaled with the total edge content of the chunks an object spans, not with the object. A chunk's edge file holds every object in that chunk, and get_chunk_edges retained all of it: the compressed blobs, the fully decompressed buffers, and a concatenated copy all coexisted before anything was filtered. Measured on minniev7, a ~260-chunk request peaked at 6.28 GiB of a 12 GiB pod limit to return 8.3 MB of edges. Both consumers discard edges that do not touch the queried object -- the edges_only path keeps edges with both endpoints in the supervoxel set, and categorize_edges_v2 drops any edge whose node_ids1 does not remap through sv_parent_d. So resolve the supervoxels first and pass them down, letting each chunk be filtered as it is parsed. The arrays from deserialize are np.frombuffer views into the decompressed chunk, so masking copies out the few edges that matter and lets the buffer be released instead of pinned until the end. Edges.filter_touching keeps edges with either endpoint in the set, which is a superset of both predicates, so filtering early cannot change either result. in_sorted avoids re-sorting the supervoxel set once per chunk. get_children moves above the read to supply the set; the edges_only path now reuses it instead of issuing a second identical read. The ingest caller passes no supervoxels and is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…absent The fallback path already existed for builds without multi-threading support, but only caught ValueError. zstandard >= 0.23 removed multi_decompress_to_buffer altogether, so on any newer version _parse_edges raised AttributeError instead of taking the fallback. The image pins zstandard==0.21.0, which is why this has not bitten in production, but the fallback should not depend on that pin -- and it currently blocks running the edge IO tests outside the image. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
get_chunk_edges issued a single cf.get for every chunk in the request, so the compressed blobs for the whole query were held at once, then handed to multi_decompress_to_buffer which materialized all of the decompressed buffers at once as well. Peak scaled with the number of chunks, which is what made a large bounding box expensive regardless of the size of the object being queried. Process the files in batches instead, accumulating only the parsed (and, when supervoxels are given, already filtered) per-chunk results. Each batch's buffers are released before the next is fetched, so peak tracks batch_size rather than len(fnames). Batch size is configurable via PCG_EDGES_BATCH_SIZE, default 64. This composes with the filter commit and depends on it for most of the benefit: with no supervoxels the parsed arrays are np.frombuffer views that pin their decompressed buffers, so those cannot be released between batches. Filtering copies out the survivors, which is what lets each batch be freed. Tests cover the invariant that neither batching nor filtering changes the result: output is identical across batch sizes 1..1000, filtering matches filtering the fully accumulated set, every file is requested exactly once, and missing chunk files are still skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ic API multi_decompress_to_buffer was removed in zstandard 0.23, and the serial fallback added for that case is roughly 4x slower. That matters: the deployment sets ZSTD_THREADS=4, so the fast path really is parallel today and falling back to serial would be a real regression on any zstandard upgrade. Decompression releases the GIL, so a thread pool recovers all of it. Measured on zstandard 0.21.0, 192 MB across 64 blobs, 4 threads: serial decompressobj() [previous fallback] 164.8 ms 1164 MB/s serial dctx.decompress() [reuse] 150.1 ms 1278 MB/s ThreadPoolExecutor(4) 41.7 ms 4593 MB/s multi_decompress_to_buffer 42.5 ms 4507 MB/s The pool matches the removed API, so this drops the version branch entirely rather than choosing between a fast path and a slow one. ZstdDecompressor is not thread-safe -- sharing one across workers silently produces corrupt output rather than raising, which cost a debugging round here -- so each worker keeps its own via threading.local. dctx.decompress needs the content size in the frame header, which put_chunk_edges writes and which multi_decompress_to_buffer also required; decompressobj covers any frame lacking it. Verified against both zstandard 0.21.0 (the image pin) and 0.25.0: identical output for 1/2/4/8 threads, threaded matches serial, order preserved, empty input, and frames written with write_content_size=False. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This limits the columns read by the tabular_changelog_recent endpoint and adds a uwsgi parameter to reload workers that have large memory usage, which should limit memory leaks.