Fix memory leak in caching execution information location, fixes #8623 - #8625
Draft
mattcasters wants to merge 3 commits into
Draft
mattcasters wants to merge 3 commits into
mattcasters wants to merge 3 commits into
Conversation
…he#8623 - Bound in-memory cache to 50 LRU entries by default with LinkedHashMap - Reduce default maxCacheAge to 10 minutes (600000 ms) - Add maxCacheSize and update maxCacheAge GUI widgets and i18n - Enforce LRU eviction in BaseCachingExecutionInfoLocation and clear cache on close() - Fix CacheEntry.isTooOld() to eliminate immortal unwritten entries - Fix query and PreparedStatement collision/leak in CachingDatabaseExecutionInfoLocation.retrieveIds() - Reuse Jackson ObjectMapper with HopJson.newMapper() - Protect Database.insertRow() with try-finally closeInsert() - Harden Pipeline and LocalPipelineEngine lifecycle so timers and locations are always closed on completion, abort, or error
… write The live pipeline entry copied the whole log buffer on every tick. Store only the new lines and keep the newest 2 million characters. Leave an entry in memory when its save fails, and do not close the location on a safe stop. A missing max cache age stays at one day; new locations use 10 minutes.
…n info A single-threaded parent drives the mapping again on every iteration, and the child does not finish between batches. A location on that child kept a second caching session open for the whole parent run. Drop the location from a copy of the run configuration, and stop the child on dispose.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #8623
Description
This PR addresses and fixes the memory leak observed when using the caching database (and file) execution information locations:
Bounded In-Memory LRU Cache:
HashMapinBaseCachingExecutionInfoLocationwith an access-orderLinkedHashMap.maxCacheSizemetadata property and GUI widget (default:50, order"905"), configurable in the metadata dialog.maxCacheAgedefault from 24 hours to 10 minutes (600000ms) and updated i18n tooltips and labels.enforceMaxCacheSize()which evicts LRU entries and persists dirty entries prior to eviction.close()clears the cache (finally { cache.clear(); }) and synchronizedclearCaches().Eliminated Immortal Unwritten Cache Entries:
CacheEntry.isTooOld(), fixed the condition so entries fall back tocreationDatewhenlastReadandlastWrittenare null, preventing entries from lingering indefinitely.Prevented PreparedStatement Collision & Statement Leak in
retrieveIds():CachingDatabaseExecutionInfoLocation.retrieveIds(), collected parent IDs into a list first and closed the outer query/ResultSetbefore callingloadCacheEntry()for child executions, preventing statement overwrite and leak onDatabase.pstmt.ObjectMapperinstance withHopJson.newMapper().Resource Safety in Core Database:
Database.insertRow(schemaName, tableName, fields, data), wrapped insert execution intry ... finally { closeInsert(); }to ensure prepared statements are always closed even on error.Lifecycle and Execution Info Timer Hardening:
Pipeline.fireExecutionFinishedListeners(), ensuredpipelineCompleted(), extension point, andreleaseVfsNamespace()are guaranteed to execute even if a listener throws an exception.LocalPipelineEngine, madestopTransformExecutionInfoTimer()synchronized and idempotent, stopped the timer on startup failure instartThreads(), and registered an execution-stopped listener to trigger timer/location cleanup.Tests:
CachingDatabaseExecutionInfoLocationTest:lruCacheEvictionEnforcesMaxSize: verifies LRU eviction bounds memory atmaxCacheSizewhile evicted entries remain persistent in the database.closeClearsCacheMap: verifies memory cache is cleared onclose().retrieveIdsWithChildrenLoadsChildrenCorrectly: verifies child entry retrieval works without statement collisions.mvn clean install apache-rat:checkto make sure basic checks pass. A more thorough check will be performed on your pull request automatically.git rebase -i.addresses #123), if applicable.To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.