Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
…pache#68090) ### What problem does this PR solve? Related PR: apache#57922 Problem Summary: apache#57922 moved file cache TTL management from block granularity to tablet granularity. As part of that change every path stopped computing an expiration timestamp and started passing the raw ttl_seconds instead: expiration_time = tablet_meta->ttl_seconds(); The receiving side was never updated. KeyMeta::expiration_time still documents itself as an absolute time, and that value is what gets persisted into the block meta store and compared by the cache consistency check. Those fields now hold a duration such as 3600 rather than a timestamp, so the only information left in them is "non-zero means this is a TTL block". Meanwhile the expiration decision itself moved into BlockFileCacheTtlMgr, which sweeps by tablet_ctime + ttl < now, anchored at the tablet creation time. Two consequences: 1. Blocks are created as TTL blocks regardless of whether the tablet is still within its TTL window. Once a tablet is past creation_time + ttl_seconds, the load, compaction and query paths keep putting blocks into the TTL queue and the background sweep keeps pulling them straight back out. Each conversion writes the block meta store and takes the cache lock to move the block between LRU queues, and the loop never terminates for the rest of the tablet's life. 2. The two writer paths disagreed with each other. The packed file path in RowsetWriterContext still computed an absolute newest_write_timestamp + ttl, while the regular file writer path passed the relative ttl_seconds, so segment files and packed small files of the same rowset were stamped with different kinds of value. check_file_cache_consistency reports that as EXPIRATION_TIME_INCONSISTENT. This PR keeps the tablet creation time as the anchor and gives that deadline a single definition, TabletMeta::file_cache_ttl_expiration_time(), which returns the absolute creation_time + ttl_seconds, or 0 when the tablet has no TTL or is already past the deadline. The load, compaction, schema change, query and warm up paths all stamp the blocks they create with that value, so a block's recorded expiration time now agrees with the sweep that acts on it, and a tablet past its deadline has its blocks created as NORMAL directly instead of cycling through the TTL queue. BlockFileCacheTtlMgr additionally no longer promotes the blocks of an already expired tablet on the edge where it first sees that tablet, which previously caused one full promote/demote pass per TTL tablet after every BE restart. The existing regression tests could not catch this: they all create a table and load into it immediately, so the tablet creation time and the data write time coincide and the bug is invisible. ### Release note Fixed the file cache TTL expiration time being computed inconsistently across the load, compaction, query and warm up paths. Data belonging to a tablet that is past its TTL deadline is now written directly into the normal cache queue instead of repeatedly entering and leaving the TTL queue.
liaoxin01
force-pushed
the
codex/pick-68090-to-branch-4.1
branch
from
September 21, 2026 06:30
d7f79aa to
b7b0df5
Compare
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pick #68090