feat(sync): extend --id-file to source state-load ID-targeting under --minimize-reads - #646
Merged
Merged
Conversation
…--minimize-reads Adds host_tags and metrics_metadata to _ID_FILE_SUPPORTED_TYPES. For types whose state key is not surfaced in the stored body, --filter cannot produce Name=id ExactMatch args (they match zero rows), so extract_exact_id_filters returns None and --minimize-reads degrades to type-scoped loading — reading every blob of the type rather than the intended subset. When --minimize-reads is set and --filter did not yield exact IDs, sourcing _state_exact_ids from --id-file (scoped to --resources) restores ID-targeted loading for these types. Import-command --id-file behavior is unchanged; the --filter path still wins when it produces exact IDs. Effect: per-invocation state-load transitions from O(N-stored-blobs) to O(K-batch-ids), matching what --minimize-reads already delivers for filterable types.
riyazsh
force-pushed
the
riyazsh/id-file-sync-state-load
branch
from
July 27, 2026 20:17
bdaf628 to
75e1a23
Compare
michael-richey
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extend
--id-fileto source_state_exact_idsfor thesynccommand with--minimize-reads. Fixes a gap where--minimize-readsdegrades to type-scoped loading for types whose state key is not surfaced in the stored body —host_tags,metrics_metadata. Per-invocation state-load transitions fromO(N-stored-blobs)toO(K-batch-ids).Why
For these types,
--filter=Type=X;Name=id;Value=…;Operator=ExactMatchmatches zero rows (no field matches the state key), so callers omit the filter.extract_exact_id_filtersreturnsNone,_state_exact_idsstaysNone, andState.load_statefalls back to reading every blob of the type via_storage.get. With 24k+ blobs and 100-ID batches this is ~244× amplification.Change
_ID_FILE_SUPPORTED_TYPESinto_ID_FILE_IMPORT_SUPPORTED_TYPES(unchanged:monitors,authn_mappings,team_memberships) and_ID_FILE_STATE_LOAD_SUPPORTED_TYPES(addshost_tags,metrics_metadata).build_config: parse--id-filebeforeStateconstruction; when--minimize-readsis set andextract_exact_id_filtersyieldsNone, source_state_exact_idsfromid_payloadscoped to--resources.resources_handler.pyto_ID_FILE_IMPORT_SUPPORTED_TYPESso types added purely for state-load scoping don't enter the per-ID GET path.--id-filehelp text.Backwards compatibility
--id-file: unchanged (--filterpath still wins when it produces exact IDs).Test plan