Skip to content

feat(sync): extend --id-file to source state-load ID-targeting under --minimize-reads - #646

Merged
riyazsh merged 1 commit into
mainfrom
riyazsh/id-file-sync-state-load
Jul 27, 2026
Merged

feat(sync): extend --id-file to source state-load ID-targeting under --minimize-reads#646
riyazsh merged 1 commit into
mainfrom
riyazsh/id-file-sync-state-load

Conversation

@riyazsh

@riyazsh riyazsh commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Extend --id-file to source _state_exact_ids for the sync command with --minimize-reads. Fixes a gap where --minimize-reads degrades to type-scoped loading for types whose state key is not surfaced in the stored body — host_tags, metrics_metadata. Per-invocation state-load transitions from O(N-stored-blobs) to O(K-batch-ids).

Why

For these types, --filter=Type=X;Name=id;Value=…;Operator=ExactMatch matches zero rows (no field matches the state key), so callers omit the filter. extract_exact_id_filters returns None, _state_exact_ids stays None, and State.load_state falls back to reading every blob of the type via _storage.get. With 24k+ blobs and 100-ID batches this is ~244× amplification.

Change

  • Split _ID_FILE_SUPPORTED_TYPES into _ID_FILE_IMPORT_SUPPORTED_TYPES (unchanged: monitors, authn_mappings, team_memberships) and _ID_FILE_STATE_LOAD_SUPPORTED_TYPES (adds host_tags, metrics_metadata).
  • In build_config: parse --id-file before State construction; when --minimize-reads is set and extract_exact_id_filters yields None, source _state_exact_ids from id_payload scoped to --resources.
  • Narrow the import-command per-ID GET gate in resources_handler.py to _ID_FILE_IMPORT_SUPPORTED_TYPES so types added purely for state-load scoping don't enter the per-ID GET path.
  • Update --id-file help text.

Backwards compatibility

  • Import with pre-existing types: unchanged.
  • Sync without --id-file: unchanged (--filter path still wins when it produces exact IDs).
  • Filterable types on sync: unchanged.

Test plan

  • Allowlist membership tests (both allowlists, singletons excluded from both, union invariant).
  • Parser accepts new types, rejects unknown.
  • Help-text drift guard.
  • 53 existing tests pass unchanged.

@riyazsh
riyazsh requested a review from a team as a code owner July 27, 2026 20:09
…--minimize-reads

Adds host_tags and metrics_metadata to _ID_FILE_SUPPORTED_TYPES. For types
whose state key is not surfaced in the stored body, --filter cannot produce
Name=id ExactMatch args (they match zero rows), so extract_exact_id_filters
returns None and --minimize-reads degrades to type-scoped loading — reading
every blob of the type rather than the intended subset.

When --minimize-reads is set and --filter did not yield exact IDs, sourcing
_state_exact_ids from --id-file (scoped to --resources) restores ID-targeted
loading for these types. Import-command --id-file behavior is unchanged; the
--filter path still wins when it produces exact IDs.

Effect: per-invocation state-load transitions from O(N-stored-blobs) to
O(K-batch-ids), matching what --minimize-reads already delivers for
filterable types.
@riyazsh
riyazsh force-pushed the riyazsh/id-file-sync-state-load branch from bdaf628 to 75e1a23 Compare July 27, 2026 20:17
@riyazsh
riyazsh merged commit 2559c7f into main Jul 27, 2026
11 of 12 checks passed
@riyazsh
riyazsh deleted the riyazsh/id-file-sync-state-load branch July 27, 2026 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants