Skip to content

[Bug]: Cannot clear or rebuild an index after an index schema upgrade #167

Description

@tulayha

Primary affected area

Shared behavior across interfaces

Affected capability

Multiple capabilities

What happened?

When an existing index uses an older schema version, VidXP rejects its manifest. Attempting to reindex or run vidxp index clear --yes also fails because both operations validate that incompatible manifest before proceeding.

Users need a supported way to discard incompatible index data and rebuild from their existing imported media, without manually editing repository files or importing everything into another repository.

Recovery should preserve source media and clearly explain which generated data will be removed. It must also handle incompatible vector collections, since publishing an empty snapshot alone does not reset their embedding dimensions.

This issue concerns recovery after an intentional schema change. It does not require automatic migration or continued search support for old indexes.

How can we reproduce it?

  1. Create an active index with schema version 7.
  2. Open the same repository with code requiring schema version 8, as proposed in Persist real face embeddings during actor indexing (unblocks #74) #151.
  3. Attempt to index an existing media item again.
  4. Attempt vidxp index clear --yes.

Both paths reject the old generation manifest with IndexSchemaError, preventing recovery.

Reproduced with a scene-only generation, so this is not specific to actor indexing.

Environment

  • Operating system: macOS
  • Installation: running from source
  • Python: 3.14.7
  • Storage: local repository with Chroma
  • Scenario: opening a schema version 7 index with code requiring version 8

Logs or other details

clear: IndexSchemaError: Manifest for generation … is invalid.
reindex profile check: IndexSchemaError: Manifest for generation … is invalid.

index_schema_version
  Input should be 8 [input_value=7]

Verified by exercising the shared repository methods used by clearing and reindexing in an isolated local repository; no model inference was required.

LocalSnapshotRepository.clear() and require_compatible_profile() both read and validate the active snapshot before proceeding. Additionally, clearing currently retains the underlying Chroma collections.

Found while reviewing #151; the recovery limitation exists in the repository's current implementation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions