fix: set context=off on database.

[pantierra]

Disables expensive numberMatched/context counting on large collections.

Existing collections need to be migrated manually.

[wrynearson]

@thenav56 could you please review and merge if approved?

[thenav56]

Sure @wrynearson

Merging this PR will update both staging and production. Do you think we should try it on staging first?

[wrynearson]

@pantierra @batpad and @szabozoltan69 are working on this currently, so hold off on merging – I defer the merge/deploy strategy decision to them.

[batpad]

Am pretty sure this is independent of the database cleanups / tuning that @szabozoltan69 and @pantierra are working on. However, this will result in an API change for end-users - i.e. total counts will no longer be available in the API. It would be good to clearly document what the API change will be after setting context to off and what changes for consumers - some of the existing notebooks that use counts, for example, might break.

@pantierra would you be able document / point to documentation of exactly what the API change would be after setting context to off? And then would get a look from maybe @arunissun and @wrynearson . This one is probably good to separate out staging and prod and just deploy to staging first and let people test?

[pantierra]

With context=off, afaik the only visible API change is that numberMatched disappears from /search and /collections responses (no total-count field), while everything else stays the same.

[batpad]

@wrynearson can you take a call and sign off on whether that API change seems okay (and handle any potential comms around it)? And if that seems okay, @thenav56 to approve and merge?

[wrynearson]

@batpad @pantierra The primary use of the numberMatched (AFAIK) is to understand the amount of items, which is used for queries in the LLM-assisted interfaces for broader questions (e.g., how many floods were there in India in the last 5 years).

I think it's OK to turn it off to improve performance. We can explore elsewhere the best way to give broader count info to users (e.g., developmentseed/esa-montandon#33).