/v1/id/_:id/expanded endpoint

[thehabes]

Closes #285.

Summary

Adds a GET and POST /v1/id/:_id/expanded endpoint — a generic, unauthenticated, browser-cacheable endpoint that returns a RERUM object with the assertions of every current Annotation targeting it already merged in.

Changes

routes/id.js — /:_id/expanded registered ahead of /:_id. GET, POST (behind rest.verifyJsonContentType), and HEAD via Express's native GET handling. Every other method is a 405. OPTIONS is answered 204 by the CORS layer, which sits ahead of the route.

controllers/crud.js — idExpanded() plus three helpers:

• sanitizeExpansionFilters() reduces a POST body to the filter keys this endpoint will honor.
• assertionsFrom() reads only body and bodyValue off an Annotation, so no other property of an Annotation can leak onto the entity.
• applyExpansionAnnotations() merges those assertions onto a clone of the entity and re-appends __rerum last.

db-controller.js — exports idExpanded.

controllers/utils.js — findLeafAnnotationsFor() builds the Annotation search and reads the whole result set from a single cursor, which the driver pages server-side in strides of EXPANSION_BATCH_SIZE (200). The query plan promises no order, so the result is sorted by _id before _id is dropped. That is roughly Annotation creation order, and it makes the assembled entity byte-stable, which is what lets the weak ETag answer a revalidation with 304.

Two smaller fixes in the same file, both needed by this endpoint:

• _contextid() now skips non-string members of an @context Array. An inline term definition object names no context, and the old code called .includes() on it — _contextid(["@context" array starting with an object]) threw a TypeError. Legal JSON-LD, reachable through any /v1/id/ read.
• idNegotiation() restructured to return early instead of cloning up front and conditionally rebuilding. This prevents needless extra cloning between the routes and this helper function.

utils.js

Two changes here, neither specific to /expanded but both on paths it reads through:

• configureRerumOptions() only inherits received.__rerum when update is true. This fixes a bug where client supplied values went into the __rerum property. Affects POST /v1/api/create and POST /v1/api/bulkCreate.
• isContainerType() accepts an Array @type/type and ignores non-string members. JSON-LD serializes a single type as a one-element Array, and with the old code a numeric type threw a TypeError out of configureWebAnnoHeadersFor().

Neither form pages

Both methods gather every Annotation matching their filters before the entity is assembled. A client asks once and receives the whole thing. There is no ?limit= or ?skip= on this endpoint — the 200 in findLeafAnnotationsFor() is an internal cursor stride, not a page size.

GET — the convenience form

Imagined as the plain GET where nothing else is supplied. Recognizes ?generator= and ?creator= only. A value shaped like a URI is matched across both the http and https spellings; any other value is matched exactly. A parameter supplied more than once arrives as an Array and is dropped rather than applied, as is a bracketed form like ?generator[k]=v; the other parameter still applies if it was supplied correctly.

POST — the filtered form

The JSON body is an object of literal MongoDB filter keys, ANDed into the Annotation search — so filtering by generating app means supplying the real property name, __rerum.generatedBy. URL parameters supply no filters here. Not browser-cached.

Because every supplied filter is ANDed in, a filter can only narrow the result set — there is no way to widen it past the entity in the request URI.

The http/https doubling applies to __rerum.generatedBy and creator in this form too, since it lives in the shared helper. Every other key is applied exactly as given.

Response headers

• Cache-Control: max-age=86400, must-revalidate — only on the GET, so browsers can cache for 24h. Annotations on an entity are volatile during initial creation and stable afterward, so this trades a hard reload in the rare stale case for not re-running the expand on every visit.
• Annotations-Gathered — how many current Annotations were gathered for this expansion. This is a complete count, not the size of a page.
• Annotations-Merged — how many of those could contribute. Annotations with multiple bodies are not counted. A counted Annotation may still assert nothing mergeable, so this is not a count of properties received either.

Deleted records

A deleted record is returned exactly as GET /v1/id/:_id returns it, with no expansion attempted and both count headers reporting 0.

controllers/gog.js

Refactored onto the shared helper, which is where the bulk of the deletions come from. Three behavior changes ride along, all on /gog/id/:_id:

• hasOwnProperty → Object.hasOwn. An Annotation body asserting hasOwnProperty would otherwise shadow the method and break the merge. This one is load-bearing, not cosmetic.
• Annotations carrying multiple bodies are now skipped explicitly.
• /gog/id/:_id is now restricted to data generated by a Gallery of Glosses agent and expands only with Annotations from the entity's own generator. A deleted record short-circuits ahead of that gate and returns its tombstone, since a tombstone has no generator to match on.

Measured against production before merging, over the full population rather than a sample:

• 80,154 leaf entities are generated by the GoG production agent.
• 78,022 leaf Annotations target a rerum.io/v1/id/ URI under any of the six target keys.
• 70,299 of those are generated by the GoG production agent. The other 7,723 point at 2,614 distinct RERUM entities, and none of those 2,614 is GoG-generated.

So the generator filter drops nothing today.

Documentation

public/API.html gets both forms under GET and POST, plus an Entity Expansion section carrying the merge rules, the reserved filter keys, the protected properties, and the deleted-record case. It also picks up unrelated markup repairs in passing — unclosed </span>s, a nested <code>, and two malformed <code __rerum.generatedBy> tags.

openapi/contracts/core-provider.openapi.yaml gets /id/{id}/expanded with get, head, and post, and documents the response headers for /id/{id} as well. The reserved filter keys the POST ignores are documented in public/API.html but not yet in the contract.

[cubap]

Reviewed the diff end to end. The __rerum forgery fix and the hasOwnProperty to Object.hasOwn change are both correct and well tested. One thing needs to change before this ships.

Required: strip $-prefixed keys from the POST filter body

sanitizeExpansionFilters() drops four reserved keys (target, type, @type, __rerum.history) but leaves every other key alone. findLeafAnnotationsFor() then pushes each surviving key straight into the query:

queryObj["$and"].push({ [key]: value })

So a client can POST {"$where": "<js>"} and it lands in the query as a live MongoDB operator. $where runs JavaScript on the server, and this endpoint is unauthenticated. That is a real RCE path, scoped to POST /v1/id/:_id/expanded (the GET path only reads ?generator/?creator, so it is safe).

Add a guard in sanitizeExpansionFilters() that drops any key starting with $, and a test asserting a $where or $or key is stripped.

Recommended: confirm two behavior changes

• configureRerumOptions() now ignores client-supplied __rerum when update is false. That changes POST /v1/api/create and bulkCreate, not just the new endpoint. Worth confirming no existing client relied on the old behavior.
• gog.js now returns 404 instead of 403 for non-GoG records. Reasonable, but confirm nothing depended on the 403.

[cubap]

Reviewed the diff end to end. The __rerum forgery fix and the hasOwnProperty to Object.hasOwn change are both correct and well tested. One thing needs to change before this ships.

[cubap]

This drops four reserved keys but leaves every other key alone, including $-prefixed MongoDB operators. findLeafAnnotationsFor() pushes each surviving key straight into the query as { [key]: value }, so a client can POST {"$where": "<js>"} and it lands as a live operator. $where runs JavaScript on the server and this endpoint is unauthenticated, so that is a real RCE path scoped to POST /v1/id/:_id/expanded (the GET path only reads ?generator/?creator, so it is safe).

Add a guard here that drops any key starting with $, and a test asserting a $where or $or key is stripped.