fix(plugin-mongodb): write binary, nested and specially named fields from the grid as the values they are

[datlechin]

Stacked on #3149, the row-write channel (generateRowWrites, PluginRowWrite, PluginRowWriteRefusal). This PR is its MongoDB adoption. Retarget to main once #3149 merges.

Root cause

The grid's MongoDB writer spelled each value from its text. Four kinds of value were written as something else, and none of them said so.

Binary

• PluginCellValue.bytes has no text, the same as .null. So generateUpdate sent an edited binary cell to $unset and deleted the field.
• generateInsert left binary out of duplicated and pasted rows.
• Undoing the delete of a document that held binary was refused.
• The subtype was already lost when the grid read the value.

Empty rows

• An insert with every cell NULL or DEFAULT returned nil, which the host reported as "Could not generate SQL for changes."

Field names

• The update's $set and $unset keys are paths, measured on 7.0.43:
  • price.usd walked into a sub-document.
  • $price was refused with code 52.
  • The empty name was refused with code 56.
  • __proto__ never reached the server, because the statement is a JavaScript object and that key sets its prototype.

Nested values

• A document or array cell was rebuilt from Swift dictionaries. Keys came out sorted, an ObjectId became hex text, a date lost its milliseconds, a decimal became a bare number, and int32 and int64 printed the same.
• Editing the cell pasted that text back whole, so every value inside was retyped.

Fix

The MongoDB driver now implements generateRowWrites. Every statement names the rows it writes. A change the shell cannot carry as the value the grid shows throws PluginRowWriteRefusal with a reason, so it is never left out. PluginKit is unchanged.

Binary

• Bytes are written as {"$binary": {"base64": …, "subType": …}}.
• The subtype is tied to the value, not the column. MongoDBBinarySubtypes keeps a SHA-256 digest for each field and value on every page read. It adds to the set rather than replacing it, up to 50,000 per collection.
• Edited bytes keep the subtype of the value they replace. Any other bytes keep the subtype they were read with.
• A subtype nothing recorded is never assumed. Only bytes the user typed into a new row get subtype 0, and only where the validator declares the field binData. A new row's change lists the cells the user filled in, so copied cells are told apart from typed ones.
• Everything else with no recorded subtype refuses the save, with a reason that says the subtype is not known:
  • a Data Rewind restore after a relaunch, whose record holds the bytes without their subtype,
  • bytes set over a missing value, which is also how Data Rewind undoes Set NULL,
  • bytes copied from another collection,
  • the same bytes read with two subtypes.
• A binary _id filters on $binary with its own recorded subtype. It no longer borrows the subtype every sampled _id shares: the same bytes under another subtype are another _id, so a borrowed one could delete a different document.
• Emptying a binary cell writes empty bytes of the same subtype. Other text typed over binary is refused.

Empty rows

• NULL and DEFAULT leave a field out of a new document. A row with nothing left inserts {}. A row whose only value was refused throws instead.

Field names

• A row that touches a dotted, $-prefixed, empty or __proto__ name is sent as one pipeline updateOne using $setField and $unsetField. Every value is wrapped in $literal.
• Inside a pipeline, $set of tags.1 rewrites every element of tags and still reports success. So a pipeline row writes every changed field whole and never names a path.
• On a server older than 5.0 the save is refused before anything is sent. Ordinary rows keep the classic $set/$unset.
• A new document refuses only what the insert cannot write on this branch, measured on 7.0.43 through the shell:
  • insertOne and insertMany fail with [22] invalid document for insert: empty key for an empty name at the top level and inside a value. The shell inserts through mongoc_collection_insert_many with nil options, and libmongoc's default validation refuses the key. So an empty name at any depth refuses the row, before anything in the save is sent.
  • insertOne({"__proto__": 5, "a": 1}) succeeds and stores only a. So __proto__ refuses the row too.
  • An update carries both: a classic $set of {"": 6} and $setField of "" or __proto__ all succeed. The refusal says so: set the cell to NULL, save, then set it on the saved document. That route was checked live.

Nested values

• The grid shows each nested cell from the document's stored Extended JSON, in stored order:
  • An int32 is bare.
  • A double always has a point, so a whole-number double shows as 3.0.
  • A date shows as ISO text with milliseconds.
  • $numberLong, $oid, $numberDecimal, $binary and $timestamp keep their wrappers.
• The driver records every kind each top-level field held, across every page read (MongoDBFieldKinds). It adds to the record rather than replacing it, so a string on an earlier page is not forgotten when a later page holds only documents.
• An edit is diffed into $set/$unset on the paths that changed only when that row's value is known to be a document or array: the field held that shape and never a string, which could read the same. The column's majority kind no longer decides it. Otherwise the value is written whole, or refused when even its whole type is ambiguous:
  • JSON in a field that held strings and documents or arrays is refused, since it could be either.
  • Text that is not JSON can only be a string when it was read from a cell or typed over a value that was not a document, so it is written as one.
  • A restored value that reads as JSON, in a field this driver never read, is refused.
  • In a field that held strings and never a document or an array, a value put back, copied into a new row, or typed over a string stays a string however it reads. A stored string {"a":1} duplicated or restored used to be written as a document.
• Where a diff applies, the enclosing value is written whole when:
  • a key cannot be addressed by a path,
  • the type changed,
  • an array's length changed, or
  • more than one key was added. On 7.0.43 the server appends several new keys in its own order: 9, 10, B, _, aa, b.
• A value over 10,000 characters shows shortened, and it is not diffed. A complete value typed over it replaces the field whole. Text that is still shortened is refused.
• A whole value wraps whole-number doubles and integers past 2^53. It refuses an integer past 64 bits.
• It also refuses a key JavaScript would move or drop, rather than store it in another order. Only array-index keys move: whole numbers from 0 to 4294967294, measured in JavaScriptCore, where "4294967295" stays where it was written. The New Table field-name check uses the same rule.
• A timestamp, MinKey or MaxKey inside a value is written as the shell's own Timestamp(t, i), MinKey or MaxKey. The shell's serializer turns every JavaScript number into a wrapper, and libbson reads those three only with bare numbers. A code $scope is written as a document, so its numbers and key order hold too.

Also

• An object counts as an Extended JSON wrapper only when its keys are exactly a wrapper's: $oid alone, $binary with $type, $code with $scope, $regex with $options, and so on. One that opens with $oid and goes on to other members is a document, so {"$oid": "x", "__proto__": 1} is refused like any document the shell would change, rather than written as a wrapper that drops __proto__.
• A pasted Elasticsearch row whose nested leaf column holds something other than what its array holds is refused. The leaf is written only through its array, so it used to be dropped without a word. The leaf is compared with what the flattener reads back from the array, so a faithful paste still saves.
• DEFAULT on an update is refused. It was being written as the text __DEFAULT__.
• An _id edit and a missing _id now refuse the save instead of being skipped.
• A document _id is matched in stored order.
• Export keeps the older nested rendering, so a statement export and a cursor export agree.

Verified

• 16 suites, 389 of 389 passed: MongoDBStatementGeneratorTests, MongoDBWriteBackTypeTests, MongoDBWriteBackBinaryTests, MongoDBFieldNameUpdateTests, MongoDBNestedValueWriteTests, BsonDocumentFlattenerTests, BsonFieldPathTests, BsonFieldPathArrayTests, MongoDocumentTextTests, MongoDocumentWritePlanTests, MongoDBCollectionSchemaTests, MongoDBCollectionDDLTests, MongoScriptPreludeTests, RowChangeStatementFactoryCoverageTests, SaveCompletionTests, SidebarSaveCoverageTests.
• The relaunch case has its own test: a new generator with an empty registry refuses to restore bytes into a field the validator declares binData, where the same restore with the subtype recorded writes 05.
• The reviewer's mixed-field case has its own test: a string {"a":1} in a field of mostly documents, edited to {"a":2}, is refused and never becomes a path.
• Mutation check: four mutations in one run (subtype 0 for any unrecorded bytes, strings ignored when deciding a cell holds a document, the shared _id subtype borrowed again, the nested empty-key check limited to the top level). Ten new cases failed, among them restoreAfterARelaunchRefusesBytes, bytesOverNullWithoutASubtypeAreRefused, copiedBytesInADeclaredFieldAreRefused, binaryIdWithoutARecordedSubtypeIsRefused, jsonLookingStringInAMixedFieldIsRefused and insertRefusesANestedEmptyKey. The source was restored before the final run.
• Two tests run the generated statements through the real prelude in JavaScriptCore. They check that a duplicate and a whole rewrite of a value holding every wrapper kind hand the shell the canonical Extended JSON again.
• Codex review, round 3: a P1 that a string-only field's JSON-shaped string became a document on duplicate or undo-delete, a P2 that a wrapper-first document skipped the key checks, and a P2 that an Elasticsearch paste dropped a leaf that disagreed with its array. All three fixed in the last commit. stringOnlyFieldKeepsJSONShapedText, wrapperShapedDocumentIsCheckedAsADocument and insertRefusesPastedLeafThatDisagreesWithItsArray fail against the round-2 sources and pass here.
• Rebuilt on fix(datagrid): refuse a save that would leave out a change its driver cannot write #3149's head (0eab09894, on main at c21dc512e): the branch's own changes are unchanged, and only main's and fix(datagrid): refuse a save that would leave out a change its driver cannot write #3149's files differ from the round-2 head. The changed suites plus ElasticsearchDriverTests and StringCatalogIntegrityTests: 207 of 207.
• verify.sh build (TablePro and MongoDBDriver) and verify.sh plugins (all 40) pass.
• Lint found 0 violations in the 10 Swift files this round changed. verify.sh docs passes, and every plugin string is in the catalog.
• Live on MongoDB 7.0.43, with a harness running this branch's driver:
  • Relaunch, in a collection whose validator declares sig as binData, on a new driver that had read the schema and another row:
    • A deleted {sig: BinData(5, "AAECAw==")} used to be restored as subtype 00. The restore is now refused. The driver that read the row still restores it as 05.
    • Bytes set back over the missing field used to be written as 00. They are now refused.
    • Bytes typed into a new row are stored as subtype 0. The same bytes copied into a new row are refused.
  • A field holding {a: 1}, {a: 1, b: 2}, {c: 3}, the string '{"a":1}' and the string "{abc":
    • The string row edited to {"a":2} used to send {"$set": {"f.a": 2}}, which failed with [28] Cannot create field 'a' in element. It is now refused before anything is sent.
    • "{abc" edited to "{abcd" used to be refused as not valid JSON. It is now stored as the string {abcd.
    • "{abc" edited to {"x":1} used to turn the string into a document. It is now refused.
    • A field holding only documents still sends {"$set": {"f.a": 5}}.
  • A string read on one page and documents on a later page: an edit of a document is refused. A new driver that read only the documents diffs the same edit.
  • Empty names: the grid edit of the "" cell saves through $setField. A duplicate is refused with the new reason, and with that cell set to NULL it saves. A duplicate of {m: {"": 5, k: 1}} is refused. With m set to NULL it saves, and setting m on the saved copy stores {"": 5, k: 1}.
  • The first two rounds' harness scenarios rerun with the same outcomes: binary duplicate and hex edit, the ambiguous subtype, the binary _id, the empty row, special names, nested edits and duplicates, two tabs, nested timestamps, the 4294967295 key and shortened values.
  • Every probe_grid-serializer_* database was dropped.

Deliberately not fixed here

• Lift the empty-name guard when fix(plugin-mongodb): store fields named "" and top-level $ names on insert #3157 lands. That PR changes the insert options so libmongoc stops refusing empty keys. After it, emptyFieldNameInNewDocument and emptyKeyInNewDocument refuse writes that would succeed, so both cases and their strings should go in the same change. __proto__ stays refused until grid writes leave the JavaScript shell.
• A restore the driver refuses reaches the user as "MongoDB cannot restore a deleted row with its original key." generateIdentityPreservingInsert can only return nil, so the reason (an unknown subtype, an empty key, a shortened value) is logged but not shown. Showing it needs a throwing restore on the row-write channel, which belongs to the channel PR rather than a competing API here.
• In a field that holds strings in some documents and documents or arrays in others, a document can no longer be edited from the grid at all: JSON there could be either type. Telling one row's type needs each value's own type from the read, which the grid cell does not carry.
• In an unvalidated field that holds only strings, text typed into a new row that reads as JSON, a number or true is still written as a document, a number or a boolean, the same as on main, because nothing says the user meant a string. The validator's string declaration keeps it text.
• After a relaunch, a restore types scalar values from their text when their field has not been read since: a date or an ObjectId comes back as a string. This is the same on main.
• NULL versus a missing field. Set NULL still removes the field, as on main. Storing null (Part A) needs the grid to show and remove a missing field too (Part B). Otherwise Data Rewind would write null where it should remove a field. So both go together in a follow-up PR.
• Typed _id on a page mixing _id types, such as the string "1001" next to the int 1001. This needs the row locators from the document-editor work, which is not in this base.
• A top-level timestamp, MinKey or MaxKey shows as text (Timestamp(1700000000, 7), MinKey) in a VARCHAR column, and a duplicate writes that text as a string. This was measured on 7.0.43 and is the same as on main. Nested ones keep their type.
• The inspector still offers Set NULL, Set DEFAULT and SQL functions on MongoDB fields without the grid's gates. The driver now refuses DEFAULT at save.
• Grid writes still go through the JavaScript shell. Moving them to libmongoc, as Insert Document does, would remove the key-order and __proto__ refusals and the number wrapping.
• The top-level field order of a duplicated or restored document follows the grid's columns, not the stored order.
• Found while testing, not fixed:
  • A filter holding $type, such as {sig: {$type: "binData"}}, is refused by libbson as its legacy binary spelling. Fixed in fix(plugin-mongodb): send $type, $regex and $options objects as the operator documents they are written as #3156.
  • The shell sends a whole-number double outside Int64, such as 1e20, as an invalid $numberLong. Fixed in fix(plugin-mongodb): store shell numbers past 2^53 as the doubles they are, define every constructor autocomplete offers, and write stored regexes and decimals back unchanged #3158.

No UI test: CI has no MongoDB server, and every path here needs a connected collection. Unit tests cover the generator, the display and the per-field record, and the prelude round-trip tests cover the shell. The existing SaveCompletionTests and RowChangeStatementFactoryCoverageTests cover the channel's save behaviour.