fix(plugin-mongodb): write binary, nested and specially named fields from the grid as the values they are

CHANGELOG.md

@@ -132,6 +132,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Pre-connect script failures sometimes reported without the script's own error message.
 - Failed MongoDB statements, including writes the server rejected, reported as successful with an empty result.
 - Save reporting success after leaving out an edit it could not write, such as a new MongoDB document left empty. (#3132)
+- Binary fields missing from duplicated or pasted MongoDB rows. (#3132)
+- Undo of a MongoDB delete refused when the document held binary data. (#3132)
+- Binary field deleted when editing a MongoDB binary cell. (#3132)
+- Error saving a new MongoDB row with every cell empty. (#3132)
+- **Set DEFAULT** on a MongoDB field storing the text `__DEFAULT__`. (#3132)
+- Edits to MongoDB fields named with a dot, a leading `$` or `__proto__` changing another field or nothing. (#3132)
+- Nested MongoDB values shown with sorted keys, and saved with ObjectIds, dates and numbers retyped. (#3132)
 - `tablepro-mcp` crashing when its standard input was non-blocking.
 - `tablepro-mcp` using a full CPU core, or crashing, when its standard output or error was non-blocking.
 - Server connections piling up while browsing many databases or schemas, and staying open after a failed connect. (#3103)

CLAUDE.md

@@ -238,7 +238,7 @@ To ship one: add the record type or field in CloudKit Console (or `xcrun cktool
 
 Four rules follow. A statement whose *answer* is session-scoped is not replayable even though it changes nothing and leaves the footprint clean: `LAST_INSERT_ID`, `ROW_COUNT`, `FOUND_ROWS` and `CONNECTION_ID` are in `mysqlSideEffectingMarkers` for that reason, measured as a fresh connection answering `SELECT LAST_INSERT_ID()` with `0`. The open transaction is the server's own answer rather than a reading of the text, through `mariadb_get_info(MARIADB_CONNECTION_SERVER_STATUS)` in `MariaDBPluginConnection.recordTransactionState` and `footprint.observeServerTransaction`: measured, that reports the transaction `SET autocommit = 0` plus a plain `SELECT` opens, the one inside `/*!40101 BEGIN */` and the one an `XA START` opens, none of which any text scan can see. Only the driver's own statements go in with `countsAsActivity: false`, and only because the driver puts them back itself: the query timeout, and the `USE` behind a database switch, which reconnects through `_activeDatabase` and refreshes `lastActivity` by hand so the idle timer still sees the switch as use. And `/*!40101 ... */` is not a comment, it is SQL the server runs, so `SQLStatementSplitting` keeps it whole rather than stripping it and `MySQLSessionFootprint` reads the body, trailing text included; a mysqldump preamble run from the editor sets the character set, the time zone and eight `@OLD_` variables inside them, and dropping them left the footprint reading clean. The version number is not checked against the server, so a statement the server is too old to run still counts, which holds a clean connection rather than releasing a dirty one.
 
-**A MongoDB update or delete is anchored on `_id` or it does not run**: `generateDelete` used to fall back to a filter built from the remaining columns, which silently dropped every value it could not stringify (all binary) and then `deleteOne`d the first partial match, so a document with a binary `_id` could delete a different document. Both paths now skip with a logged warning instead, matching what `generateUpdate` already did.
+**A MongoDB update or delete is anchored on `_id` or it does not run**: `generateDelete` used to fall back to a filter built from the remaining columns, which silently dropped every value it could not stringify (all binary) and then `deleteOne`d the first partial match, so a document with a binary `_id` could delete a different document. A row with no `_id`, or a binary `_id` whose subtype is not known, now throws `PluginRowWriteRefusal` from `generateRowWrites`, so the whole save is refused with the row named rather than skipped. The same goes for every value the shell cannot carry as the grid shows it (`MongoDBWriteRefusal`): a write the generator cannot express faithfully is refused, never left out or written as something else.
 
 **Redis Cluster routing follows the server's own answer, and the curated table is only a fallback**: `RedisCommandRouting` fetches `COMMAND` once at connect, which supplies key positions on every Redis and, from Redis 7, the `request_policy` / `response_policy` tips that say which commands fan out and how their replies combine. A policy lives on the *subcommand* entry, not the container (`COMMAND INFO config` carries no tips at all; `config|set` is what says `all_nodes`), so the table is keyed `container|sub`. Redis 6 reports no tips, so a parsed reply is merged *over* the curated table rather than replacing it, or `DBSIZE`, `KEYS` and `FLUSHDB` would each go to one shard of a cluster and report success. The curated table is a hand-written list that has to agree with Redis and that nothing at runtime checks, so `scripts/check-redis-command-routing.sh [host] [port]` diffs it against a live Redis 7+; it found 32 disagreements the first time it ran, including a container command hashed on its literal subcommand name and `MSETNX` marked splittable when splitting it breaks the guarantee it exists for. Two rules follow. A container command takes no key of its own, so `OBJECT`, `MEMORY` and `CONFIG` declare no key positions and their keyed subcommands are listed separately, at the index the key sits in the *full* argument list (`OBJECT ENCODING k` puts it at 2, not 1). And an unknown command is routed as keyless rather than by hashing `argv[1]`: a keyless container like `SCRIPT LOAD` answers `+OK` from one node and never sends a `MOVED` to correct the guess.
 

Plugins/ElasticsearchDriverPlugin/ElasticsearchStatementGenerator.swift

@@ -122,8 +122,9 @@ struct ElasticsearchStatementGenerator {
         return .init(method: "POST", path: "/\(encodedIndex)/_doc\(Self.refreshQuery)", body: body)
     }
 
-    /// A new row's leaf value reaches the server only inside its array, so one the user typed, or
-    /// one whose array is empty, would be dropped. Metadata other than `_id` is the server's to set.
+    /// A new row's leaf value reaches the server only inside its array, so one the user typed, one
+    /// whose array is empty, or one that says something other than its array would be dropped.
+    /// Metadata other than `_id` is the server's to set.
     private func unwritableInsertValue(in change: PluginRowChange, values: [String: PluginCellValue]) -> String? {
         for cellChange in change.cellChanges where !cellChange.newValue.isNull {
             let column = cellChange.columnName
@@ -135,15 +136,36 @@ struct ElasticsearchStatementGenerator {
             }
         }
         for column in columns {
-            guard let parent = nestedParentByLeaf[column],
-                  values[column]?.isNull == false,
-                  values[parent]?.isNull ?? true
-            else { continue }
-            return Self.nestedLeafReason(leaf: column, parent: parent)
+            guard let parent = nestedParentByLeaf[column], let leaf = values[column], !leaf.isNull else { continue }
+            guard let array = values[parent]?.asText,
+                  Self.sameCell(leaf, projectedLeaf(column, of: parent, arrayText: array)) else {
+                return Self.nestedLeafReason(leaf: column, parent: parent)
+            }
         }
         return nil
     }
 
+    /// What the grid shows for `leaf` once `arrayText` is saved into `parent`, read by the same
+    /// flattener that reads the document back.
+    private func projectedLeaf(_ leaf: String, of parent: String, arrayText: String) -> PluginCellValue {
+        let placed = parent.split(separator: ".").reversed().reduce(jsonValue(arrayText, for: parent)) { inner, key in
+            [String(key): inner]
+        }
+        guard let source = placed as? [String: Any] else { return .null }
+        return ElasticsearchMappingFlattener.flattenSource(source, nestedParents: [parent])[leaf] ?? .null
+    }
+
+    /// Two cells that hold the same JSON value compare equal however it is spaced.
+    private static func sameCell(_ lhs: PluginCellValue, _ rhs: PluginCellValue) -> Bool {
+        guard let left = lhs.asText, let right = rhs.asText else { return lhs.isNull && rhs.isNull }
+        guard let leftValue = parsedJSON(left), let rightValue = parsedJSON(right) else { return left == right }
+        return (leftValue as AnyObject).isEqual(rightValue)
+    }
+
+    private static func parsedJSON(_ text: String) -> Any? {
+        try? JSONSerialization.jsonObject(with: Data(text.utf8), options: .fragmentsAllowed)
+    }
+
     // MARK: - UPDATE
 
     private func updateRequest(for change: PluginRowChange) throws -> ElasticsearchWriteRequest? {

Plugins/MongoDBDriverPlugin/BsonDocumentFlattener.swift

@@ -35,6 +35,14 @@ enum BsonValueKind: Hashable {
     }
 }
 
+/// Documents as Swift values beside the canonical Extended JSON each was read from. The
+/// dictionaries have lost the stored field order and which numeric type each number was; the text
+/// has not.
+struct MongoReadDocuments {
+    let dictionaries: [[String: Any]]
+    let texts: [String]
+}
+
 struct BsonDocumentFlattener {
     // MARK: - Public API
 
@@ -67,22 +75,47 @@ struct BsonDocumentFlattener {
     }
 
     /// Flatten documents into a grid. Missing fields become nil cells.
-    /// Nested objects/arrays are serialized as compact JSON strings.
+    /// Nested objects/arrays are serialized as compact JSON strings, taken from each document's
+    /// stored text when there is one, so they keep their field order and their BSON types.
     static func flatten(
         documents: [[String: Any]],
         columns: [String],
         kinds: [BsonValueKind],
-        representation: MongoDBUuidRepresentation
+        representation: MongoDBUuidRepresentation,
+        storedTexts: [String] = []
     ) -> [[PluginCellValue]] {
-        documents.map { doc in
-            columns.enumerated().map { index, column in
+        documents.enumerated().map { offset, doc in
+            var stored: [String: MongoDocumentText.Value]?
+            return columns.enumerated().map { index, column in
                 guard let value = doc[column] else { return PluginCellValue.null }
+                if isNestedValue(value), offset < storedTexts.count {
+                    if stored == nil { stored = storedMembers(of: storedTexts[offset]) }
+                    if let member = stored?[column] { return .text(nestedDisplayText(member)) }
+                }
                 let kind = index < kinds.count ? kinds[index] : .string
                 return cellValue(for: value, kind: kind, representation: representation)
             }
         }
     }
 
+    /// A document or an array, and not the `$code` or DBRef shapes that render as their own text.
+    private static func isNestedValue(_ value: Any) -> Bool {
+        if value is [Any] { return true }
+        guard let dict = value as? [String: Any] else { return false }
+        let isCode = dict["$code"] is String
+        let isReference = dict["$ref"] is String && dict["$id"] != nil
+        return !isCode && !isReference
+    }
+
+    private static func storedMembers(of text: String) -> [String: MongoDocumentText.Value] {
+        guard case .object(let members) = try? MongoDocumentText.Value(parsing: text) else { return [:] }
+        return Dictionary(members.map { ($0.key, $0.value) }, uniquingKeysWith: { first, _ in first })
+    }
+
+    static func nestedDisplayText(_ canonical: MongoDocumentText.Value) -> String {
+        JSONTruncation.truncate(MongoExtendedJsonForm.display(canonical).compactText, maxLength: maxNestedJsonLength)
+    }
+
     /// Infer the dominant value kind for each column by majority-vote over document values.
     static func columnKinds(
         for columns: [String],
@@ -387,6 +420,20 @@ struct BsonDocumentFlattener {
         return shared
     }
 
+    /// Every kind each top-level field holds in the documents, nulls aside.
+    static func heldKinds(
+        in documents: [[String: Any]],
+        representation: MongoDBUuidRepresentation
+    ) -> [String: Set<BsonValueKind>] {
+        var held: [String: Set<BsonValueKind>] = [:]
+        for doc in documents {
+            for (field, value) in doc where !(value is NSNull) {
+                held[field, default: []].insert(valueKind(for: value, representation: representation))
+            }
+        }
+        return held
+    }
+
     private static func inferValueKind(
         for field: String,
         in documents: [[String: Any]],

Plugins/MongoDBDriverPlugin/MongoDBBinarySubtypes.swift

@@ -0,0 +1,67 @@
+//
+//  MongoDBBinarySubtypes.swift
+//  MongoDBDriverPlugin
+//
+
+import CryptoKit
+import Foundation
+import TableProPluginKit
+
+/// The BSON subtype of each binary value the grid was handed, found again by the value itself.
+///
+/// A grid cell carries binary data as bytes alone, and a write has to send the subtype back. A
+/// column-wide answer is wrong twice over: one field can hold several subtypes, and the kinds a
+/// driver remembers are overwritten by whichever result it built last, which may be another tab's.
+/// Keying by the field and a digest of the bytes ties the subtype to the value the user copied or
+/// edited, and a value seen with two subtypes answers nothing rather than a guess.
+struct MongoDBBinarySubtypes: Sendable {
+    private var subtypesByField: [String: [Data: Set<UInt8>]] = [:]
+    private(set) var count = 0
+
+    static let empty = MongoDBBinarySubtypes()
+
+    var isEmpty: Bool { subtypesByField.isEmpty }
+
+    /// Every top-level binary value in the documents, which is every value a cell can hold as bytes.
+    static func recording(_ documents: [[String: Any]]) -> MongoDBBinarySubtypes {
+        var recorded = MongoDBBinarySubtypes()
+        for document in documents {
+            for (field, value) in document {
+                if let binary = value as? MongoDBBinaryValue {
+                    recorded.record(binary.data, subtype: binary.subtype, field: field)
+                } else if let data = value as? Data {
+                    recorded.record(data, subtype: 0, field: field)
+                }
+            }
+        }
+        return recorded
+    }
+
+    mutating func record(_ data: Data, subtype: UInt8, field: String) {
+        insert(subtype, digest: Self.digest(of: data), field: field)
+    }
+
+    /// Every subtype the value was seen with in this field: one is the answer, none or several are not.
+    func subtypes(of data: Data, in field: String) -> Set<UInt8> {
+        subtypesByField[field]?[Self.digest(of: data)] ?? []
+    }
+
+    func merging(_ other: MongoDBBinarySubtypes) -> MongoDBBinarySubtypes {
+        var merged = self
+        for (field, digests) in other.subtypesByField {
+            for (digest, subtypes) in digests {
+                subtypes.forEach { merged.insert($0, digest: digest, field: field) }
+            }
+        }
+        return merged
+    }
+
+    private mutating func insert(_ subtype: UInt8, digest: Data, field: String) {
+        guard subtypesByField[field, default: [:]][digest, default: []].insert(subtype).inserted else { return }
+        count += 1
+    }
+
+    private static func digest(of data: Data) -> Data {
+        Data(SHA256.hash(data: data))
+    }
+}

Plugins/MongoDBDriverPlugin/MongoDBCapabilities.swift

@@ -19,6 +19,13 @@ struct MongoDBCapabilities: Sendable, Equatable {
         major >= 4
     }
 
+    /// `$setField` and `$unsetField` arrived in 5.0. Nil when the version is not known, so the
+    /// server answers for itself.
+    var supportsFieldExpressions: Bool? {
+        guard self != .unknown else { return nil }
+        return major >= 5
+    }
+
     static func parse(_ version: String?) -> MongoDBCapabilities {
         guard let version else { return .unknown }
         let parts = version.split(separator: ".")

Plugins/MongoDBDriverPlugin/MongoDBCollectionDDL.swift

@@ -114,15 +114,18 @@ enum MongoDBCollectionDDL {
         return nil
     }
 
-    /// The statement is a JavaScript object literal, and JavaScript lists integer-like keys first in
+    /// The statement is a JavaScript object literal, and JavaScript lists array-index keys first in
     /// ascending order and reads `__proto__` as the prototype rather than as a key, so neither kind of
-    /// name reaches the server where it was written.
-    private static func isReorderedByTheShell(_ name: String) -> Bool {
+    /// name reaches the server where it was written. An array index is an integer from 0 to
+    /// 4294967294 written without a leading zero; a larger one stays where it was written.
+    static func isReorderedByTheShell(_ name: String) -> Bool {
         if name == "__proto__" { return true }
-        guard !name.isEmpty, name.utf8.allSatisfy({ (UInt8(ascii: "0") ... UInt8(ascii: "9")).contains($0) }) else {
+        guard name.utf8.allSatisfy({ (UInt8(ascii: "0") ... UInt8(ascii: "9")).contains($0) }),
+              name == "0" || !name.hasPrefix("0"),
+              let index = UInt32(name) else {
             return false
         }
-        return name == "0" || !name.hasPrefix("0")
+        return index < UInt32.max
     }
 
     // MARK: - Indexes

Plugins/MongoDBDriverPlugin/MongoDBConnection+ScriptHelpers.swift

@@ -16,13 +16,17 @@ struct MongoScriptDocumentBatch: Sendable {
 
     var jsonArray: String { "[\(json.joined(separator: ","))]" }
 
-    var dictionaries: [[String: Any]] {
-        json.compactMap { document in
+    var readDocuments: MongoReadDocuments {
+        var dictionaries: [[String: Any]] = []
+        var texts: [String] = []
+        for document in json {
             guard let data = document.data(using: .utf8),
                   let object = try? JSONSerialization.jsonObject(with: data),
-                  let dictionary = object as? [String: Any] else { return nil }
-            return MongoDBConnection.unwrapExtendedJson(dictionary) as? [String: Any] ?? dictionary
+                  let dictionary = object as? [String: Any] else { continue }
+            dictionaries.append(MongoDBConnection.unwrapExtendedJson(dictionary) as? [String: Any] ?? dictionary)
+            texts.append(document)
         }
+        return MongoReadDocuments(dictionaries: dictionaries, texts: texts)
     }
 }
 

Plugins/MongoDBDriverPlugin/MongoDBFieldKinds.swift

@@ -0,0 +1,48 @@
+//
+//  MongoDBFieldKinds.swift
+//  MongoDBDriverPlugin
+//
+
+import Foundation
+import TableProPluginKit
+
+/// Every kind of value each top-level field held in the documents the grid was handed.
+///
+/// A cell shows a document and a string that reads the same as the same text, and the majority
+/// kind a column is typed by says nothing about one row. A field that has held documents and no
+/// strings shows a document in every cell that opens with `{`; one that has held both cannot say
+/// which a cell holds. Added to rather than replaced, like the binary subtypes, so a string read on
+/// an earlier page is not forgotten when a later page holds only documents.
+struct MongoDBFieldKinds: Sendable {
+    private var kindsByField: [String: Set<BsonValueKind>]
+
+    static let empty = MongoDBFieldKinds([:])
+
+    /// Past this many fields a collection records no new ones, and a field it never recorded is
+    /// treated as unknown rather than as holding one kind.
+    static let fieldLimit = 10_000
+
+    init(_ kindsByField: [String: Set<BsonValueKind>]) {
+        self.kindsByField = kindsByField
+    }
+
+    static func recording(_ documents: [[String: Any]], representation: MongoDBUuidRepresentation) -> MongoDBFieldKinds {
+        MongoDBFieldKinds(BsonDocumentFlattener.heldKinds(in: documents, representation: representation))
+    }
+
+    var isEmpty: Bool { kindsByField.isEmpty }
+
+    /// The kinds the field was seen holding, or nil when it was never seen.
+    func kinds(of field: String) -> Set<BsonValueKind>? {
+        kindsByField[field]
+    }
+
+    func merging(_ other: MongoDBFieldKinds) -> MongoDBFieldKinds {
+        var merged = self
+        for (field, kinds) in other.kindsByField {
+            guard merged.kindsByField[field] != nil || merged.kindsByField.count < Self.fieldLimit else { continue }
+            merged.kindsByField[field, default: []].formUnion(kinds)
+        }
+        return merged
+    }
+}