From 95a18f2729bc3ff17c685316bf82a6ecc167026d Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Sat, 26 Sep 2026 16:18:28 +0700 Subject: [PATCH] fix(plugin-mongodb): store Set NULL as null, tell a missing field from null, and add Remove Field --- CHANGELOG.md | 5 + .../BsonDocumentFlattener.swift | 11 + .../MongoDBCollectionSchema.swift | 76 ++- .../MongoDBDriverPlugin/MongoDBPlugin.swift | 1 + .../MongoDBPluginDriver.swift | 17 +- .../MongoDBStatementGenerator.swift | 50 +- .../MongoDBWriteRefusal.swift | 4 +- .../MongoScriptResultBuilder.swift | 8 +- Plugins/TableProPluginKit/DriverPlugin.swift | 7 + .../PluginDatabaseDriver.swift | 15 + .../TableProPluginKit/PluginQueryResult.swift | 5 + .../ChangeTracking/AnyChangeManager.swift | 53 ++ .../ChangeTracking/DataChangeManager.swift | 204 +++++--- .../ChangeTracking/DataChangeModels.swift | 74 ++- .../Core/ChangeTracking/PendingChanges.swift | 124 +++-- .../Coordinators/PaginationCoordinator.swift | 2 +- .../QueryExecutionCoordinator+Batches.swift | 3 +- .../QueryExecutionCoordinator+Helpers.swift | 6 +- ...yExecutionCoordinator+MultiStatement.swift | 3 +- ...QueryExecutionCoordinator+Parameters.swift | 3 +- .../RowEditingCoordinator+Discard.swift | 6 +- .../Coordinators/RowEditingCoordinator.swift | 52 +- TablePro/Core/DataWrite/DataWritePlan.swift | 22 + .../DataWrite/Rewind/RewindExecutor.swift | 14 +- .../Core/DataWrite/Rewind/RewindPlanner.swift | 101 +++- .../DataWrite/RowChangeStatementFactory.swift | 27 +- .../DataWrite/RowWriteOperationBuilder.swift | 10 +- .../Core/Plugins/PluginDriverAdapter.swift | 1 + .../Plugins/PluginManager+Registration.swift | 5 + ...ginMetadataRegistry+RegistryDefaults.swift | 1 + .../Core/Plugins/PluginMetadataRegistry.swift | 2 + .../Infrastructure/ClipboardService.swift | 20 + .../Core/Services/Query/QueryExecutor.swift | 12 +- .../Services/Query/RowOperationsManager.swift | 114 +++-- TablePro/Models/Query/ParsedRow.swift | 3 + TablePro/Models/Query/QueryResult.swift | 3 + TablePro/Models/Query/Row.swift | 7 + TablePro/Models/Query/TableRows.swift | 78 ++- .../Query/VisibleColumnProjection.swift | 6 + TablePro/Models/UI/FieldValueState.swift | 13 +- TablePro/Models/UI/MultiRowEditState.swift | 108 +++- TablePro/Resources/Localizable.xcstrings | 14 +- .../MainContentCoordinator+QueryHelpers.swift | 6 +- ...MainContentCoordinator+RowOperations.swift | 14 +- .../MainContentCoordinator+SidebarSave.swift | 11 +- .../MainContentView+EventHandlers.swift | 14 +- .../Views/Main/MainContentCoordinator.swift | 3 +- .../Cells/DataGridCellAppearance.swift | 2 +- .../Results/Cells/DataGridCellContent.swift | 11 +- .../Views/Results/DataGridCellFactory.swift | 4 +- TablePro/Views/Results/DataGridRowView.swift | 38 +- .../Results/DataGridView+RowActions.swift | 36 +- .../Extensions/DataGridView+CellCommit.swift | 50 +- .../Extensions/DataGridView+Columns.swift | 5 +- .../Extensions/DataGridView+Editing.swift | 6 + .../FieldEditors/BooleanPickerView.swift | 7 +- .../FieldEditors/EnumPickerView.swift | 7 +- .../FieldEditors/FieldMenuContent.swift | 9 +- .../FieldEditors/FieldPickerSentinel.swift | 22 +- .../RowInspector/InspectorFieldListView.swift | 7 + .../RowInspector/InspectorFieldRow.swift | 4 + .../FieldAbsenceTrackingTests.swift | 286 +++++++++++ .../DataWrite/FieldAbsenceWriteTests.swift | 471 ++++++++++++++++++ .../Core/DataWrite/RewindPlannerTests.swift | 20 +- .../MongoDB/BsonDocumentFlattenerTests.swift | 20 +- .../PluginFieldRemovalCapabilityTests.swift | 82 +++ .../RowOperationsFieldAbsenceTests.swift | 154 ++++++ .../Models/FieldValueStateTests.swift | 32 ++ .../MultiRowEditStateFieldAbsenceTests.swift | 158 ++++++ .../MongoDBCollectionSchemaTests.swift | 104 ++++ .../Plugins/MongoDBFieldNameUpdateTests.swift | 13 +- .../MongoDBFieldPresenceWriteTests.swift | 137 +++++ .../MongoDBStatementGeneratorTests.swift | 35 +- .../Plugins/MongoDBWriteBackBinaryTests.swift | 12 +- .../Main/InspectorFieldEditStagingTests.swift | 6 +- .../Views/Main/SidebarSaveCoverageTests.swift | 4 +- .../Main/ValueFilterEditedRowTests.swift | 2 +- .../Results/DataGridFieldRemovalTests.swift | 139 ++++++ docs/databases/mongodb.mdx | 14 +- docs/features/change-tracking.mdx | 2 + docs/features/table-structure.mdx | 2 +- 81 files changed, 2862 insertions(+), 377 deletions(-) create mode 100644 TableProTests/Core/ChangeTracking/FieldAbsenceTrackingTests.swift create mode 100644 TableProTests/Core/DataWrite/FieldAbsenceWriteTests.swift create mode 100644 TableProTests/Core/Plugins/PluginFieldRemovalCapabilityTests.swift create mode 100644 TableProTests/Core/Services/RowOperationsFieldAbsenceTests.swift create mode 100644 TableProTests/Models/MultiRowEditStateFieldAbsenceTests.swift create mode 100644 TableProTests/Plugins/MongoDBFieldPresenceWriteTests.swift create mode 100644 TableProTests/Views/Results/DataGridFieldRemovalTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index b154604f31..4bf1e29e18 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - **Insert Document…** for MongoDB collections, written as Extended JSON. (#3132) +- **Remove Field** for MongoDB cells, and **No Field** for a field a document does not have. (#3132) - **Agent** mode: one session with the whole connection window, sessions to start and delete, and what each one ran. - Row previews and the query editor sized to the display on iPad and on iPhone Duo's inner display. - Table list and table browser side by side on iPad and on iPhone Duo's inner display. @@ -139,6 +140,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Set DEFAULT** on a MongoDB field storing the text `__DEFAULT__`. (#3132) - Edits to MongoDB fields named with a dot, a leading `$` or `__proto__` changing another field or nothing. (#3132) - Nested MongoDB values shown with sorted keys, and saved with ObjectIds, dates and numbers retyped. (#3132) +- **Set NULL** on a MongoDB field deleting the field. (#3132) +- Null fields dropped from duplicated, pasted or restored MongoDB documents. (#3132) +- **Restore Previous Values** comparing the wrong MongoDB fields when a document lacks one. (#3132) +- Row inspector accepting edits to a MongoDB `_id` or another read-only field that were never saved. (#3132) - `tablepro-mcp` crashing when its standard input was non-blocking. - `tablepro-mcp` using a full CPU core, or crashing, when its standard output or error was non-blocking. - Server connections piling up while browsing many databases or schemas, and staying open after a failed connect. (#3103) diff --git a/Plugins/MongoDBDriverPlugin/BsonDocumentFlattener.swift b/Plugins/MongoDBDriverPlugin/BsonDocumentFlattener.swift index 35d4706234..be250cea97 100644 --- a/Plugins/MongoDBDriverPlugin/BsonDocumentFlattener.swift +++ b/Plugins/MongoDBDriverPlugin/BsonDocumentFlattener.swift @@ -98,6 +98,17 @@ struct BsonDocumentFlattener { } } + /// The columns each document has no field for, by row. A field holding null is there, and only + /// rows missing something appear. + static func absentCells(of documents: [[String: Any]], columns: [String]) -> [Int: Set] { + var absent: [Int: Set] = [:] + for (row, document) in documents.enumerated() { + let missing = Set(columns.indices.filter { document[columns[$0]] == nil }) + if !missing.isEmpty { absent[row] = missing } + } + return absent + } + /// A document or an array, and not the `$code` or DBRef shapes that render as their own text. private static func isNestedValue(_ value: Any) -> Bool { if value is [Any] { return true } diff --git a/Plugins/MongoDBDriverPlugin/MongoDBCollectionSchema.swift b/Plugins/MongoDBDriverPlugin/MongoDBCollectionSchema.swift index a6a4604d3a..5c7629f8f7 100644 --- a/Plugins/MongoDBDriverPlugin/MongoDBCollectionSchema.swift +++ b/Plugins/MongoDBDriverPlugin/MongoDBCollectionSchema.swift @@ -48,6 +48,10 @@ struct MongoDBDeclaredField: Equatable, Sendable { let bsonTypes: [String] let isRequired: Bool let allowedValues: [String]? + /// Whether the validator lets the field hold null. Separate from `isRequired`, which is about + /// the field being there at all: a required field typed `["date", "null"]` takes null, and an + /// optional field typed `string` refuses it. + var admitsNull = true var valueKind: BsonValueKind? { let valueTypes = bsonTypes.filter { $0 != "null" } @@ -83,6 +87,16 @@ enum MongoDBCollectionShape { struct MongoDBCollectionSchema: Equatable, Sendable { let fields: [MongoDBDeclaredField] + /// Whether a field the validator's `properties` do not name may hold null: an + /// `additionalProperties` schema decides it for every such field. + var undeclaredFieldsAdmitNull = true + /// A rule this reader does not model reaches every field: a combinator, `patternProperties`, + /// `dependencies` or an `enum` over the whole document, or a query operator beside + /// `$jsonSchema`. Each can refuse null in any field, so none is taken to admit it. + var hasUnmodeledRule = false + /// False for `validationAction: "warn"` and `validationLevel: "off"`, where the server stores + /// what the validator would refuse. + var isEnforced = true static let empty = MongoDBCollectionSchema(fields: []) @@ -92,6 +106,13 @@ struct MongoDBCollectionSchema: Equatable, Sendable { fields.first { $0.name == name } } + /// Whether the server takes null in this field, which is what offers **Set Value > NULL**. + func admitsNull(fieldNamed name: String) -> Bool { + guard isEnforced else { return true } + guard !hasUnmodeledRule else { return false } + return field(named: name)?.admitsNull ?? undeclaredFieldsAdmitNull + } + var valueKinds: [String: BsonValueKind] { var kinds: [String: BsonValueKind] = [:] for field in fields { @@ -127,32 +148,44 @@ struct MongoDBCollectionSchema: Equatable, Sendable { let batch = MongoScriptJson.member(of: cursor, key: "firstBatch"), let collection = MongoScriptJson.topLevelElements(batch).first, let options = MongoScriptJson.member(of: collection, key: "options"), - let validator = MongoScriptJson.member(of: options, key: "validator"), - let jsonSchema = MongoScriptJson.member(of: validator, key: "$jsonSchema") else { + let validator = MongoScriptJson.member(of: options, key: "validator") else { return .empty } - return parse(jsonSchema: jsonSchema) + var schema = MongoScriptJson.member(of: validator, key: "$jsonSchema").map { parse(jsonSchema: $0) } ?? .empty + if MongoScriptJson.members(of: validator).contains(where: { $0.key != "$jsonSchema" }) { + schema.hasUnmodeledRule = true + } + let settings = decodeObject(options) ?? [:] + schema.isEnforced = settings["validationAction"] as? String != "warn" + && settings["validationLevel"] as? String != "off" + return schema } static func parse(jsonSchema: String) -> MongoDBCollectionSchema { - guard let schema = decodeObject(jsonSchema), - let properties = schema["properties"] as? [String: Any], - let propertiesText = MongoScriptJson.member(of: jsonSchema, key: "properties") else { - return .empty + guard let schema = decodeObject(jsonSchema) else { return .empty } + return MongoDBCollectionSchema( + fields: declaredFields(in: schema, text: jsonSchema), + undeclaredFieldsAdmitNull: undeclaredFieldsAdmitNull(schema["additionalProperties"]), + hasUnmodeledRule: documentWideRules.contains { schema[$0] != nil } + ) + } + + private static func declaredFields(in schema: [String: Any], text: String) -> [MongoDBDeclaredField] { + guard let properties = schema["properties"] as? [String: Any], + let propertiesText = MongoScriptJson.member(of: text, key: "properties") else { + return [] } let required = Set(schema["required"] as? [String] ?? []) - let orderedNames = orderedKeys(of: propertiesText, in: properties) - - let fields = orderedNames.compactMap { name -> MongoDBDeclaredField? in + return orderedKeys(of: propertiesText, in: properties).compactMap { name -> MongoDBDeclaredField? in guard let spec = properties[name] as? [String: Any] else { return nil } return MongoDBDeclaredField( name: name, bsonTypes: declaredTypes(in: spec), isRequired: required.contains(name), - allowedValues: stringEnum(spec["enum"]) + allowedValues: stringEnum(spec["enum"]), + admitsNull: admitsNull(spec) ) } - return MongoDBCollectionSchema(fields: fields) } private static func orderedKeys(of objectText: String, in decoded: [String: Any]) -> [String] { @@ -174,6 +207,25 @@ struct MongoDBCollectionSchema: Equatable, Sendable { return [] } + /// The server applies every keyword of a field's rule, so null passes only when each one that + /// can refuse it lets it through: a declared type has to list null, and an `enum` has to hold + /// it. A combinator is not modeled and is taken to refuse null. The keywords for strings, + /// numbers, arrays and documents never see null, measured on 7.0 for `minLength` and `pattern`. + private static func admitsNull(_ spec: [String: Any]) -> Bool { + let typeAdmitsNull = (spec["bsonType"] == nil && spec["type"] == nil) || declaredTypes(in: spec).contains("null") + let enumAdmitsNull = (spec["enum"] as? [Any])?.contains { $0 is NSNull } ?? true + return typeAdmitsNull && enumAdmitsNull && !unmodeledConstraints.contains { spec[$0] != nil } + } + + private static let unmodeledConstraints = ["anyOf", "oneOf", "allOf", "not"] + + private static let documentWideRules = unmodeledConstraints + ["dependencies", "patternProperties", "enum"] + + private static func undeclaredFieldsAdmitNull(_ additionalProperties: Any?) -> Bool { + if let spec = additionalProperties as? [String: Any] { return admitsNull(spec) } + return additionalProperties as? Bool ?? true + } + private static func stringEnum(_ value: Any?) -> [String]? { guard let array = value as? [Any], !array.isEmpty, array.allSatisfy({ $0 is String }) else { return nil } diff --git a/Plugins/MongoDBDriverPlugin/MongoDBPlugin.swift b/Plugins/MongoDBDriverPlugin/MongoDBPlugin.swift index 10bd65fe46..3a6ae2fd61 100644 --- a/Plugins/MongoDBDriverPlugin/MongoDBPlugin.swift +++ b/Plugins/MongoDBDriverPlugin/MongoDBPlugin.swift @@ -16,6 +16,7 @@ final class MongoDBPlugin: NSObject, TableProPlugin, DriverPlugin { static let supportsRenameTable = true static let supportsDocumentEditing = true + static let supportsFieldRemoval = true static let databaseDisplayName = "MongoDB" static let iconName = "mongodb-icon" static let defaultPort = 27_017 diff --git a/Plugins/MongoDBDriverPlugin/MongoDBPluginDriver.swift b/Plugins/MongoDBDriverPlugin/MongoDBPluginDriver.swift index 136c99daa1..55c3202ddf 100644 --- a/Plugins/MongoDBDriverPlugin/MongoDBPluginDriver.swift +++ b/Plugins/MongoDBDriverPlugin/MongoDBPluginDriver.swift @@ -266,7 +266,7 @@ final class MongoDBPluginDriver: PluginDatabaseDriver, @unchecked Sendable { guard let rowCap, MongoDBFindLimitPolicy.isTruncated(rowCount: result.rows.count, rowCap: rowCap) else { return result } - return PluginQueryResult( + var capped = PluginQueryResult( columns: result.columns, columnTypeNames: result.columnTypeNames, rows: Array(result.rows.prefix(rowCap)), @@ -275,6 +275,8 @@ final class MongoDBPluginDriver: PluginDatabaseDriver, @unchecked Sendable { isTruncated: true, statusMessage: result.statusMessage ) + capped.absentCells = result.absentCells?.filter { $0.key < rowCap } + return capped } private func mapExecutionError(_ error: Error) -> Error { @@ -348,7 +350,7 @@ final class MongoDBPluginDriver: PluginDatabaseDriver, @unchecked Sendable { PluginColumnInfo( name: name, dataType: BsonDocumentFlattener.typeName(for: kinds[index], representation: uuidRepresentation), - isNullable: name != MongoDBCollectionDDL.idField && schema.field(named: name)?.isRequired != true, + isNullable: name != MongoDBCollectionDDL.idField && schema.admitsNull(fieldNamed: name), isPrimaryKey: name == MongoDBCollectionDDL.idField, defaultValue: nil, extra: nil, charset: nil, collation: nil, comment: nil, allowedValues: schema.allowedValues[name] @@ -365,7 +367,7 @@ final class MongoDBPluginDriver: PluginDatabaseDriver, @unchecked Sendable { name: name, dataType: field?.columnTypeName(representation: uuidRepresentation) ?? BsonDocumentFlattener.typeName(for: .objectId, representation: uuidRepresentation), - isNullable: !isKey && field?.isRequired != true, + isNullable: !isKey && schema.admitsNull(fieldNamed: name), isPrimaryKey: isKey, defaultValue: nil, extra: nil, charset: nil, collation: nil, comment: nil, allowedValues: schema.allowedValues[name] @@ -807,9 +809,10 @@ final class MongoDBPluginDriver: PluginDatabaseDriver, @unchecked Sendable { schema: String?, columns: [String], primaryKeyColumns: [String], - rows: [[PluginCellValue]] + rows: [[PluginCellValue]], + absentCells: [Int: Set] ) -> [(statement: String, parameters: [PluginCellValue])]? { - writeGenerator(for: table, columns: columns).generateRestore(rows: rows) + writeGenerator(for: table, columns: columns).generateRestore(rows: rows, absentCells: absentCells) } private func writeGenerator(for table: String, columns: [String]) -> MongoDBStatementGenerator { @@ -955,12 +958,14 @@ final class MongoDBPluginDriver: PluginDatabaseDriver, @unchecked Sendable { representation: uuidRepresentation, storedTexts: read.texts ) - return PluginQueryResult( + var result = PluginQueryResult( columns: columns, columnTypeNames: typeNames, rows: rows, rowsAffected: 0, executionTime: Date().timeIntervalSince(startTime), isTruncated: isTruncated ) + result.absentCells = BsonDocumentFlattener.absentCells(of: documents, columns: columns) + return result } // MARK: - Helpers diff --git a/Plugins/MongoDBDriverPlugin/MongoDBStatementGenerator.swift b/Plugins/MongoDBDriverPlugin/MongoDBStatementGenerator.swift index 319b6519b5..c01c194833 100644 --- a/Plugins/MongoDBDriverPlugin/MongoDBStatementGenerator.swift +++ b/Plugins/MongoDBDriverPlugin/MongoDBStatementGenerator.swift @@ -110,8 +110,9 @@ struct MongoDBStatementGenerator { // MARK: - INSERT - /// NULL and DEFAULT leave a field out of a new document. A row with nothing else in it is the - /// empty document, which the server stores with a generated `_id`. + /// A field the new row has none of, and DEFAULT, leave the field out of the document; NULL is + /// written as null, except in `_id`, which the grid never lets anyone set. A row with nothing + /// else in it is the empty document, which the server stores with a generated `_id`. /// /// A new row's change lists the cells the user filled in; every other value came with the row, /// copied from the row it duplicates or pastes. @@ -119,23 +120,28 @@ struct MongoDBStatementGenerator { for change: PluginRowChange, insertedRowData: [Int: [PluginCellValue]] ) throws -> String { - let filledIn = change.cellChanges.map { (field: $0.columnName, value: $0.newValue) } + let filledIn = change.cellChanges.map { (column: $0.columnIndex, field: $0.columnName, value: $0.newValue) } let typed = Dictionary(filledIn.map { ($0.field, $0.value) }, uniquingKeysWith: { _, last in last }) - let cells: [(field: String, value: PluginCellValue)] + let cells: [(column: Int, field: String, value: PluginCellValue)] if let values = insertedRowData[change.rowIndex] { - cells = zip(columns, values).map { (field: $0, value: $1) } + cells = zip(columns, values).enumerated().map { (column: $0, field: $1.0, value: $1.1) } } else { cells = filledIn } - let entries = try cells.filter { !isLeftOut($0.value) }.map { cell in + let absent = change.absentColumns ?? [] + let written = cells.filter { cell in + let serverAssignsIdentity = cell.field == MongoDBCollectionDDL.idField && cell.value.isNull + return !serverAssignsIdentity && !isLeftOut(cell.value, isAbsent: absent.contains(cell.column)) + } + let entries = try written.map { cell in let provenance: Provenance = typed[cell.field] == cell.value ? .typedIntoNewDocument : .copiedIntoNewDocument return "\(quotedKey(cell.field)): \(try documentValueJson(cell.value, field: cell.field, provenance: provenance))" } return "\(collectionAccessor).insertOne({\(entries.joined(separator: ", "))})" } - private func isLeftOut(_ value: PluginCellValue) -> Bool { - value.isNull || value.asText == Self.defaultMarker + private func isLeftOut(_ value: PluginCellValue, isAbsent: Bool) -> Bool { + isAbsent || value.asText == Self.defaultMarker } /// A value of a whole document the grid writes: a new row, a duplicate or a paste, or a @@ -164,16 +170,21 @@ struct MongoDBStatementGenerator { /// A new row leaves `_id` to the server. Undoing a delete is the opposite requirement: a new /// `_id` is a different document, and anything that referenced the old one still points at /// nothing. A value that cannot be written refuses the restore rather than dropping the field. - func generateRestore(rows: [[PluginCellValue]]) -> [(statement: String, parameters: [PluginCellValue])]? { + /// A field the document did not have stays missing, and one that held null holds it again. + func generateRestore( + rows: [[PluginCellValue]], + absentCells: [Int: Set] = [:] + ) -> [(statement: String, parameters: [PluginCellValue])]? { guard let idIndex = idColumnIndex else { return nil } do { - return try rows.map { row in + return try rows.enumerated().map { rowIndex, row in guard idIndex < row.count else { throw MongoDBWriteRefusal.missingIdentity } + let absent = absentCells[rowIndex] ?? [] let idField = MongoDBCollectionDDL.idField var entries = ["\(quotedKey(idField)): \(try documentValueJson(row[idIndex], field: idField, provenance: .restored))"] for (index, value) in row.enumerated() where index != idIndex && index < columns.count { - guard !isLeftOut(value) else { continue } + guard !isLeftOut(value, isAbsent: absent.contains(index)) else { continue } let field = columns[index] entries.append("\(quotedKey(field)): \(try documentValueJson(value, field: field, provenance: .restored))") } @@ -204,21 +215,32 @@ struct MongoDBStatementGenerator { } } + /// A field the change names in `absentColumns` is removed with `$unset`; NULL is stored as null. private func updateStatement(for change: PluginRowChange) throws -> String? { guard !change.cellChanges.isEmpty else { return nil } let identity = try identityJson(of: change) + let removed = change.absentColumns ?? [] let cellWrites = try change.cellChanges.map { - try cellWrite(field: $0.columnName, from: $0.oldValue, to: $0.newValue) + try cellWrite( + field: $0.columnName, from: $0.oldValue, to: $0.newValue, + removesField: removed.contains($0.columnIndex) + ) } let update = try updateDocument(for: cellWrites) return "\(collectionAccessor).updateOne({\"_id\": \(identity)}, \(update))" } - private func cellWrite(field: String, from oldValue: PluginCellValue, to newValue: PluginCellValue) throws -> CellWrite { + private func cellWrite( + field: String, + from oldValue: PluginCellValue, + to newValue: PluginCellValue, + removesField: Bool + ) throws -> CellWrite { guard field != MongoDBCollectionDDL.idField else { throw MongoDBWriteRefusal.identityChanged } + guard !removesField else { return .remove(field: field) } switch newValue { case .null: - return .remove(field: field) + return .whole(field: field, json: "null") case .bytes(let data): return .whole(field: field, json: try binaryJson(data, field: field, provenance: .edit(replacing: oldValue))) case .text(let text): diff --git a/Plugins/MongoDBDriverPlugin/MongoDBWriteRefusal.swift b/Plugins/MongoDBDriverPlugin/MongoDBWriteRefusal.swift index bda65c8dad..2b5b208355 100644 --- a/Plugins/MongoDBDriverPlugin/MongoDBWriteRefusal.swift +++ b/Plugins/MongoDBDriverPlugin/MongoDBWriteRefusal.swift @@ -46,14 +46,14 @@ enum MongoDBWriteRefusal: Error, Equatable { field ) case .emptyFieldNameInNewDocument: - return String(localized: "The MongoDB client library will not insert a field whose name is empty. Set it to NULL, then set it on the saved document.") + return String(localized: "The MongoDB client library will not insert a field whose name is empty. Remove the field, then set it on the saved document.") case .emptyKeyInNewDocument(let field): return String( format: String(localized: "%@ holds an empty key, which the MongoDB client library will not insert. Set it to NULL, then set it on the saved document."), field ) case .prototypeFieldInNewDocument: - return String(localized: "The shell drops a field named __proto__ from a new document. Set it to NULL, then set it on the saved document.") + return String(localized: "The shell drops a field named __proto__ from a new document. Remove the field, then set it on the saved document.") case .fieldNeedsMongoDB5(let field): return String( format: String(localized: "A field named \u{201C}%@\u{201D} can only be changed on MongoDB 5.0 or later, which can address a dot or a leading $ in a name."), diff --git a/Plugins/MongoDBDriverPlugin/MongoScriptResultBuilder.swift b/Plugins/MongoDBDriverPlugin/MongoScriptResultBuilder.swift index 4a9dedc163..c0934698bb 100644 --- a/Plugins/MongoDBDriverPlugin/MongoScriptResultBuilder.swift +++ b/Plugins/MongoDBDriverPlugin/MongoScriptResultBuilder.swift @@ -75,7 +75,7 @@ enum MongoScriptResultBuilder { private extension PluginQueryResult { func withRowsAffected(_ count: Int) -> PluginQueryResult { guard count != rowsAffected else { return self } - return PluginQueryResult( + var copy = PluginQueryResult( columns: columns, columnTypeNames: columnTypeNames, rows: rows, @@ -84,10 +84,12 @@ private extension PluginQueryResult { isTruncated: isTruncated, statusMessage: statusMessage ) + copy.absentCells = absentCells + return copy } func withStatus(_ message: String) -> PluginQueryResult { - PluginQueryResult( + var copy = PluginQueryResult( columns: columns, columnTypeNames: columnTypeNames, rows: rows, @@ -96,5 +98,7 @@ private extension PluginQueryResult { isTruncated: isTruncated, statusMessage: message ) + copy.absentCells = absentCells + return copy } } diff --git a/Plugins/TableProPluginKit/DriverPlugin.swift b/Plugins/TableProPluginKit/DriverPlugin.swift index bcf3675861..4dd26ac8ff 100644 --- a/Plugins/TableProPluginKit/DriverPlugin.swift +++ b/Plugins/TableProPluginKit/DriverPlugin.swift @@ -77,6 +77,12 @@ public protocol DriverPlugin: TableProPlugin { /// a binary built before these existed does not offer it. static var supportsDocumentEditing: Bool { get } + /// Whether a row can lack a field altogether, apart from holding NULL, and the driver removes + /// one when a change's `absentColumns` names it. Offers Remove Field and a No Field placeholder, + /// and starts a new row with its fields missing. Read from the loaded plugin, so a binary built + /// before this existed offers none of it. + static var supportsFieldRemoval: Bool { get } + /// Whether a schema is something the engine makes from a statement. False on Oracle, where a /// schema is a user, and on the engines whose namespace is the database. static var supportsCreateSchema: Bool { get } @@ -183,6 +189,7 @@ public extension DriverPlugin { static var supportsRenameDatabase: Bool { false } static var supportsRenameSchema: Bool { false } static var supportsDocumentEditing: Bool { false } + static var supportsFieldRemoval: Bool { false } static var supportsCreateSchema: Bool { false } static var supportsSchemaOwner: Bool { false } static var supportsSchemaPrivileges: Bool { false } diff --git a/Plugins/TableProPluginKit/PluginDatabaseDriver.swift b/Plugins/TableProPluginKit/PluginDatabaseDriver.swift index 025fd287a2..e1c4bf9a5c 100644 --- a/Plugins/TableProPluginKit/PluginDatabaseDriver.swift +++ b/Plugins/TableProPluginKit/PluginDatabaseDriver.swift @@ -58,6 +58,11 @@ public struct PluginRowChange: Sendable { public let cellChanges: [(columnIndex: Int, columnName: String, oldValue: PluginCellValue, newValue: PluginCellValue)] public let originalRow: [PluginCellValue]? + /// The columns the row has no field for once the change is applied: a field an update removes, + /// or a field a new row leaves out. Only an engine that declares `supportsFieldRemoval` is + /// shown rows with such cells, so every other driver can ignore it. + public var absentColumns: Set? + public init( rowIndex: Int, type: ChangeType, @@ -323,6 +328,11 @@ public protocol PluginDatabaseDriver: AnyObject, Sendable { /// something close. func generateIdentityPreservingInsert(table: String, schema: String?, columns: [String], primaryKeyColumns: [String], rows: [[PluginCellValue]]) -> [(statement: String, parameters: [PluginCellValue])]? + /// The same restore for an engine that tells a missing field from NULL. `absentCells` names, by + /// row index, the columns that row had no field for, which must stay missing rather than come + /// back as null. The default ignores them and asks the requirement above. + func generateIdentityPreservingInsert(table: String, schema: String?, columns: [String], primaryKeyColumns: [String], rows: [[PluginCellValue]], absentCells: [Int: Set]) -> [(statement: String, parameters: [PluginCellValue])]? + // Database switching (SQL Server USE, ClickHouse database switch, etc.) func switchDatabase(to database: String) async throws @@ -993,6 +1003,11 @@ public extension PluginDatabaseDriver { } } func generateIdentityPreservingInsert(table: String, schema: String?, columns: [String], primaryKeyColumns: [String], rows: [[PluginCellValue]]) -> [(statement: String, parameters: [PluginCellValue])]? { nil } + func generateIdentityPreservingInsert(table: String, schema: String?, columns: [String], primaryKeyColumns: [String], rows: [[PluginCellValue]], absentCells: [Int: Set]) -> [(statement: String, parameters: [PluginCellValue])]? { + generateIdentityPreservingInsert( + table: table, schema: schema, columns: columns, primaryKeyColumns: primaryKeyColumns, rows: rows + ) + } func generateAddColumnSQL(table: String, column: PluginColumnDefinition) -> String? { nil } func generateModifyColumnSQL(table: String, oldColumn: PluginColumnDefinition, newColumn: PluginColumnDefinition) -> String? { nil } diff --git a/Plugins/TableProPluginKit/PluginQueryResult.swift b/Plugins/TableProPluginKit/PluginQueryResult.swift index c660134850..21f0e739c8 100644 --- a/Plugins/TableProPluginKit/PluginQueryResult.swift +++ b/Plugins/TableProPluginKit/PluginQueryResult.swift @@ -11,6 +11,10 @@ public struct PluginQueryResult: Codable, Sendable { public let columnMeta: [PluginColumnInfo]? public let timing: PluginQueryTiming + /// The cells whose row has no field for the column at all, as row index to column indices. A + /// document store sets it, because a missing field and a field holding null both read `.null`. + public var absentCells: [Int: Set]? + public init( columns: [String], columnTypeNames: [String], @@ -94,6 +98,7 @@ public struct PluginQueryResult: Codable, Sendable { columnMeta = try container.decodeIfPresent([PluginColumnInfo].self, forKey: .columnMeta) timing = try container.decodeIfPresent(PluginQueryTiming.self, forKey: .timing) ?? PluginQueryTiming(total: executionTime) + absentCells = try container.decodeIfPresent([Int: Set].self, forKey: .absentCells) } public static let empty = PluginQueryResult( diff --git a/TablePro/Core/ChangeTracking/AnyChangeManager.swift b/TablePro/Core/ChangeTracking/AnyChangeManager.swift index 7154d29a8e..a9575b01a5 100644 --- a/TablePro/Core/ChangeTracking/AnyChangeManager.swift +++ b/TablePro/Core/ChangeTracking/AnyChangeManager.swift @@ -10,6 +10,7 @@ protocol ChangeManaging: AnyObject { var rowChanges: [RowChange] { get } var insertedRowIDs: Set { get } var generatedColumns: Set { get } + var supportsFieldRemoval: Bool { get } func isRowDeleted(_ rowID: RowID) -> Bool func recordCellChange( rowID: RowID, @@ -19,6 +20,15 @@ protocol ChangeManaging: AnyObject { newValue: PluginCellValue, originalRow: [PluginCellValue]? ) + func recordCellChange( + rowID: RowID, + columnIndex: Int, + columnName: String, + oldValue: PluginCellValue, + newValue: PluginCellValue, + originalRow: [PluginCellValue]?, + absence: FieldAbsence + ) func undoRowDeletion(rowID: RowID) } @@ -26,6 +36,28 @@ protocol ChangeManaging: AnyObject { /// inspector grids edit schema definitions, where the concept does not apply. extension ChangeManaging { var generatedColumns: Set { [] } + + var supportsFieldRemoval: Bool { false } + + /// Only the data grid can show a row without a field; every other grid records the value. + func recordCellChange( + rowID: RowID, + columnIndex: Int, + columnName: String, + oldValue: PluginCellValue, + newValue: PluginCellValue, + originalRow: [PluginCellValue]?, + absence: FieldAbsence + ) { + recordCellChange( + rowID: rowID, + columnIndex: columnIndex, + columnName: columnName, + oldValue: oldValue, + newValue: newValue, + originalRow: originalRow + ) + } } @MainActor @@ -38,6 +70,7 @@ final class AnyChangeManager: ObservableObject { var rowChanges: [RowChange] { wrapped.rowChanges } var insertedRowIDs: Set { wrapped.insertedRowIDs } var generatedColumns: Set { wrapped.generatedColumns } + var supportsFieldRemoval: Bool { wrapped.supportsFieldRemoval } func isRowDeleted(_ rowID: RowID) -> Bool { wrapped.isRowDeleted(rowID) @@ -61,6 +94,26 @@ final class AnyChangeManager: ObservableObject { ) } + func recordCellChange( + rowID: RowID, + columnIndex: Int, + columnName: String, + oldValue: PluginCellValue, + newValue: PluginCellValue, + originalRow: [PluginCellValue], + absence: FieldAbsence + ) { + wrapped.recordCellChange( + rowID: rowID, + columnIndex: columnIndex, + columnName: columnName, + oldValue: oldValue, + newValue: newValue, + originalRow: originalRow, + absence: absence + ) + } + func undoRowDeletion(rowID: RowID) { wrapped.undoRowDeletion(rowID: rowID) } diff --git a/TablePro/Core/ChangeTracking/DataChangeManager.swift b/TablePro/Core/ChangeTracking/DataChangeManager.swift index 6326fa938e..ce65505f93 100644 --- a/TablePro/Core/ChangeTracking/DataChangeManager.swift +++ b/TablePro/Core/ChangeTracking/DataChangeManager.swift @@ -17,6 +17,8 @@ struct UndoResult { let needsRowRemoval: Bool let needsRowRestore: Bool let restoreRow: [PluginCellValue]? + /// The columns `restoreRow` has no field for. + let restoreAbsentColumns: Set let delta: Delta init( @@ -24,12 +26,14 @@ struct UndoResult { needsRowRemoval: Bool, needsRowRestore: Bool, restoreRow: [PluginCellValue]?, + restoreAbsentColumns: Set = [], delta: Delta = .none ) { self.action = action self.needsRowRemoval = needsRowRemoval self.needsRowRestore = needsRowRestore self.restoreRow = restoreRow + self.restoreAbsentColumns = restoreAbsentColumns self.delta = delta } } @@ -82,6 +86,11 @@ final class DataChangeManager: ObservableObject, ChangeManaging { let previousValue: PluginCellValue var newValue: PluginCellValue let originalRow: [PluginCellValue]? + var absence: FieldAbsence + + var isNoOp: Bool { + previousValue == newValue && absence.wasAbsent == absence.isAbsent + } } private struct CoalescedCellKey: Hashable { @@ -170,6 +179,13 @@ final class DataChangeManager: ObservableObject, ChangeManaging { self.rowMatchPolicy = rowMatchPolicy } + /// Whether the engine tells a missing field from NULL, which is what offers Remove Field and + /// starts a new row with its fields missing. + var supportsFieldRemoval: Bool { + guard let databaseType else { return false } + return PluginManager.shared.supportsFieldRemoval(for: databaseType) + } + /// Whether the app may send a value for this column at all: the server computes or allocates it, /// or the driver declares it immutable, as MongoDB does for `_id`. Both halves belong here, /// because this is the boundary every staging path crosses and the grid's own copy of the @@ -180,6 +196,13 @@ final class DataChangeManager: ObservableObject, ChangeManaging { return !PluginManager.shared.immutableColumns(for: databaseType).contains(columnName) } + /// The columns among `columns` the app may not send a value for, which the inspector shows + /// read-only. The same answer as `isColumnWritable`, so the inspector cannot offer an edit that + /// staging then refuses and leaves pending in the field. + func unwritableColumns(among columns: [String]) -> Set { + Set(columns.filter { !isColumnWritable($0) }) + } + // MARK: - Change Tracking func recordCellChange( @@ -189,11 +212,27 @@ final class DataChangeManager: ObservableObject, ChangeManaging { oldValue: PluginCellValue, newValue: PluginCellValue, originalRow: [PluginCellValue]? = nil + ) { + recordCellChange( + rowID: rowID, columnIndex: columnIndex, columnName: columnName, + oldValue: oldValue, newValue: newValue, originalRow: originalRow, + absence: FieldAbsence() + ) + } + + func recordCellChange( + rowID: RowID, + columnIndex: Int, + columnName: String, + oldValue: PluginCellValue, + newValue: PluginCellValue, + originalRow: [PluginCellValue]?, + absence: FieldAbsence ) { record( rowID: rowID, columnIndex: columnIndex, columnName: columnName, oldValue: oldValue, newValue: newValue, originalRow: originalRow, - coalescesWithPrevious: false + absence: absence, coalescesWithPrevious: false ) } @@ -205,12 +244,13 @@ final class DataChangeManager: ObservableObject, ChangeManaging { columnName: String, oldValue: PluginCellValue, newValue: PluginCellValue, - originalRow: [PluginCellValue]? = nil + originalRow: [PluginCellValue]? = nil, + absence: FieldAbsence = FieldAbsence() ) { record( rowID: rowID, columnIndex: columnIndex, columnName: columnName, oldValue: oldValue, newValue: newValue, originalRow: originalRow, - coalescesWithPrevious: true + absence: absence, coalescesWithPrevious: true ) } @@ -221,6 +261,7 @@ final class DataChangeManager: ObservableObject, ChangeManaging { oldValue: PluginCellValue, newValue: PluginCellValue, originalRow: [PluginCellValue]?, + absence: FieldAbsence, coalescesWithPrevious: Bool ) { /// The last gate before a change becomes pending, and the only one every path crosses. The @@ -240,7 +281,8 @@ final class DataChangeManager: ObservableObject, ChangeManaging { columnName: columnName, oldValue: oldValue, newValue: newValue, - originalRow: originalRow + originalRow: originalRow, + absence: absence ) guard recorded else { hasChanges = !pending.isEmpty @@ -249,13 +291,14 @@ final class DataChangeManager: ObservableObject, ChangeManaging { if coalescesWithPrevious { bufferCoalescedEdit( rowID: rowID, columnIndex: columnIndex, columnName: columnName, - oldValue: oldValue, newValue: newValue, originalRow: originalRow + oldValue: oldValue, newValue: newValue, originalRow: originalRow, absence: absence ) } else { registerUndo(actionName: String(localized: "Edit Cell")) { target in target.applyDataUndo(.cellEdit( rowID: rowID, columnIndex: columnIndex, columnName: columnName, - previousValue: oldValue, newValue: newValue, originalRow: originalRow + previousValue: oldValue, newValue: newValue, originalRow: originalRow, + absence: absence )) } } @@ -268,17 +311,19 @@ final class DataChangeManager: ObservableObject, ChangeManaging { columnName: String, oldValue: PluginCellValue, newValue: PluginCellValue, - originalRow: [PluginCellValue]? + originalRow: [PluginCellValue]?, + absence: FieldAbsence ) { let key = CoalescedCellKey(rowID: rowID, columnIndex: columnIndex) if var existing = coalescedEdits[key] { existing.newValue = newValue + existing.absence.isAbsent = absence.isAbsent coalescedEdits[key] = existing return } coalescedEdits[key] = CoalescedCellEdit( rowID: rowID, columnIndex: columnIndex, columnName: columnName, - previousValue: oldValue, newValue: newValue, originalRow: originalRow + previousValue: oldValue, newValue: newValue, originalRow: originalRow, absence: absence ) coalescedOrder.append(key) } @@ -290,7 +335,7 @@ final class DataChangeManager: ObservableObject, ChangeManaging { /// over a value that already matches the server. func endCoalescedUndoRun() { guard !coalescedOrder.isEmpty else { return } - let edits = coalescedOrder.compactMap { coalescedEdits[$0] }.filter { $0.previousValue != $0.newValue } + let edits = coalescedOrder.compactMap { coalescedEdits[$0] }.filter { !$0.isNoOp } discardCoalescedUndoRun() guard !edits.isEmpty, let undoManager = undoManagerProvider?() else { return } @@ -299,7 +344,8 @@ final class DataChangeManager: ObservableObject, ChangeManaging { undoManager.registerUndo(withTarget: self) { target in target.applyDataUndo(.cellEdit( rowID: edit.rowID, columnIndex: edit.columnIndex, columnName: edit.columnName, - previousValue: edit.previousValue, newValue: edit.newValue, originalRow: edit.originalRow + previousValue: edit.previousValue, newValue: edit.newValue, originalRow: edit.originalRow, + absence: edit.absence )) } } @@ -312,33 +358,38 @@ final class DataChangeManager: ObservableObject, ChangeManaging { coalescedOrder.removeAll() } - func recordRowDeletion(rowID: RowID, originalRow: [PluginCellValue]) { - pending.recordRowDeletion(rowID: rowID, originalRow: originalRow) + func recordRowDeletion(rowID: RowID, originalRow: [PluginCellValue], absentColumns: Set = []) { + pending.recordRowDeletion(rowID: rowID, originalRow: originalRow, absentColumns: absentColumns) registerUndo(actionName: String(localized: "Delete Row")) { target in - target.applyDataUndo(.rowDeletion(rowID: rowID, originalRow: originalRow)) + target.applyDataUndo(.rowDeletion(rowID: rowID, originalRow: originalRow, absentColumns: absentColumns)) } hasChanges = true } - func recordBatchRowDeletion(rows: [(rowID: RowID, originalRow: [PluginCellValue])]) { + func recordBatchRowDeletion( + rows: [(rowID: RowID, originalRow: [PluginCellValue])], + absentColumns: [RowID: Set] = [:] + ) { guard rows.count > 1 else { if let row = rows.first { - recordRowDeletion(rowID: row.rowID, originalRow: row.originalRow) + recordRowDeletion( + rowID: row.rowID, originalRow: row.originalRow, absentColumns: absentColumns[row.rowID] ?? [] + ) } return } for (rowID, originalRow) in rows { - pending.recordRowDeletion(rowID: rowID, originalRow: originalRow) + pending.recordRowDeletion(rowID: rowID, originalRow: originalRow, absentColumns: absentColumns[rowID] ?? []) } let batchData = rows registerUndo(actionName: String(localized: "Delete Rows")) { target in - target.applyDataUndo(.batchRowDeletion(rows: batchData)) + target.applyDataUndo(.batchRowDeletion(rows: batchData, absentColumns: absentColumns)) } hasChanges = true } - func recordRowInsertion(rowID: RowID, values: [PluginCellValue]) { - pending.recordRowInsertion(rowID: rowID, values: values) + func recordRowInsertion(rowID: RowID, values: [PluginCellValue], absentColumns: Set = []) { + pending.recordRowInsertion(rowID: rowID, values: values, absentColumns: absentColumns) registerUndo(actionName: String(localized: "Insert Row")) { target in target.applyDataUndo(.rowInsertion(rowID: rowID)) } @@ -355,11 +406,14 @@ final class DataChangeManager: ObservableObject, ChangeManaging { func undoBatchRowInsertion(rows: [InsertedRowLocation]) { let validRows = rows.filter { pending.isRowInserted($0.rowID) } guard !validRows.isEmpty else { return } + let rowAbsentColumns = validRows.map { pending.insertedAbsentColumns(forRow: $0.rowID) } let rowValues = pending.undoBatchRowInsertion( rowIDs: validRows.map(\.rowID), columnCount: columns.count ) registerUndo(actionName: String(localized: "Insert Rows")) { target in - target.applyDataUndo(.batchRowInsertion(rows: validRows, rowValues: rowValues)) + target.applyDataUndo(.batchRowInsertion( + rows: validRows, rowValues: rowValues, rowAbsentColumns: rowAbsentColumns + )) } hasChanges = !pending.isEmpty } @@ -368,24 +422,27 @@ final class DataChangeManager: ObservableObject, ChangeManaging { private func applyDataUndo(_ action: UndoAction) { switch action { - case .cellEdit(let rowID, let columnIndex, let columnName, let previousValue, let newValue, let originalRow): + case .cellEdit(let rowID, let columnIndex, let columnName, let previousValue, let newValue, let originalRow, + let absence): applyCellEditUndo( rowID: rowID, columnIndex: columnIndex, columnName: columnName, previousValue: previousValue, newValue: newValue, originalRow: originalRow, - action: action + absence: absence, action: action ) - case .rowInsertion(let rowID): - applyRowInsertionUndo(rowID: rowID, action: action) + case .rowInsertion(let rowID, let image): + applyRowInsertionUndo(rowID: rowID, restoring: image, action: action) - case .rowDeletion(let rowID, let originalRow): - applyRowDeletionUndo(rowID: rowID, originalRow: originalRow, action: action) + case .rowDeletion(let rowID, let originalRow, let absentColumns): + applyRowDeletionUndo(rowID: rowID, originalRow: originalRow, absentColumns: absentColumns, action: action) - case .batchRowDeletion(let rows): - applyBatchRowDeletionUndo(rows: rows, action: action) + case .batchRowDeletion(let rows, let absentColumns): + applyBatchRowDeletionUndo(rows: rows, absentColumns: absentColumns, action: action) - case .batchRowInsertion(let rows, let rowValues): - applyBatchRowInsertionUndo(rows: rows, rowValues: rowValues, action: action) + case .batchRowInsertion(let rows, let rowValues, let rowAbsentColumns): + applyBatchRowInsertionUndo( + rows: rows, rowValues: rowValues, rowAbsentColumns: rowAbsentColumns, action: action + ) } hasChanges = !pending.isEmpty @@ -398,12 +455,13 @@ final class DataChangeManager: ObservableObject, ChangeManaging { private func applyCellEditUndo( rowID: RowID, columnIndex: Int, columnName: String, previousValue: PluginCellValue, newValue: PluginCellValue, originalRow: [PluginCellValue]?, - action: UndoAction + absence: FieldAbsence, action: UndoAction ) { registerUndo(actionName: String(localized: "Edit Cell")) { target in target.applyDataUndo(.cellEdit( rowID: rowID, columnIndex: columnIndex, columnName: columnName, - previousValue: newValue, newValue: previousValue, originalRow: originalRow + previousValue: newValue, newValue: previousValue, originalRow: originalRow, + absence: absence.reversed )) } @@ -411,19 +469,21 @@ final class DataChangeManager: ObservableObject, ChangeManaging { if updateChange.cellChanges.contains(where: { $0.columnIndex == columnIndex }) { pending.revertUpdateCell( rowID: rowID, columnIndex: columnIndex, - columnName: columnName, previousValue: previousValue + columnName: columnName, previousValue: previousValue, + previousIsAbsent: absence.wasAbsent ) } } else if pending.change(forRow: rowID, type: .insert) != nil { pending.updateInsertedCellDirectly( rowID: rowID, columnIndex: columnIndex, - columnName: columnName, newValue: previousValue + columnName: columnName, newValue: previousValue, isAbsent: absence.wasAbsent ) } else { pending.reapplyCellChange( rowID: rowID, columnIndex: columnIndex, columnName: columnName, - originalDBValue: newValue, newValue: previousValue, originalRow: originalRow + originalDBValue: newValue, newValue: previousValue, originalRow: originalRow, + absence: absence.reversed ) } lastUndoResult = UndoResult( @@ -431,31 +491,42 @@ final class DataChangeManager: ObservableObject, ChangeManaging { ) } - private func applyRowInsertionUndo(rowID: RowID, action: UndoAction) { - let savedValues = pending.savedInsertedValues(forRow: rowID) - registerUndo(actionName: String(localized: "Insert Row")) { [savedValues] target in - if let savedValues { - target.pending.restoreInsertedValues(forRow: rowID, values: savedValues) + /// Undoing takes the row out and hands the redo everything it held. Redoing puts that back, + /// and the undo registered then reads the row as it stands again. + private func applyRowInsertionUndo(rowID: RowID, restoring image: InsertedRowImage?, action: UndoAction) { + guard pending.isRowInserted(rowID) else { + registerUndo(actionName: String(localized: "Insert Row")) { target in + target.applyDataUndo(.rowInsertion(rowID: rowID)) } - target.applyDataUndo(.rowInsertion(rowID: rowID)) - } - - if pending.isRowInserted(rowID) { - _ = pending.undoRowInsertion(rowID: rowID) - lastUndoResult = UndoResult( - action: action, needsRowRemoval: true, needsRowRestore: false, restoreRow: nil + let absentColumns = image?.absentColumns ?? [] + pending.reinsertRow( + rowID: rowID, columns: columns, savedValues: image?.values, absentColumns: absentColumns ) - } else { - pending.reinsertRow(rowID: rowID, columns: columns, savedValues: savedValues) lastUndoResult = UndoResult( - action: action, needsRowRemoval: false, needsRowRestore: true, restoreRow: savedValues + action: action, needsRowRemoval: false, needsRowRestore: true, restoreRow: image?.values, + restoreAbsentColumns: absentColumns ) + return } + + let removed = InsertedRowImage( + values: pending.savedInsertedValues(forRow: rowID), + absentColumns: pending.insertedAbsentColumns(forRow: rowID) + ) + registerUndo(actionName: String(localized: "Insert Row")) { target in + target.applyDataUndo(.rowInsertion(rowID: rowID, restoring: removed)) + } + _ = pending.undoRowInsertion(rowID: rowID) + lastUndoResult = UndoResult( + action: action, needsRowRemoval: true, needsRowRestore: false, restoreRow: nil + ) } - private func applyRowDeletionUndo(rowID: RowID, originalRow: [PluginCellValue], action: UndoAction) { + private func applyRowDeletionUndo( + rowID: RowID, originalRow: [PluginCellValue], absentColumns: Set, action: UndoAction + ) { registerUndo(actionName: String(localized: "Delete Row")) { target in - target.applyDataUndo(.rowDeletion(rowID: rowID, originalRow: originalRow)) + target.applyDataUndo(.rowDeletion(rowID: rowID, originalRow: originalRow, absentColumns: absentColumns)) } if pending.isRowDeleted(rowID) { @@ -465,7 +536,7 @@ final class DataChangeManager: ObservableObject, ChangeManaging { delta: .fullReplace ) } else { - pending.reapplyRowDeletion(rowID: rowID, originalRow: originalRow) + pending.reapplyRowDeletion(rowID: rowID, originalRow: originalRow, absentColumns: absentColumns) lastUndoResult = UndoResult( action: action, needsRowRemoval: true, needsRowRestore: false, restoreRow: nil, delta: .fullReplace @@ -474,10 +545,12 @@ final class DataChangeManager: ObservableObject, ChangeManaging { } private func applyBatchRowDeletionUndo( - rows: [(rowID: RowID, originalRow: [PluginCellValue])], action: UndoAction + rows: [(rowID: RowID, originalRow: [PluginCellValue])], + absentColumns: [RowID: Set], + action: UndoAction ) { registerUndo(actionName: String(localized: "Delete Rows")) { target in - target.applyDataUndo(.batchRowDeletion(rows: rows)) + target.applyDataUndo(.batchRowDeletion(rows: rows, absentColumns: absentColumns)) } let isUndo = rows.contains { pending.isRowDeleted($0.rowID) } @@ -491,7 +564,9 @@ final class DataChangeManager: ObservableObject, ChangeManaging { ) } else { for (rowID, originalRow) in rows { - pending.reapplyRowDeletion(rowID: rowID, originalRow: originalRow) + pending.reapplyRowDeletion( + rowID: rowID, originalRow: originalRow, absentColumns: absentColumns[rowID] ?? [] + ) } lastUndoResult = UndoResult( action: action, needsRowRemoval: true, needsRowRestore: false, restoreRow: nil, @@ -501,10 +576,12 @@ final class DataChangeManager: ObservableObject, ChangeManaging { } private func applyBatchRowInsertionUndo( - rows: [InsertedRowLocation], rowValues: [[PluginCellValue]], action: UndoAction + rows: [InsertedRowLocation], rowValues: [[PluginCellValue]], rowAbsentColumns: [Set], action: UndoAction ) { registerUndo(actionName: String(localized: "Insert Rows")) { target in - target.applyDataUndo(.batchRowInsertion(rows: rows, rowValues: rowValues)) + target.applyDataUndo(.batchRowInsertion( + rows: rows, rowValues: rowValues, rowAbsentColumns: rowAbsentColumns + )) } let rowIDs = rows.map(\.rowID) @@ -515,7 +592,9 @@ final class DataChangeManager: ObservableObject, ChangeManaging { action: action, needsRowRemoval: true, needsRowRestore: false, restoreRow: nil ) } else { - pending.reinsertBatch(rowIDs: rowIDs, rowValues: rowValues, columns: columns) + pending.reinsertBatch( + rowIDs: rowIDs, rowValues: rowValues, rowAbsentColumns: rowAbsentColumns, columns: columns + ) lastUndoResult = UndoResult( action: action, needsRowRemoval: false, needsRowRestore: true, restoreRow: nil ) @@ -609,14 +688,15 @@ final class DataChangeManager: ObservableObject, ChangeManaging { // MARK: - Actions - func getOriginalValues() -> [(rowID: RowID, columnIndex: Int, value: PluginCellValue)] { - var originals: [(rowID: RowID, columnIndex: Int, value: PluginCellValue)] = [] + func getOriginalValues() -> [(rowID: RowID, columnIndex: Int, value: PluginCellValue, isAbsent: Bool)] { + var originals: [(rowID: RowID, columnIndex: Int, value: PluginCellValue, isAbsent: Bool)] = [] for change in pending.changes where change.type == .update { for cellChange in change.cellChanges { originals.append(( rowID: change.rowID, columnIndex: cellChange.columnIndex, - value: cellChange.oldValue + value: cellChange.oldValue, + isAbsent: cellChange.oldIsAbsent )) } } diff --git a/TablePro/Core/ChangeTracking/DataChangeModels.swift b/TablePro/Core/ChangeTracking/DataChangeModels.swift index 1ef76b0850..9416a3bf4c 100644 --- a/TablePro/Core/ChangeTracking/DataChangeModels.swift +++ b/TablePro/Core/ChangeTracking/DataChangeModels.swift @@ -12,24 +12,53 @@ enum ChangeType: Hashable { case delete } +/// Which side of a cell edit has no field at all, for an engine that tells a missing field from +/// NULL. A missing field reads `.null`, so the value alone cannot say. Every other engine keeps the +/// default, where the field is there on both sides. +struct FieldAbsence: Equatable, Sendable { + /// The cell had no field before the edit. + var wasAbsent = false + /// The edit takes the field out of the row. + var isAbsent = false + /// The columns the row had no field for before any of its edits, kept with its first change so + /// a rewind can leave them missing. + var originalRow: Set = [] + + var reversed: FieldAbsence { + FieldAbsence(wasAbsent: isAbsent, isAbsent: wasAbsent, originalRow: originalRow) + } +} + struct CellChange: Identifiable, Equatable { let id: UUID let columnIndex: Int let columnName: String let oldValue: PluginCellValue let newValue: PluginCellValue + let oldIsAbsent: Bool + let newIsAbsent: Bool init( columnIndex: Int, columnName: String, oldValue: PluginCellValue, - newValue: PluginCellValue + newValue: PluginCellValue, + oldIsAbsent: Bool = false, + newIsAbsent: Bool = false ) { self.id = UUID() self.columnIndex = columnIndex self.columnName = columnName self.oldValue = oldValue self.newValue = newValue + self.oldIsAbsent = oldIsAbsent + self.newIsAbsent = newIsAbsent + } + + /// Whether the edit leaves the cell as it was read: the same value, and the field there or + /// missing on both sides. + var restoresOriginal: Bool { + oldValue == newValue && oldIsAbsent == newIsAbsent } } @@ -40,6 +69,10 @@ struct RowChange: Identifiable, Equatable { var cellChanges: [CellChange] let originalRow: [PluginCellValue]? + /// The columns the row has no field for: as it was read for an update or a delete, and as it + /// stands for an insert. Empty on every engine that cannot tell a missing field from NULL. + var absentColumns: Set + /// The order the user made this change in. /// /// Not the array position. `PendingChanges` removes a cancelled change by swapping the last @@ -53,6 +86,7 @@ struct RowChange: Identifiable, Equatable { type: ChangeType, cellChanges: [CellChange] = [], originalRow: [PluginCellValue]? = nil, + absentColumns: Set = [], sequence: Int = 0 ) { self.id = UUID() @@ -60,8 +94,21 @@ struct RowChange: Identifiable, Equatable { self.type = type self.cellChanges = cellChanges self.originalRow = originalRow + self.absentColumns = absentColumns self.sequence = sequence } + + /// The columns the row has no field for once this update is written: those it lacked, less the + /// ones an edit gave a value, plus the ones an edit removed. + var absentColumnsAfterUpdate: Set { + cellChanges.reduce(into: absentColumns) { absent, cellChange in + if cellChange.newIsAbsent { + absent.insert(cellChange.columnIndex) + } else { + absent.remove(cellChange.columnIndex) + } + } + } } struct RowChangeKey: Hashable { @@ -74,6 +121,13 @@ struct InsertedRowLocation { let storageIndex: Int } +/// What a new row held when its insertion was undone: its values, and the fields it had none of. +/// Redoing the insertion puts back both, since the row's change is gone by then and cannot say. +struct InsertedRowImage { + let values: [PluginCellValue]? + let absentColumns: Set +} + enum UndoAction { case cellEdit( rowID: RowID, @@ -81,10 +135,18 @@ enum UndoAction { columnName: String, previousValue: PluginCellValue, newValue: PluginCellValue, - originalRow: [PluginCellValue]? + originalRow: [PluginCellValue]?, + absence: FieldAbsence = FieldAbsence() + ) + case rowInsertion(rowID: RowID, restoring: InsertedRowImage? = nil) + case rowDeletion(rowID: RowID, originalRow: [PluginCellValue], absentColumns: Set = []) + case batchRowDeletion( + rows: [(rowID: RowID, originalRow: [PluginCellValue])], + absentColumns: [RowID: Set] = [:] + ) + case batchRowInsertion( + rows: [InsertedRowLocation], + rowValues: [[PluginCellValue]], + rowAbsentColumns: [Set] = [] ) - case rowInsertion(rowID: RowID) - case rowDeletion(rowID: RowID, originalRow: [PluginCellValue]) - case batchRowDeletion(rows: [(rowID: RowID, originalRow: [PluginCellValue])]) - case batchRowInsertion(rows: [InsertedRowLocation], rowValues: [[PluginCellValue]]) } diff --git a/TablePro/Core/ChangeTracking/PendingChanges.swift b/TablePro/Core/ChangeTracking/PendingChanges.swift index 28dd97c47f..7d6e999e66 100644 --- a/TablePro/Core/ChangeTracking/PendingChanges.swift +++ b/TablePro/Core/ChangeTracking/PendingChanges.swift @@ -56,6 +56,9 @@ struct PendingChanges: Equatable { /// Whether the recorded edit is a no-op (oldValue == newValue with no prior modification). /// Returns the result so the caller can decide whether to register undo. + /// + /// A field going missing, or coming back, is a change even when the value reads `.null` on + /// both sides. @discardableResult mutating func recordCellChange( rowID: RowID, @@ -63,11 +66,12 @@ struct PendingChanges: Equatable { columnName: String, oldValue: PluginCellValue, newValue: PluginCellValue, - originalRow: [PluginCellValue]? = nil + originalRow: [PluginCellValue]? = nil, + absence: FieldAbsence = FieldAbsence() ) -> Bool { - if oldValue == newValue { + if oldValue == newValue && absence.wasAbsent == absence.isAbsent { return rollbackCellIfMatchesOriginal( - rowID: rowID, columnIndex: columnIndex, restoredValue: newValue + rowID: rowID, columnIndex: columnIndex, restoredValue: newValue, restoredIsAbsent: absence.isAbsent ) } @@ -75,12 +79,14 @@ struct PendingChanges: Equatable { columnIndex: columnIndex, columnName: columnName, oldValue: oldValue, - newValue: newValue + newValue: newValue, + oldIsAbsent: absence.wasAbsent, + newIsAbsent: absence.isAbsent ) if let insertIdx = changeIndex[RowChangeKey(rowID: rowID, type: .insert)] { updateInsertedCell(at: insertIdx, columnIndex: columnIndex, - columnName: columnName, newValue: newValue) + columnName: columnName, newValue: newValue, isAbsent: absence.isAbsent) return true } @@ -90,7 +96,8 @@ struct PendingChanges: Equatable { } else { let row = RowChange( rowID: rowID, type: .update, - cellChanges: [cellChange], originalRow: originalRow + cellChanges: [cellChange], originalRow: originalRow, + absentColumns: absence.originalRow ) changes.append(row) changeIndex[updateKey] = changes.count - 1 @@ -99,21 +106,24 @@ struct PendingChanges: Equatable { return true } - mutating func recordRowDeletion(rowID: RowID, originalRow: [PluginCellValue]) { + mutating func recordRowDeletion(rowID: RowID, originalRow: [PluginCellValue], absentColumns: Set = []) { guard !deletedRowIDs.contains(rowID) else { return } removeChange(rowID: rowID, type: .update) modifiedCells.removeValue(forKey: rowID) - appendChange(RowChange(rowID: rowID, type: .delete, originalRow: originalRow)) + appendChange(RowChange(rowID: rowID, type: .delete, originalRow: originalRow, absentColumns: absentColumns)) deletedRowIDs.insert(rowID) } - mutating func recordRowInsertion(rowID: RowID, values: [PluginCellValue]) { + mutating func recordRowInsertion(rowID: RowID, values: [PluginCellValue], absentColumns: Set = []) { guard !insertedRowIDs.contains(rowID) else { insertedRowData[rowID] = values + if let insertIdx = changeIndex[RowChangeKey(rowID: rowID, type: .insert)] { + changes[insertIdx].absentColumns = absentColumns + } return } insertedRowData[rowID] = values - appendChange(RowChange(rowID: rowID, type: .insert, cellChanges: [])) + appendChange(RowChange(rowID: rowID, type: .insert, cellChanges: [], absentColumns: absentColumns)) insertedRowIDs.insert(rowID) } @@ -154,8 +164,8 @@ struct PendingChanges: Equatable { // MARK: - Replay (driven by NSUndoManager invocation) /// Re-apply a deletion during undo replay (skips undo registration). - mutating func reapplyRowDeletion(rowID: RowID, originalRow: [PluginCellValue]) { - recordRowDeletion(rowID: rowID, originalRow: originalRow) + mutating func reapplyRowDeletion(rowID: RowID, originalRow: [PluginCellValue], absentColumns: Set = []) { + recordRowDeletion(rowID: rowID, originalRow: originalRow, absentColumns: absentColumns) } /// Re-apply a cell edit during undo replay (skips undo registration). @@ -167,18 +177,21 @@ struct PendingChanges: Equatable { columnName: String, originalDBValue: PluginCellValue, newValue: PluginCellValue, - originalRow: [PluginCellValue]? + originalRow: [PluginCellValue]?, + absence: FieldAbsence = FieldAbsence() ) { let cellChange = CellChange( columnIndex: columnIndex, columnName: columnName, oldValue: originalDBValue, - newValue: newValue + newValue: newValue, + oldIsAbsent: absence.wasAbsent, + newIsAbsent: absence.isAbsent ) if let insertIdx = changeIndex[RowChangeKey(rowID: rowID, type: .insert)] { updateInsertedCell(at: insertIdx, columnIndex: columnIndex, - columnName: columnName, newValue: newValue) + columnName: columnName, newValue: newValue, isAbsent: absence.isAbsent) return } @@ -188,7 +201,8 @@ struct PendingChanges: Equatable { } else { let row = RowChange( rowID: rowID, type: .update, - cellChanges: [cellChange], originalRow: originalRow + cellChanges: [cellChange], originalRow: originalRow, + absentColumns: absence.originalRow ) changes.append(row) changeIndex[updateKey] = changes.count - 1 @@ -201,10 +215,13 @@ struct PendingChanges: Equatable { rowID: RowID, columnIndex: Int, columnName: String, - newValue: PluginCellValue + newValue: PluginCellValue, + isAbsent: Bool = false ) { guard let insertIdx = changeIndex[RowChangeKey(rowID: rowID, type: .insert)] else { return } - updateInsertedCell(at: insertIdx, columnIndex: columnIndex, columnName: columnName, newValue: newValue) + updateInsertedCell( + at: insertIdx, columnIndex: columnIndex, columnName: columnName, newValue: newValue, isAbsent: isAbsent + ) } /// Restore a cell's value during undo replay when an existing change matches. @@ -212,14 +229,15 @@ struct PendingChanges: Equatable { rowID: RowID, columnIndex: Int, columnName: String, - previousValue: PluginCellValue + previousValue: PluginCellValue, + previousIsAbsent: Bool = false ) { guard let updateIdx = changeIndex[RowChangeKey(rowID: rowID, type: .update)], let cellIdx = changes[updateIdx].cellChanges.firstIndex(where: { $0.columnIndex == columnIndex }) else { return } - let originalOldValue = changes[updateIdx].cellChanges[cellIdx].oldValue - if previousValue == originalOldValue { + let original = changes[updateIdx].cellChanges[cellIdx] + if previousValue == original.oldValue && previousIsAbsent == original.oldIsAbsent { changes[updateIdx].cellChanges.remove(at: cellIdx) removeModifiedCell(rowID: rowID, columnIndex: columnIndex) if changes[updateIdx].cellChanges.isEmpty { @@ -229,22 +247,30 @@ struct PendingChanges: Equatable { changes[updateIdx].cellChanges[cellIdx] = CellChange( columnIndex: columnIndex, columnName: columnName, - oldValue: originalOldValue, - newValue: previousValue + oldValue: original.oldValue, + newValue: previousValue, + oldIsAbsent: original.oldIsAbsent, + newIsAbsent: previousIsAbsent ) } } /// Insert a synthetic .insert RowChange for undo replay (e.g., after redoing a deletion's undo). - mutating func reinsertRow(rowID: RowID, columns: [String], savedValues: [PluginCellValue]?) { + mutating func reinsertRow( + rowID: RowID, + columns: [String], + savedValues: [PluginCellValue]?, + absentColumns: Set = [] + ) { insertedRowIDs.insert(rowID) let cellChanges = columns.enumerated().map { index, columnName in CellChange( columnIndex: index, columnName: columnName, - oldValue: nil, newValue: savedValues?[safe: index] ?? nil + oldValue: nil, newValue: savedValues?[safe: index] ?? nil, + newIsAbsent: absentColumns.contains(index) ) } - appendChange(RowChange(rowID: rowID, type: .insert, cellChanges: cellChanges)) + appendChange(RowChange(rowID: rowID, type: .insert, cellChanges: cellChanges, absentColumns: absentColumns)) if let savedValues { insertedRowData[rowID] = savedValues } @@ -252,17 +278,22 @@ struct PendingChanges: Equatable { /// Insert a batch of rows (for undo replay of a batch deletion's undo). mutating func reinsertBatch( - rowIDs: [RowID], rowValues: [[PluginCellValue]], columns: [String] + rowIDs: [RowID], + rowValues: [[PluginCellValue]], + rowAbsentColumns: [Set] = [], + columns: [String] ) { - for (rowID, values) in zip(rowIDs, rowValues) { + for (offset, (rowID, values)) in zip(rowIDs, rowValues).enumerated() { + let absentColumns = rowAbsentColumns[safe: offset] ?? [] let cellChanges = values.enumerated().map { colIndex, value in CellChange( columnIndex: colIndex, columnName: columns[safe: colIndex] ?? "", - oldValue: nil, newValue: value + oldValue: nil, newValue: value, + newIsAbsent: absentColumns.contains(colIndex) ) } - appendChange(RowChange(rowID: rowID, type: .insert, cellChanges: cellChanges)) + appendChange(RowChange(rowID: rowID, type: .insert, cellChanges: cellChanges, absentColumns: absentColumns)) insertedRowIDs.insert(rowID) insertedRowData[rowID] = values } @@ -273,9 +304,9 @@ struct PendingChanges: Equatable { insertedRowData[rowID] } - /// Restore inserted-row values when undo restores a row. - mutating func restoreInsertedValues(forRow rowID: RowID, values: [PluginCellValue]) { - insertedRowData[rowID] = values + /// The columns an inserted row has no field for, saved beside its values for the same replay. + func insertedAbsentColumns(forRow rowID: RowID) -> Set { + change(forRow: rowID, type: .insert)?.absentColumns ?? [] } // MARK: - Reset / persistence @@ -358,17 +389,22 @@ struct PendingChanges: Equatable { } private mutating func updateInsertedCell( - at insertIdx: Int, columnIndex: Int, columnName: String, newValue: PluginCellValue + at insertIdx: Int, columnIndex: Int, columnName: String, newValue: PluginCellValue, isAbsent: Bool ) { let rowID = changes[insertIdx].rowID if var stored = insertedRowData[rowID], columnIndex < stored.count { stored[columnIndex] = newValue insertedRowData[rowID] = stored } + if isAbsent { + changes[insertIdx].absentColumns.insert(columnIndex) + } else { + changes[insertIdx].absentColumns.remove(columnIndex) + } let replacement = CellChange( columnIndex: columnIndex, columnName: columnName, - oldValue: nil, newValue: newValue + oldValue: nil, newValue: newValue, newIsAbsent: isAbsent ) if let cellIdx = changes[insertIdx].cellChanges.firstIndex(where: { $0.columnIndex == columnIndex }) { changes[insertIdx].cellChanges[cellIdx] = replacement @@ -387,15 +423,18 @@ struct PendingChanges: Equatable { return } - let originalOldValue = changes[updateIdx].cellChanges[cellIdx].oldValue - changes[updateIdx].cellChanges[cellIdx] = CellChange( + let original = changes[updateIdx].cellChanges[cellIdx] + let merged = CellChange( columnIndex: cellChange.columnIndex, columnName: cellChange.columnName, - oldValue: originalOldValue, - newValue: cellChange.newValue + oldValue: original.oldValue, + newValue: cellChange.newValue, + oldIsAbsent: original.oldIsAbsent, + newIsAbsent: cellChange.newIsAbsent ) + changes[updateIdx].cellChanges[cellIdx] = merged - guard originalOldValue == cellChange.newValue else { return } + guard merged.restoresOriginal else { return } changes[updateIdx].cellChanges.remove(at: cellIdx) removeModifiedCell(rowID: rowID, columnIndex: cellChange.columnIndex) if changes[updateIdx].cellChanges.isEmpty { @@ -405,12 +444,13 @@ struct PendingChanges: Equatable { @discardableResult private mutating func rollbackCellIfMatchesOriginal( - rowID: RowID, columnIndex: Int, restoredValue: PluginCellValue + rowID: RowID, columnIndex: Int, restoredValue: PluginCellValue, restoredIsAbsent: Bool ) -> Bool { let updateKey = RowChangeKey(rowID: rowID, type: .update) guard let updateIdx = changeIndex[updateKey], let cellIdx = changes[updateIdx].cellChanges.firstIndex(where: { $0.columnIndex == columnIndex }), - changes[updateIdx].cellChanges[cellIdx].oldValue == restoredValue else { + changes[updateIdx].cellChanges[cellIdx].oldValue == restoredValue, + changes[updateIdx].cellChanges[cellIdx].oldIsAbsent == restoredIsAbsent else { return false } changes[updateIdx].cellChanges.remove(at: cellIdx) diff --git a/TablePro/Core/Coordinators/PaginationCoordinator.swift b/TablePro/Core/Coordinators/PaginationCoordinator.swift index a006ddb9f6..826d54c73d 100644 --- a/TablePro/Core/Coordinators/PaginationCoordinator.swift +++ b/TablePro/Core/Coordinators/PaginationCoordinator.swift @@ -379,7 +379,7 @@ final class PaginationCoordinator: ObservableObject { guard let idx = parent.tabManager.tabs.firstIndex(where: { $0.id == tabId }) else { return } let replaceDelta = parent.mutateActiveTableRows(for: tabId) { rows in - rows.replace(rows: result.rows) + rows.replace(rows: result.rows, absentCells: result.absentCells) } parent.tabManager.mutate(at: idx) { tab in tab.execution.executionTime = result.executionTime diff --git a/TablePro/Core/Coordinators/QueryExecutionCoordinator+Batches.swift b/TablePro/Core/Coordinators/QueryExecutionCoordinator+Batches.swift index 5532a57c47..fe4ba8cddb 100644 --- a/TablePro/Core/Coordinators/QueryExecutionCoordinator+Batches.swift +++ b/TablePro/Core/Coordinators/QueryExecutionCoordinator+Batches.swift @@ -538,7 +538,8 @@ extension QueryExecutionCoordinator { tableRows: TableRows.from( queryRows: result.rows, columns: result.columns.map { String($0) }, - columnTypes: result.columnTypes + columnTypes: result.columnTypes, + absentCells: result.absentCells ) ) resultSet.statementAnchor = anchor diff --git a/TablePro/Core/Coordinators/QueryExecutionCoordinator+Helpers.swift b/TablePro/Core/Coordinators/QueryExecutionCoordinator+Helpers.swift index 4223b0dc42..d2f67e2551 100644 --- a/TablePro/Core/Coordinators/QueryExecutionCoordinator+Helpers.swift +++ b/TablePro/Core/Coordinators/QueryExecutionCoordinator+Helpers.swift @@ -212,7 +212,8 @@ extension QueryExecutionCoordinator { anchor: StatementAnchor? = nil, timing: PluginQueryTiming? = nil, viewport: GridReloadIntent = .firstRow, - serverOutput: PluginServerOutput = .none + serverOutput: PluginServerOutput = .none, + absentCells: [Int: Set] = [:] ) { guard let idx = parent.tabManager.tabs.firstIndex(where: { $0.id == tabId }) else { return } @@ -261,7 +262,8 @@ extension QueryExecutionCoordinator { generatedColumns: generatedColumns, rowMatchPolicy: resolved.rowMatchPolicy, hasAuthoritativeSchema: resolved.hasAuthoritativeSchema, - foreignKeysFetched: resolved.foreignKeysFetched + foreignKeysFetched: resolved.foreignKeysFetched, + absentCells: absentCells ) let previousTableName = parent.tabManager.tabs[idx].tableContext.tableName let definitionChanged = parent.tabSessionRegistry.needsDefinition(existingTabId) diff --git a/TablePro/Core/Coordinators/QueryExecutionCoordinator+MultiStatement.swift b/TablePro/Core/Coordinators/QueryExecutionCoordinator+MultiStatement.swift index 02f909b149..3697f468ed 100644 --- a/TablePro/Core/Coordinators/QueryExecutionCoordinator+MultiStatement.swift +++ b/TablePro/Core/Coordinators/QueryExecutionCoordinator+MultiStatement.swift @@ -48,7 +48,8 @@ extension QueryExecutionCoordinator { let rows = TableRows.from( queryRows: result.rows, columns: result.columns.map { String($0) }, - columnTypes: result.columnTypes + columnTypes: result.columnTypes, + absentCells: result.absentCells ) let resultSet = ResultSet( label: ResultSet.label(tableName: tableName, anchor: anchor, index: index), diff --git a/TablePro/Core/Coordinators/QueryExecutionCoordinator+Parameters.swift b/TablePro/Core/Coordinators/QueryExecutionCoordinator+Parameters.swift index 22dabedcdc..971e3ced9a 100644 --- a/TablePro/Core/Coordinators/QueryExecutionCoordinator+Parameters.swift +++ b/TablePro/Core/Coordinators/QueryExecutionCoordinator+Parameters.swift @@ -640,7 +640,8 @@ extension QueryExecutionCoordinator { historySQL: originalSQL, anchor: anchor, timing: fetchResult.resolvedTiming, - serverOutput: fetchResult.serverOutput + serverOutput: fetchResult.serverOutput, + absentCells: fetchResult.absentCells ) let parameterValues = nativeParameters.map { $0 as? String } diff --git a/TablePro/Core/Coordinators/RowEditingCoordinator+Discard.swift b/TablePro/Core/Coordinators/RowEditingCoordinator+Discard.swift index 56bb65ba2b..ec58cc7e9f 100644 --- a/TablePro/Core/Coordinators/RowEditingCoordinator+Discard.swift +++ b/TablePro/Core/Coordinators/RowEditingCoordinator+Discard.swift @@ -82,15 +82,19 @@ extension RowEditingCoordinator { let tabId = tab.id let insertedIDs = parent.changeManager.insertedRowIDs var restoredCells: [(rowID: RowID, columnIndex: Int)] = [] + var absentCells: Set = [] let edits = parent.changeManager.getOriginalValues().compactMap { original in tableRows.index(of: original.rowID).map { storageRow -> (row: Int, column: Int, value: PluginCellValue) in restoredCells.append((rowID: original.rowID, columnIndex: original.columnIndex)) + if original.isAbsent { + absentCells.insert(CellPosition(row: storageRow, column: original.columnIndex)) + } return (row: storageRow, column: original.columnIndex, value: original.value) } } if !edits.isEmpty { let editDelta = parent.mutateActiveTableRows(for: tabId) { rows in - rows.editMany(edits) + rows.editMany(edits, absentCells: absentCells) } /// `editMany` names the rows it changed by their position in storage, and the grid /// reads a delta's rows as display positions. diff --git a/TablePro/Core/Coordinators/RowEditingCoordinator.swift b/TablePro/Core/Coordinators/RowEditingCoordinator.swift index 4c506abd1f..336725a06b 100644 --- a/TablePro/Core/Coordinators/RowEditingCoordinator.swift +++ b/TablePro/Core/Coordinators/RowEditingCoordinator.swift @@ -193,14 +193,38 @@ final class RowEditingCoordinator: ObservableObject { /// /// A field the selected rows disagree on has no value of its own, and clearing it asks for each /// row's own value back rather than for one value across all of them. - func revertInspectorFieldEdit(columnIndex: Int, valuesByRow: [RowID: PluginCellValue]) { - stageInspectorEdits(valuesByRow: valuesByRow, columnIndex: columnIndex, continuity: .typing) + /// + /// `absentRowIDs` are the rows that had no field for the column, which it goes back to missing. + func revertInspectorFieldEdit( + columnIndex: Int, + valuesByRow: [RowID: PluginCellValue], + absentRowIDs: Set = [] + ) { + stageInspectorEdits( + valuesByRow: valuesByRow, + columnIndex: columnIndex, + continuity: .typing, + removingFieldFrom: absentRowIDs + ) + } + + /// Takes the field out of every row the inspector is showing, for an engine that tells a + /// missing field from NULL. + func stageInspectorFieldRemoval(columnIndex: Int, rowIDs: [RowID]) { + guard parent.changeManager.supportsFieldRemoval else { return } + stageInspectorEdits( + valuesByRow: Dictionary(rowIDs.map { ($0, PluginCellValue.null) }, uniquingKeysWith: { first, _ in first }), + columnIndex: columnIndex, + continuity: .discrete, + removingFieldFrom: Set(rowIDs) + ) } private func stageInspectorEdits( valuesByRow: [RowID: PluginCellValue], columnIndex: Int, - continuity: FieldEditContinuity + continuity: FieldEditContinuity, + removingFieldFrom removedRowIDs: Set = [] ) { if continuity == .discrete { endInspectorEditRun() @@ -216,11 +240,18 @@ final class RowEditingCoordinator: ObservableObject { guard parent.changeManager.isColumnWritable(columnName) else { return } var edits: [(row: Int, column: Int, value: PluginCellValue)] = [] + var removedCells: Set = [] var editedRowIDs: Set = [] for (rowID, value) in valuesByRow { guard let storageRow = tableRows.index(of: rowID) else { continue } - let values = Array(tableRows.rows[storageRow].values) - guard values.indices.contains(columnIndex), values[columnIndex] != value else { continue } + let row = tableRows.rows[storageRow] + let values = Array(row.values) + let removesField = removedRowIDs.contains(rowID) && parent.changeManager.supportsFieldRemoval + let absence = FieldAbsence( + wasAbsent: row.isAbsent(columnIndex), isAbsent: removesField, originalRow: row.absentColumns + ) + guard values.indices.contains(columnIndex), + values[columnIndex] != value || absence.wasAbsent != absence.isAbsent else { continue } if continuity == .typing { parent.changeManager.recordTypedCellChange( rowID: rowID, @@ -228,7 +259,8 @@ final class RowEditingCoordinator: ObservableObject { columnName: columnName, oldValue: values[columnIndex], newValue: value, - originalRow: values + originalRow: values, + absence: absence ) } else { parent.changeManager.recordCellChange( @@ -237,15 +269,19 @@ final class RowEditingCoordinator: ObservableObject { columnName: columnName, oldValue: values[columnIndex], newValue: value, - originalRow: values + originalRow: values, + absence: absence ) } edits.append((row: storageRow, column: columnIndex, value: value)) + if removesField { + removedCells.insert(CellPosition(row: storageRow, column: columnIndex)) + } editedRowIDs.insert(rowID) } guard !edits.isEmpty else { return } - parent.mutateActiveTableRows(for: tabId) { rows in rows.editMany(edits) } + parent.mutateActiveTableRows(for: tabId) { rows in rows.editMany(edits, absentCells: removedCells) } parent.tabManager.mutate(at: tabIndex) { $0.hasUserInteraction = true } repaintInspectorEdit(rowIDs: editedRowIDs, columnIndex: columnIndex, in: tableRows) parent.inspectorRowContentChanged.send() diff --git a/TablePro/Core/DataWrite/DataWritePlan.swift b/TablePro/Core/DataWrite/DataWritePlan.swift index c764f43762..2107ae8c87 100644 --- a/TablePro/Core/DataWrite/DataWritePlan.swift +++ b/TablePro/Core/DataWrite/DataWritePlan.swift @@ -58,8 +58,30 @@ struct RowWriteOperation: Codable, Sendable, Equatable { let writtenColumns: [String] /// Why this row cannot be rewound, or nil when it can. let refusal: RewindRefusal? + /// The columns the row had no field for before the write, on an engine that tells a missing + /// field from NULL. Nil in a record saved before that was captured. + var preImageAbsentColumns: Set? + /// The same for the row as the write left it. + var postImageAbsentColumns: Set? var isReversible: Bool { refusal == nil } + + /// The fields to leave missing when the row is put back as it was before the write. + /// + /// A record saved before absence was captured holds NULL for both, and the app of that time + /// put every NULL back as a missing field, so that is what it still means. + var absentColumnsBeforeWrite: Set { + preImageAbsentColumns ?? Self.nullColumns(of: preImage) + } + + var absentColumnsAfterWrite: Set { + postImageAbsentColumns ?? Self.nullColumns(of: postImage) + } + + private static func nullColumns(of image: [PluginCellValue]?) -> Set { + guard let image else { return [] } + return Set(image.indices.filter { image[$0].isNull }) + } } /// One statement, and how many rows it is allowed to touch. diff --git a/TablePro/Core/DataWrite/Rewind/RewindExecutor.swift b/TablePro/Core/DataWrite/Rewind/RewindExecutor.swift index 6886c0cfdc..143b50e5d1 100644 --- a/TablePro/Core/DataWrite/Rewind/RewindExecutor.swift +++ b/TablePro/Core/DataWrite/Rewind/RewindExecutor.swift @@ -40,6 +40,8 @@ struct RewindExecutor { ) let queries = planner.readQueries() let route = DatabaseManager.shared.executionRoute(for: scope) + let recordedColumns = record.operations.first?.columns ?? [] + let matchingByName = PluginManager.shared.supportsFieldRemoval(for: connection.type) /// Untracked: nothing offers a Stop for a rewind plan, so registering it only exposed the /// read to whatever else on the connection was being cancelled. let currentRows = try await DatabaseManager.shared.withScopedDriver( @@ -47,9 +49,13 @@ struct RewindExecutor { route: route, cancellation: .untracked ) { driver in - var rows: [[PluginCellValue]] = [] + var rows: [RewindCurrentRow] = [] for query in queries { - rows.append(contentsOf: try await driver.execute(query: query).rows) + rows.append(contentsOf: RewindCurrentRow.rows( + of: try await driver.execute(query: query), + alignedTo: recordedColumns, + matchingByName: matchingByName + )) } return rows } @@ -90,7 +96,9 @@ struct RewindExecutor { preImage: source.postImage, postImage: source.preImage, writtenColumns: source.writtenColumns, - refusal: nil + refusal: nil, + preImageAbsentColumns: source.postImageAbsentColumns, + postImageAbsentColumns: source.preImageAbsentColumns ) } guard !operations.isEmpty else { return } diff --git a/TablePro/Core/DataWrite/Rewind/RewindPlanner.swift b/TablePro/Core/DataWrite/Rewind/RewindPlanner.swift index cdb9df9d42..dc5cd29d84 100644 --- a/TablePro/Core/DataWrite/Rewind/RewindPlanner.swift +++ b/TablePro/Core/DataWrite/Rewind/RewindPlanner.swift @@ -37,7 +37,7 @@ struct RewindPlanner { readQueries(for: record.reversibleOperations.filter(keyIsComparable)) } - func plan(currentRows: [[PluginCellValue]]) throws -> RewindPlan { + func plan(currentRows: [RewindCurrentRow]) throws -> RewindPlan { let current = indexByKey(currentRows) var rows: [RewindRowPlan] = [] @@ -62,10 +62,10 @@ struct RewindPlanner { continue } let key = primaryKey(of: operation) - let currentImage = key.flatMap { current[$0] } - let outcome = classify(operation, currentImage: currentImage) + let currentRow = key.flatMap { current[$0] } + let outcome = classify(operation, current: currentRow) rows.append( - RewindRowPlan(id: UUID(), operation: operation, outcome: outcome, currentImage: currentImage) + RewindRowPlan(id: UUID(), operation: operation, outcome: outcome, currentImage: currentRow?.values) ) if outcome.restores { restorable.append(operation) @@ -77,48 +77,63 @@ struct RewindPlanner { // MARK: - Classification - private func classify(_ operation: RowWriteOperation, currentImage: [PluginCellValue]?) -> RewindRowOutcome { + private func classify(_ operation: RowWriteOperation, current: RewindCurrentRow?) -> RewindRowOutcome { switch operation.kind { case .update: - guard let currentImage else { return .rowMissing } + guard let current else { return .rowMissing } guard let postImage = operation.postImage, let preImage = operation.preImage else { return .notReversible(.serverComputedValue) } - if matches(currentImage, preImage, on: operation) { return .alreadyRestored } - guard matches(currentImage, postImage, on: operation) else { return .changedSinceSave } + if matches(current, preImage, absent: operation.preImageAbsentColumns, on: operation) { + return .alreadyRestored + } + guard matches(current, postImage, absent: operation.postImageAbsentColumns, on: operation) else { + return .changedSinceSave + } return .willRestore case .delete: - if currentImage != nil { return .rowAlreadyPresent } + if current != nil { return .rowAlreadyPresent } return .willRestore case .insert: - guard let currentImage else { return .alreadyRestored } + guard let current else { return .alreadyRestored } guard let postImage = operation.postImage else { return .notReversible(.serverAssignedKey) } - guard matches(currentImage, postImage, on: operation) else { return .changedSinceSave } + guard matches(current, postImage, absent: operation.postImageAbsentColumns, on: operation) else { + return .changedSinceSave + } return .willRestore } } + /// Whether the row holds what the image says on every column the save wrote: the same value, + /// and the field there or missing as it was. Set NULL on a missing field and Remove Field on a + /// NULL one leave the value alone, so comparing values alone reads either as already restored. + /// + /// A record saved before absence was captured has none to compare, so it is compared on values + /// alone, as it was then. private func matches( - _ current: [PluginCellValue], + _ current: RewindCurrentRow, _ expected: [PluginCellValue], + absent expectedAbsent: Set?, on operation: RowWriteOperation ) -> Bool { operation.writtenColumns.allSatisfy { column in guard let index = operation.columns.firstIndex(of: column), - index < current.count, index < expected.count + index < current.values.count, index < expected.count, + current.values[index] == expected[index] else { return false } - return current[index] == expected[index] + guard let expectedAbsent else { return true } + return current.absentColumns.contains(index) == expectedAbsent.contains(index) } } // MARK: - Reading - private func indexByKey(_ rows: [[PluginCellValue]]) -> [RewindRowKey: [PluginCellValue]] { + private func indexByKey(_ rows: [RewindCurrentRow]) -> [RewindRowKey: RewindCurrentRow] { let columns = record.operations.first?.columns ?? [] let keyColumns = record.operations.first?.primaryKeyColumns ?? [] - var byKey: [RewindRowKey: [PluginCellValue]] = [:] + var byKey: [RewindRowKey: RewindCurrentRow] = [:] for row in rows { - guard let key = primaryKey(of: row, columns: columns, keyColumns: keyColumns) else { continue } + guard let key = primaryKey(of: row.values, columns: columns, keyColumns: keyColumns) else { continue } byKey[key] = row } return byKey @@ -194,22 +209,28 @@ struct RewindPlanner { switch operation.kind { case .update: guard let preImage = operation.preImage, let postImage = operation.postImage else { return [] } + let absentBefore = operation.absentColumnsBeforeWrite + let absentAfter = operation.absentColumnsAfterWrite let cellChanges = operation.writtenColumns.compactMap { column -> CellChange? in guard let index = operation.columns.firstIndex(of: column), index < preImage.count, index < postImage.count else { return nil } return CellChange( columnIndex: index, columnName: column, - oldValue: postImage[index], newValue: preImage[index] + oldValue: postImage[index], newValue: preImage[index], + oldIsAbsent: absentAfter.contains(index), newIsAbsent: absentBefore.contains(index) ) } guard !cellChanges.isEmpty else { return [] } return try factory.statements( - for: [RowChange(rowID: .existing(0), type: .update, cellChanges: cellChanges, originalRow: postImage)] + for: [RowChange( + rowID: .existing(0), type: .update, cellChanges: cellChanges, + originalRow: postImage, absentColumns: absentAfter + )] ) case .delete: guard let preImage = operation.preImage else { return [] } - return try factory.restoreStatements(rows: [preImage]) + return try factory.restoreStatements(rows: [preImage], absentCells: [0: operation.absentColumnsBeforeWrite]) case .insert: guard let postImage = operation.postImage else { return [] } return try factory.statements( @@ -263,3 +284,43 @@ struct RewindPlanner { struct RewindRowKey: Hashable, Sendable { let values: [PluginCellValue] } + +/// A row a rewind read back, lined up with the columns its record was written against. +struct RewindCurrentRow: Sendable, Equatable { + let values: [PluginCellValue] + /// The recorded columns the row has no field for, which only an engine that tells a missing + /// field from NULL reports. + var absentColumns: Set = [] + + /// A read's rows in the recorded column order. + /// + /// A SQL read projects the recorded columns, so its rows already line up. A document store + /// returns the fields its documents have, in its own order, and a field none of them has is + /// not a column at all, so with `matchingByName` each row is matched to the record by field + /// name, and a recorded field the read did not return is one the row does not have. + static func rows( + of result: QueryResult, + alignedTo columns: [String], + matchingByName: Bool + ) -> [RewindCurrentRow] { + guard matchingByName, result.columns != columns else { + return result.rows.enumerated().map { index, values in + RewindCurrentRow(values: values, absentColumns: result.absentCells[index] ?? []) + } + } + let sourceIndices = columns.map { result.columns.firstIndex(of: $0) } + return result.rows.enumerated().map { index, values in + let absentInResult = result.absentCells[index] ?? [] + var absent: Set = [] + let aligned = sourceIndices.enumerated().map { column, source -> PluginCellValue in + guard let source, source < values.count else { + absent.insert(column) + return .null + } + if absentInResult.contains(source) { absent.insert(column) } + return values[source] + } + return RewindCurrentRow(values: aligned, absentColumns: absent) + } + } +} diff --git a/TablePro/Core/DataWrite/RowChangeStatementFactory.swift b/TablePro/Core/DataWrite/RowChangeStatementFactory.swift index 58269b0b68..6f152bc919 100644 --- a/TablePro/Core/DataWrite/RowChangeStatementFactory.swift +++ b/TablePro/Core/DataWrite/RowChangeStatementFactory.swift @@ -130,14 +130,20 @@ struct RowChangeStatementFactory { /// the server pick the key: MongoDB's drops `_id` on purpose. Replaying that to undo a delete /// produces a different document rather than the one that went missing, so a driver with its /// own statement generation has to answer this separately or say it cannot. - func restoreStatements(rows: [[PluginCellValue]]) throws -> [ParameterizedStatement] { + /// + /// `absentCells` names, by row index, the fields a row did not have, which stay missing. + func restoreStatements( + rows: [[PluginCellValue]], + absentCells: [Int: Set] = [:] + ) throws -> [ParameterizedStatement] { if let pluginDriver { if let restored = pluginDriver.generateIdentityPreservingInsert( table: tableName, schema: schemaName, columns: columns, primaryKeyColumns: primaryKeyColumns, - rows: rows + rows: rows, + absentCells: absentCells ) { return restored.map { ParameterizedStatement(sql: $0.statement, parameters: $0.parameters.map(\.asAny)) @@ -319,5 +325,22 @@ private extension PluginRowChange { }, originalRow: change.originalRow ) + absentColumns = Self.absentColumns(after: change) + } + + /// The fields the row has none of once the change is written: the ones an update removes, or + /// the ones a new row leaves out. Nil when there are none, which is every engine but a + /// document store. + static func absentColumns(after change: RowChange) -> Set? { + let absent: Set + switch change.type { + case .update: + absent = Set(change.cellChanges.filter(\.newIsAbsent).map(\.columnIndex)) + case .insert: + absent = change.absentColumns + case .delete: + return nil + } + return absent.isEmpty ? nil : absent } } diff --git a/TablePro/Core/DataWrite/RowWriteOperationBuilder.swift b/TablePro/Core/DataWrite/RowWriteOperationBuilder.swift index d6da086be3..3c9934f966 100644 --- a/TablePro/Core/DataWrite/RowWriteOperationBuilder.swift +++ b/TablePro/Core/DataWrite/RowWriteOperationBuilder.swift @@ -87,7 +87,9 @@ enum RowWriteOperationBuilder { refusal: updateRefusal( change, preImage: preImage, primaryKeyColumns: primaryKeyColumns, generatedColumns: generatedColumns, containsTableOperation: containsTableOperation - ) + ), + preImageAbsentColumns: change.absentColumns, + postImageAbsentColumns: change.absentColumnsAfterUpdate ) } @@ -132,7 +134,8 @@ enum RowWriteOperationBuilder { preImage: change.originalRow, postImage: nil, writtenColumns: columns, - refusal: refusal + refusal: refusal, + preImageAbsentColumns: change.absentColumns ) } @@ -168,7 +171,8 @@ enum RowWriteOperationBuilder { preImage: nil, postImage: values, writtenColumns: columns, - refusal: refusal + refusal: refusal, + postImageAbsentColumns: change.absentColumns ) } diff --git a/TablePro/Core/Plugins/PluginDriverAdapter.swift b/TablePro/Core/Plugins/PluginDriverAdapter.swift index 2cabe5e355..1ee1b1dfe1 100644 --- a/TablePro/Core/Plugins/PluginDriverAdapter.swift +++ b/TablePro/Core/Plugins/PluginDriverAdapter.swift @@ -934,6 +934,7 @@ final class PluginDriverAdapter: DatabaseDriver, SchemaSwitchable, DatabaseRepor result.columnMeta = pluginResult.columnMeta?.map { ResultColumnMeta(isPrimaryKey: $0.isPrimaryKey, isNullable: $0.isNullable, isAutoIncrement: $0.isIdentity) } + result.absentCells = pluginResult.absentCells ?? [:] return result } diff --git a/TablePro/Core/Plugins/PluginManager+Registration.swift b/TablePro/Core/Plugins/PluginManager+Registration.swift index 4e1629068c..21fd9483f1 100644 --- a/TablePro/Core/Plugins/PluginManager+Registration.swift +++ b/TablePro/Core/Plugins/PluginManager+Registration.swift @@ -599,6 +599,11 @@ extension PluginManager { .capabilities.supportsDocumentEditing ?? false } + func supportsFieldRemoval(for databaseType: DatabaseType) -> Bool { + PluginMetadataRegistry.shared.snapshot(for: databaseType)? + .capabilities.supportsFieldRemoval ?? false + } + func supportsRenameTable(for databaseType: DatabaseType) -> Bool { PluginMetadataRegistry.shared.snapshot(for: databaseType)? .capabilities.supportsRenameTable ?? false diff --git a/TablePro/Core/Plugins/PluginMetadataRegistry+RegistryDefaults.swift b/TablePro/Core/Plugins/PluginMetadataRegistry+RegistryDefaults.swift index 75083b5ee1..82a5c078a4 100644 --- a/TablePro/Core/Plugins/PluginMetadataRegistry+RegistryDefaults.swift +++ b/TablePro/Core/Plugins/PluginMetadataRegistry+RegistryDefaults.swift @@ -40,6 +40,7 @@ extension PluginMetadataRegistry { requiresReconnectForDatabaseSwitch: false, supportsDropDatabase: true, supportsDocumentEditing: true, + supportsFieldRemoval: true, supportsOpportunisticTLS: false, authenticationIsDatabaseScoped: true ), diff --git a/TablePro/Core/Plugins/PluginMetadataRegistry.swift b/TablePro/Core/Plugins/PluginMetadataRegistry.swift index e9057ad406..ad1fbc1f1d 100644 --- a/TablePro/Core/Plugins/PluginMetadataRegistry.swift +++ b/TablePro/Core/Plugins/PluginMetadataRegistry.swift @@ -52,6 +52,7 @@ struct PluginMetadataSnapshot: Sendable { var supportsRenameDatabase: Bool = false var supportsRenameSchema: Bool = false var supportsDocumentEditing: Bool = false + var supportsFieldRemoval: Bool = false // `var` with defaults so existing call sites compile without passing these fields var supportsDropSchema: Bool = false var supportsCreateSchema: Bool = false @@ -683,6 +684,7 @@ final class PluginMetadataRegistry: @unchecked Sendable { supportsRenameDatabase: driverType.supportsRenameDatabase, supportsRenameSchema: driverType.supportsRenameSchema, supportsDocumentEditing: driverType.supportsDocumentEditing, + supportsFieldRemoval: driverType.supportsFieldRemoval, supportsDropSchema: driverType.supportsDropSchema, supportsCreateSchema: driverType.supportsCreateSchema, supportsSchemaOwner: driverType.supportsSchemaOwner, diff --git a/TablePro/Core/Services/Infrastructure/ClipboardService.swift b/TablePro/Core/Services/Infrastructure/ClipboardService.swift index 6440713766..715af7f1c3 100644 --- a/TablePro/Core/Services/Infrastructure/ClipboardService.swift +++ b/TablePro/Core/Services/Infrastructure/ClipboardService.swift @@ -11,6 +11,26 @@ import UniformTypeIdentifiers struct GridRowsClipboardPayload: Codable, Equatable { let columns: [String] let rows: [[PluginCellValue]] + /// Row index to the columns that row has no field for, so a paste leaves them missing. + var absentCells: [Int: Set]? +} + +extension GridRowsClipboardPayload { + /// Rows copied out of a grid, in the columns the copy carries. Every copy builds its payload + /// here, so the fields a row has none of travel with its values rather than each copy path + /// deciding for itself whether to carry them. + init(columns: [String], copying copiedRows: [Row], projection: VisibleColumnProjection) { + var absentCells: [Int: Set] = [:] + for (index, row) in copiedRows.enumerated() { + let absent = projection.absentColumns(row.absentColumns) + if !absent.isEmpty { absentCells[index] = absent } + } + self.init( + columns: columns, + rows: copiedRows.map { projection.values(Array($0.values)) }, + absentCells: absentCells.isEmpty ? nil : absentCells + ) + } } protocol ClipboardProvider { diff --git a/TablePro/Core/Services/Query/QueryExecutor.swift b/TablePro/Core/Services/Query/QueryExecutor.swift index 3b2208cc11..b8c6733ce1 100644 --- a/TablePro/Core/Services/Query/QueryExecutor.swift +++ b/TablePro/Core/Services/Query/QueryExecutor.swift @@ -20,6 +20,9 @@ struct QueryFetchResult { /// What the statement printed on the server, read on its own session. var serverOutput: PluginServerOutput = .none + /// Row index to the columns that row has no field for. See `PluginQueryResult.absentCells`. + var absentCells: [Int: Set] = [:] + var resolvedTiming: PluginQueryTiming { timing ?? PluginQueryTiming(total: executionTime) } @@ -154,7 +157,8 @@ final class QueryExecutor { statusMessage: result.statusMessage, isTruncated: result.isTruncated, resultColumnMeta: result.columnMeta, - timing: result.timing + timing: result.timing, + absentCells: result.absentCells ) } @@ -180,7 +184,8 @@ final class QueryExecutor { statusMessage: result.statusMessage, isTruncated: result.isTruncated, resultColumnMeta: result.columnMeta, - timing: result.timing + timing: result.timing, + absentCells: result.absentCells ) } @@ -204,7 +209,8 @@ final class QueryExecutor { statusMessage: result.statusMessage, isTruncated: result.isTruncated, resultColumnMeta: result.columnMeta, - timing: result.timing + timing: result.timing, + absentCells: result.absentCells ) } diff --git a/TablePro/Core/Services/Query/RowOperationsManager.swift b/TablePro/Core/Services/Query/RowOperationsManager.swift index 545c48c677..45fce4b452 100644 --- a/TablePro/Core/Services/Query/RowOperationsManager.swift +++ b/TablePro/Core/Services/Query/RowOperationsManager.swift @@ -43,16 +43,21 @@ final class RowOperationsManager { self.changeManager = changeManager } + /// A new row holds DEFAULT where the server assigns the value and NULL elsewhere. On an engine + /// that tells a missing field from NULL those other fields start missing instead, so a document + /// saved as it was added holds only what the user typed. func addNewRow(tableRows: inout TableRows) -> AddNewRowResult? { var newRowValues: [PluginCellValue] = [] - for column in tableRows.columns { + var absentColumns: Set = [] + for (index, column) in tableRows.columns.enumerated() { if tableRows.generatedColumns.contains(column) || tableRows.serverAssignsValue(forColumn: column) { newRowValues.append(.text("__DEFAULT__")) } else { newRowValues.append(.null) + if changeManager.supportsFieldRemoval { absentColumns.insert(index) } } } - return appendInsertedRow(values: newRowValues, to: &tableRows) + return appendInsertedRow(values: newRowValues, absentColumns: absentColumns, to: &tableRows) } func duplicateRow( @@ -61,7 +66,9 @@ final class RowOperationsManager { ) -> AddNewRowResult? { guard sourceRowIndex >= 0, sourceRowIndex < tableRows.count else { return nil } - var newValues = Array(tableRows.rows[sourceRowIndex].values) + let source = tableRows.rows[sourceRowIndex] + var newValues = Array(source.values) + var absentColumns = source.absentColumns /// An identity column is not always the primary key, and copying its value verbatim is /// what the server rejects. @@ -71,15 +78,20 @@ final class RowOperationsManager { for resetColumn in resetColumns { if let index = tableRows.columns.firstIndex(of: resetColumn), index < newValues.count { newValues[index] = .text("__DEFAULT__") + absentColumns.remove(index) } } - return appendInsertedRow(values: newValues, to: &tableRows) + return appendInsertedRow(values: newValues, absentColumns: absentColumns, to: &tableRows) } - private func appendInsertedRow(values: [PluginCellValue], to tableRows: inout TableRows) -> AddNewRowResult { + private func appendInsertedRow( + values: [PluginCellValue], + absentColumns: Set, + to tableRows: inout TableRows + ) -> AddNewRowResult { let rowID = RowID.inserted(UUID()) - let delta = tableRows.appendInsertedRow(id: rowID, values: values) - changeManager.recordRowInsertion(rowID: rowID, values: values) + let delta = tableRows.appendInsertedRow(id: rowID, values: values, absentColumns: absentColumns) + changeManager.recordRowInsertion(rowID: rowID, values: values, absentColumns: absentColumns) return AddNewRowResult(rowID: rowID, values: values, delta: delta) } @@ -95,6 +107,7 @@ final class RowOperationsManager { let displayCountBefore = displayIDs?.count ?? tableRows.count var insertedRowsToRemove: [InsertedRowLocation] = [] var existingRowsToDelete: [(rowID: RowID, originalRow: [PluginCellValue])] = [] + var deletedAbsentColumns: [RowID: Set] = [:] for displayIndex in selectedIndices.sorted(by: >) { guard let storageIndex = DisplayRowMapping.rowIndex( @@ -105,6 +118,9 @@ final class RowOperationsManager { insertedRowsToRemove.append(InsertedRowLocation(rowID: row.id, storageIndex: storageIndex)) } else if !changeManager.isRowDeleted(row.id) { existingRowsToDelete.append((rowID: row.id, originalRow: Array(row.values))) + if !row.absentColumns.isEmpty { + deletedAbsentColumns[row.id] = row.absentColumns + } } } @@ -115,7 +131,7 @@ final class RowOperationsManager { } if !existingRowsToDelete.isEmpty { - changeManager.recordBatchRowDeletion(rows: existingRowsToDelete) + changeManager.recordBatchRowDeletion(rows: existingRowsToDelete, absentColumns: deletedAbsentColumns) } return DeleteRowsResult( @@ -148,14 +164,16 @@ final class RowOperationsManager { func applyUndoResult(_ result: UndoResult, tableRows: inout TableRows) -> UndoApplicationResult { switch result.action { - case .cellEdit(let rowID, let columnIndex, _, let previousValue, _, _): + case .cellEdit(let rowID, let columnIndex, _, let previousValue, _, _, let absence): guard let storageRow = tableRows.index(of: rowID) else { return UndoApplicationResult(adjustedSelection: nil, delta: .none) } - let delta = tableRows.edit(row: storageRow, column: columnIndex, value: previousValue) + let delta = tableRows.edit( + row: storageRow, column: columnIndex, value: previousValue, isAbsent: absence.wasAbsent + ) return UndoApplicationResult(adjustedSelection: nil, delta: delta) - case .rowInsertion(let rowID): + case .rowInsertion(let rowID, _): if result.needsRowRemoval { let delta = tableRows.remove(rowIDs: [rowID]) guard delta != .none else { @@ -166,7 +184,9 @@ final class RowOperationsManager { if result.needsRowRestore { let values = result.restoreRow ?? [PluginCellValue](repeating: .null, count: tableRows.columns.count) - let delta = tableRows.appendInsertedRow(id: rowID, values: values) + let delta = tableRows.appendInsertedRow( + id: rowID, values: values, absentColumns: result.restoreAbsentColumns + ) return UndoApplicationResult(adjustedSelection: nil, delta: delta) } return UndoApplicationResult(adjustedSelection: nil, delta: .none) @@ -174,7 +194,7 @@ final class RowOperationsManager { case .rowDeletion, .batchRowDeletion: return UndoApplicationResult(adjustedSelection: nil, delta: result.delta) - case .batchRowInsertion(let rows, let rowValues): + case .batchRowInsertion(let rows, let rowValues, let rowAbsentColumns): if result.needsRowRemoval { let delta = tableRows.remove(rowIDs: Set(rows.map(\.rowID))) return UndoApplicationResult(adjustedSelection: nil, delta: delta) @@ -182,7 +202,9 @@ final class RowOperationsManager { if result.needsRowRestore { return UndoApplicationResult( adjustedSelection: nil, - delta: restoreInsertedRows(rows, values: rowValues, into: &tableRows) + delta: restoreInsertedRows( + rows, values: rowValues, absentColumns: rowAbsentColumns, into: &tableRows + ) ) } return UndoApplicationResult(adjustedSelection: nil, delta: .none) @@ -192,13 +214,18 @@ final class RowOperationsManager { private func restoreInsertedRows( _ rows: [InsertedRowLocation], values rowValues: [[PluginCellValue]], + absentColumns rowAbsentColumns: [Set], into tableRows: inout TableRows ) -> Delta { var insertedIndices = IndexSet() - let ascending = zip(rows, rowValues).sorted { $0.0.storageIndex < $1.0.storageIndex } - for (location, values) in ascending { - let index = min(location.storageIndex, tableRows.count) - guard tableRows.insertInsertedRow(at: index, id: location.rowID, values: values) != .none else { + let ascending = zip(rows, rowValues).enumerated() + .map { (location: $0.element.0, values: $0.element.1, absent: rowAbsentColumns[safe: $0.offset] ?? []) } + .sorted { $0.location.storageIndex < $1.location.storageIndex } + for restored in ascending { + let index = min(restored.location.storageIndex, tableRows.count) + guard tableRows.insertInsertedRow( + at: index, id: restored.location.rowID, values: restored.values, absentColumns: restored.absent + ) != .none else { continue } insertedIndices.insert(index) @@ -232,8 +259,8 @@ final class RowOperationsManager { let estimatedRowLength = max(columns.count, 1) * 12 var result = "" result.reserveCapacity(indicesToCopy.count * estimatedRowLength) - var structuredRows: [[PluginCellValue]] = [] - structuredRows.reserveCapacity(indicesToCopy.count) + var copiedRows: [Row] = [] + copiedRows.reserveCapacity(indicesToCopy.count) if includeHeaders, !columns.isEmpty { for (colIdx, col) in columns.enumerated() { @@ -246,9 +273,8 @@ final class RowOperationsManager { guard let row = DisplayRowMapping.row(forDisplay: displayIndex, displayIDs: displayIDs, in: tableRows) else { continue } if !result.isEmpty { result.append("\n") } - let cells = projection.values(Array(row.values)) - structuredRows.append(cells) - for (colIdx, cell) in cells.enumerated() { + copiedRows.append(row) + for (colIdx, cell) in projection.values(Array(row.values)).enumerated() { if colIdx > 0 { result.append("\t") } switch cell { case .null: @@ -265,7 +291,7 @@ final class RowOperationsManager { result.append("\n(truncated, showing first \(Self.maxClipboardRows) of \(totalSelected) rows)") } - let payload = GridRowsClipboardPayload(columns: columns, rows: structuredRows) + let payload = GridRowsClipboardPayload(columns: columns, copying: copiedRows, projection: projection) ClipboardService.shared.writeRows(tsv: result, html: nil, gridRows: payload) } @@ -282,8 +308,11 @@ final class RowOperationsManager { primaryKeyColumns: primaryKeyColumns ) + let keepsMissingFields = changeManager.supportsFieldRemoval if parser == nil, let payload = clipboardProvider.readGridRows() { - let parsedRows = Self.reconcileStructuredRows(payload, schema: schema) + let parsedRows = Self.reconcileStructuredRows( + payload, schema: schema, keepsMissingFields: keepsMissingFields + ) return insertParsedRows(parsedRows, into: &tableRows) } @@ -296,7 +325,10 @@ final class RowOperationsManager { switch parseResult { case .success(let parsedRows): - return insertParsedRows(parsedRows, into: &tableRows) + return insertParsedRows( + keepsMissingFields ? parsedRows.map(Self.nullsAsMissingFields) : parsedRows, + into: &tableRows + ) case .failure(let error): Self.logger.warning("Paste failed: \(error.localizedDescription)") @@ -304,26 +336,44 @@ final class RowOperationsManager { } } + /// Copied rows keep the fields they did not have, and a column the copy did not carry is a + /// field the pasted row does not have either, on an engine that tells that apart from NULL. private static func reconcileStructuredRows( _ payload: GridRowsClipboardPayload, - schema: TableSchema + schema: TableSchema, + keepsMissingFields: Bool ) -> [ParsedRow] { let sourceForDestination = sourceColumnIndices(from: payload.columns, to: schema.columns) return payload.rows.enumerated().map { index, row in - var values: [PluginCellValue] = sourceForDestination.map { sourceIndex in - guard let sourceIndex, sourceIndex < row.count else { return .null } + let sourceAbsent = payload.absentCells?[index] ?? [] + var absentColumns: Set = [] + var values: [PluginCellValue] = sourceForDestination.enumerated().map { destination, sourceIndex in + guard let sourceIndex, sourceIndex < row.count else { + if keepsMissingFields { absentColumns.insert(destination) } + return .null + } + if keepsMissingFields, sourceAbsent.contains(sourceIndex) { absentColumns.insert(destination) } return row[sourceIndex] } if let pkIndex = schema.primaryKeyIndex, pkIndex < values.count { values[pkIndex] = .text("__DEFAULT__") + absentColumns.remove(pkIndex) } - return ParsedRow(values: values, sourceLineNumber: index + 1) + return ParsedRow(values: values, sourceLineNumber: index + 1, absentColumns: absentColumns) } } + /// Text says nothing about which fields a row lacks, so on an engine that tells a missing field + /// from NULL a pasted NULL leaves the field out, which is what a new row does with it too. + private static func nullsAsMissingFields(_ row: ParsedRow) -> ParsedRow { + var missing = row + missing.absentColumns = Set(row.values.indices.filter { row.values[$0].isNull }) + return missing + } + private static func sourceColumnIndices(from source: [String], to destination: [String]) -> [Int?] { var sourceIndexByName: [String: Int] = [:] for (index, name) in source.enumerated() where sourceIndexByName[name] == nil { @@ -370,11 +420,13 @@ final class RowOperationsManager { for parsedRow in parsedRows { var rowValues = parsedRow.values + var absentColumns = parsedRow.absentColumns for index in serverOwned where index < rowValues.count { rowValues[index] = .text("__DEFAULT__") + absentColumns.remove(index) } insertedIndices.insert(tableRows.count) - let inserted = appendInsertedRow(values: rowValues, to: &tableRows) + let inserted = appendInsertedRow(values: rowValues, absentColumns: absentColumns, to: &tableRows) pastedRowInfo.append(PastedRowInfo(rowID: inserted.rowID, values: rowValues)) } diff --git a/TablePro/Models/Query/ParsedRow.swift b/TablePro/Models/Query/ParsedRow.swift index 1efa6db961..3d520f8afc 100644 --- a/TablePro/Models/Query/ParsedRow.swift +++ b/TablePro/Models/Query/ParsedRow.swift @@ -16,6 +16,9 @@ struct ParsedRow { /// Original line number in clipboard (for error reporting) let sourceLineNumber: Int + /// The columns the row has no field for, for an engine that tells a missing field from NULL. + var absentColumns: Set = [] + /// Check if row has valid data var isValid: Bool { !values.isEmpty diff --git a/TablePro/Models/Query/QueryResult.swift b/TablePro/Models/Query/QueryResult.swift index 3c6c38b016..ad94c1dad6 100644 --- a/TablePro/Models/Query/QueryResult.swift +++ b/TablePro/Models/Query/QueryResult.swift @@ -35,6 +35,9 @@ struct QueryResult { /// What the statement printed on the server, read on its own session. var serverOutput: PluginServerOutput = .none + /// Row index to the columns that row has no field for. See `PluginQueryResult.absentCells`. + var absentCells: [Int: Set] = [:] + var isEmpty: Bool { rows.isEmpty } diff --git a/TablePro/Models/Query/Row.swift b/TablePro/Models/Query/Row.swift index 92d42f207b..13dc61185d 100644 --- a/TablePro/Models/Query/Row.swift +++ b/TablePro/Models/Query/Row.swift @@ -19,6 +19,13 @@ enum RowID: Hashable, Sendable { struct Row: Equatable, Sendable { var id: RowID var values: ContiguousArray + /// The columns this row has no field for, which only an engine that tells a missing field from + /// NULL reports. Their values read `.null`. + var absentColumns: Set = [] + + func isAbsent(_ column: Int) -> Bool { + absentColumns.contains(column) + } subscript(column: Int) -> PluginCellValue { get { column >= 0 && column < values.count ? values[column] : .null } diff --git a/TablePro/Models/Query/TableRows.swift b/TablePro/Models/Query/TableRows.swift index 67472804cb..989ea23117 100644 --- a/TablePro/Models/Query/TableRows.swift +++ b/TablePro/Models/Query/TableRows.swift @@ -83,6 +83,11 @@ struct TableRows: Sendable { return rows[row][column] } + func isAbsent(row: Int, column: Int) -> Bool { + guard row >= 0, row < rows.count else { return false } + return rows[row].isAbsent(column) + } + func index(of id: RowID) -> Int? { indexByID[id] } @@ -98,36 +103,60 @@ struct TableRows: Sendable { indexByID = [:] } + /// Writes a value into a cell, or with `isAbsent` takes the field out of the row. A value + /// written into a missing field puts the field back. @discardableResult - mutating func edit(row: Int, column: Int, value: PluginCellValue) -> Delta { + mutating func edit(row: Int, column: Int, value: PluginCellValue, isAbsent: Bool = false) -> Delta { guard row >= 0, row < rows.count else { return .none } guard column >= 0, column < columns.count else { return .none } guard column < rows[row].values.count else { return .none } - if rows[row].values[column] == value { return .none } - rows[row].values[column] = value + guard write(isAbsent ? .null : value, isAbsent: isAbsent, row: row, column: column) else { return .none } return .cellChanged(row: row, column: column) } + /// Writes each value, and takes the field out of the row for every position in `absentCells`. @discardableResult - mutating func editMany(_ edits: [(row: Int, column: Int, value: PluginCellValue)]) -> Delta { + mutating func editMany( + _ edits: [(row: Int, column: Int, value: PluginCellValue)], + absentCells: Set = [] + ) -> Delta { var changed: Set = [] for edit in edits { guard edit.row >= 0, edit.row < rows.count else { continue } guard edit.column >= 0, edit.column < columns.count else { continue } guard edit.column < rows[edit.row].values.count else { continue } - if rows[edit.row].values[edit.column] == edit.value { continue } - rows[edit.row].values[edit.column] = edit.value - changed.insert(CellPosition(row: edit.row, column: edit.column)) + let position = CellPosition(row: edit.row, column: edit.column) + let isAbsent = absentCells.contains(position) + guard write(isAbsent ? .null : edit.value, isAbsent: isAbsent, row: edit.row, column: edit.column) else { + continue + } + changed.insert(position) } if changed.isEmpty { return .none } return .cellsChanged(changed) } + private mutating func write(_ value: PluginCellValue, isAbsent: Bool, row: Int, column: Int) -> Bool { + let wasAbsent = rows[row].isAbsent(column) + guard rows[row].values[column] != value || wasAbsent != isAbsent else { return false } + rows[row].values[column] = value + if isAbsent { + rows[row].absentColumns.insert(column) + } else if wasAbsent { + rows[row].absentColumns.remove(column) + } + return true + } + @discardableResult - mutating func appendInsertedRow(id: RowID = .inserted(UUID()), values: [PluginCellValue]) -> Delta { + mutating func appendInsertedRow( + id: RowID = .inserted(UUID()), + values: [PluginCellValue], + absentColumns: Set = [] + ) -> Delta { guard indexByID[id] == nil else { return .none } let normalized = Self.normalize(values: values, toCount: columns.count) - let row = Row(id: id, values: normalized) + let row = Row(id: id, values: normalized, absentColumns: Self.clamp(absentColumns, toCount: columns.count)) let newIndex = rows.count rows.append(row) indexByID[row.id] = newIndex @@ -138,11 +167,12 @@ struct TableRows: Sendable { mutating func insertInsertedRow( at index: Int, id: RowID = .inserted(UUID()), - values: [PluginCellValue] + values: [PluginCellValue], + absentColumns: Set = [] ) -> Delta { guard index >= 0, index <= rows.count, indexByID[id] == nil else { return .none } let normalized = Self.normalize(values: values, toCount: columns.count) - let row = Row(id: id, values: normalized) + let row = Row(id: id, values: normalized, absentColumns: Self.clamp(absentColumns, toCount: columns.count)) rows.insert(row, at: index) for offset in index.. Delta { + mutating func replace( + rows replacementRows: [[PluginCellValue]], + offset: Int = 0, + absentCells: [Int: Set] = [:] + ) -> Delta { var rebuilt = ContiguousArray() rebuilt.reserveCapacity(replacementRows.count) var rebuiltIndex = [RowID: Int]() rebuiltIndex.reserveCapacity(replacementRows.count) for (idx, values) in replacementRows.enumerated() { let normalized = Self.normalize(values: values, toCount: columns.count) - let row = Row(id: .existing(offset + idx), values: normalized) + let row = Row( + id: .existing(offset + idx), + values: normalized, + absentColumns: Self.clamp(absentCells[idx] ?? [], toCount: columns.count) + ) rebuilt.append(row) rebuiltIndex[row.id] = idx } @@ -284,13 +322,18 @@ struct TableRows: Sendable { generatedColumns: Set = [], rowMatchPolicy: RowMatchPolicy = .none, hasAuthoritativeSchema: Bool = false, - foreignKeysFetched: Bool = false + foreignKeysFetched: Bool = false, + absentCells: [Int: Set] = [:] ) -> TableRows { var rows = ContiguousArray() rows.reserveCapacity(queryRows.count) for (index, values) in queryRows.enumerated() { let normalized = normalize(values: values, toCount: columns.count) - rows.append(Row(id: .existing(index), values: normalized)) + rows.append(Row( + id: .existing(index), + values: normalized, + absentColumns: clamp(absentCells[index] ?? [], toCount: columns.count) + )) } return TableRows( rows: rows, @@ -339,6 +382,11 @@ struct TableRows: Sendable { return result } + private static func clamp(_ absentColumns: Set, toCount targetCount: Int) -> Set { + guard absentColumns.contains(where: { $0 < 0 || $0 >= targetCount }) else { return absentColumns } + return absentColumns.filter { $0 >= 0 && $0 < targetCount } + } + private static func buildIndex(for rows: ContiguousArray) -> [RowID: Int] { var index = [RowID: Int]() index.reserveCapacity(rows.count) diff --git a/TablePro/Models/Query/VisibleColumnProjection.swift b/TablePro/Models/Query/VisibleColumnProjection.swift index 1f89ac37b4..147ee72b71 100644 --- a/TablePro/Models/Query/VisibleColumnProjection.swift +++ b/TablePro/Models/Query/VisibleColumnProjection.swift @@ -29,4 +29,10 @@ struct VisibleColumnProjection { guard let indices else { return all } return indices.map { all.indices.contains($0) ? all[$0] : .null } } + + /// The positions, in the projected order, of the columns a row has no field for. + func absentColumns(_ all: Set) -> Set { + guard let indices else { return all } + return Set(indices.indices.filter { all.contains(indices[$0]) }) + } } diff --git a/TablePro/Models/UI/FieldValueState.swift b/TablePro/Models/UI/FieldValueState.swift index 716218c029..7f63878c0a 100644 --- a/TablePro/Models/UI/FieldValueState.swift +++ b/TablePro/Models/UI/FieldValueState.swift @@ -33,12 +33,18 @@ internal enum FieldValueState: Equatable { case pendingDefault /// Several rows are selected and they do not agree on this field. case multipleValues + /// The rows have no field for this column, which a document store tells apart from NULL. + case absent + /// The user asked for the field to be taken out of the row. + case pendingRemoval internal static func resolve(_ field: FieldEditState) -> FieldValueState { if field.isPendingNull { return .pendingNull } if field.isPendingDefault { return .pendingDefault } + if field.isPendingRemoval { return .pendingRemoval } if let pending = field.pendingValue { return .value(pending) } if field.hasMultipleValues { return .multipleValues } + if field.isAbsent { return .absent } guard let original = field.originalValue else { return .null } return .value(original) } @@ -48,7 +54,7 @@ internal enum FieldValueState: Equatable { internal var editableText: String { switch self { case .value(let text): return text - case .null, .pendingNull, .pendingDefault, .multipleValues: return "" + case .null, .pendingNull, .pendingDefault, .multipleValues, .absent, .pendingRemoval: return "" } } @@ -60,14 +66,15 @@ internal enum FieldValueState: Equatable { case .null, .pendingNull: return "NULL" case .pendingDefault: return "DEFAULT" case .multipleValues: return String(localized: "Multiple values") + case .absent, .pendingRemoval: return String(localized: "No Field") } } /// Whether the user has asked for something the stored row does not hold. internal var isPending: Bool { switch self { - case .pendingNull, .pendingDefault: return true - case .value, .null, .multipleValues: return false + case .pendingNull, .pendingDefault, .pendingRemoval: return true + case .value, .null, .multipleValues, .absent: return false } } } diff --git a/TablePro/Models/UI/MultiRowEditState.swift b/TablePro/Models/UI/MultiRowEditState.swift index 484843ed34..a4af58980e 100644 --- a/TablePro/Models/UI/MultiRowEditState.swift +++ b/TablePro/Models/UI/MultiRowEditState.swift @@ -54,8 +54,15 @@ struct FieldEditState: Identifiable { var isPendingDefault: Bool + /// No selected row has a field for the column, which only an engine that tells a missing field + /// from NULL reports. + var isAbsent = false + + /// The user asked for the field to be taken out of the row. + var isPendingRemoval = false + var hasEdit: Bool { - pendingValue != nil || isPendingNull || isPendingDefault + pendingValue != nil || isPendingNull || isPendingDefault || isPendingRemoval } var effectiveValue: String? { @@ -74,6 +81,17 @@ enum FieldEditContinuity { case discrete } +/// A field the inspector writes itself, for the save that runs when no pending change holds its +/// edits. +struct InspectorFieldEdit: Equatable { + let columnIndex: Int + let columnName: String + let newValue: String? + /// The edit takes the field out of the row, which `newValue` cannot say: it reads nil for NULL + /// as well. + var removesField = false +} + /// Manages edit state for multi-row editing in sidebar @MainActor final class MultiRowEditState: ObservableObject { @@ -83,9 +101,13 @@ final class MultiRowEditState: ObservableObject { /// undo step; choosing NULL, DEFAULT, a function or a picker value is its own step. @Published var onFieldChanged: ((Int, PluginCellValue, FieldEditContinuity) -> Void)? - /// A field the selected rows disagree on, cleared back to nothing. It has no single value to - /// send, so it asks for each row's own configured value instead. - @Published var onFieldReverted: ((Int, [RowID: PluginCellValue]) -> Void)? + /// A field cleared back to what the rows held, when they disagree on it or some of them have + /// no field for it. It has no single value to send, so it asks for each row's own configured + /// value instead, and for the field to go missing again on the rows that lacked it. + @Published var onFieldReverted: ((Int, [RowID: PluginCellValue], Set) -> Void)? + + /// Remove Field on a field, for the rows the selection was configured with. + @Published var onFieldRemoved: ((Int, [RowID]) -> Void)? /// A value window still open over a selection that has moved on. It names the rows it was /// opened for, because the fields it was opened from are gone. @@ -101,6 +123,8 @@ final class MultiRowEditState: ObservableObject { @Published private(set) var rowIDs: [RowID] = [] @Published private(set) var allRows: [[String?]] = [] + /// The columns each row in `allRows` has no field for. + @Published private(set) var absentCells: [Set] = [] @Published private(set) var columns: [String] = [] @Published private(set) var columnTypes: [ColumnType] = [] @@ -113,6 +137,7 @@ final class MultiRowEditState: ObservableObject { selectedRowIndices: Set, rowIDs: [RowID] = [], allRows: [[String?]], + absentCells: [Set] = [], columns: [String], columnTypes: [ColumnType], externallyModifiedColumns: Set = [], @@ -128,6 +153,7 @@ final class MultiRowEditState: ObservableObject { self.selectedRowIndices = selectedRowIndices self.rowIDs = rowIDs self.allRows = allRows + self.absentCells = absentCells self.columns = columns self.columnTypes = columnTypes @@ -142,9 +168,12 @@ final class MultiRowEditState: ObservableObject { let value = colIndex < row.count ? row[colIndex] : nil values.append(value) } + let absence = allRows.indices.map { absentCells[safe: $0]?.contains(colIndex) == true } let allSame = values.dropFirst().allSatisfy { $0 == values.first } + && absence.dropFirst().allSatisfy { $0 == absence.first } let hasMultipleValues = !allSame + let isAbsent = !hasMultipleValues && absence.first == true let originalValue: String? if hasMultipleValues { @@ -158,21 +187,30 @@ final class MultiRowEditState: ObservableObject { var pendingValue: String? var isPendingNull = false var isPendingDefault = false + var isPendingRemoval = false if !columnsChanged, !selectionChanged, colIndex < fields.count { let oldField = fields[colIndex] // Preserve pending edits when original data matches - if oldField.originalValue == originalValue && oldField.hasMultipleValues == hasMultipleValues { + if oldField.originalValue == originalValue && oldField.hasMultipleValues == hasMultipleValues + && oldField.isAbsent == isAbsent { preservedId = oldField.id pendingValue = oldField.pendingValue isPendingNull = oldField.isPendingNull isPendingDefault = oldField.isPendingDefault + isPendingRemoval = oldField.isPendingRemoval } } - // Mark externally modified columns (e.g., edited in data grid) - if externallyModifiedColumns.contains(colIndex), pendingValue == nil, !isPendingNull, !isPendingDefault { - pendingValue = originalValue ?? "" + // Mark externally modified columns (e.g., edited in data grid). A modified field that + // is now missing is one an edit removed. + if externallyModifiedColumns.contains(colIndex), pendingValue == nil, !isPendingNull, !isPendingDefault, + !isPendingRemoval { + if isAbsent { + isPendingRemoval = true + } else { + pendingValue = originalValue ?? "" + } } let isJson = columnTypeEnum.isJsonType || (originalValue ?? "").looksLikeJson @@ -190,7 +228,9 @@ final class MultiRowEditState: ObservableObject { hasMultipleValues: hasMultipleValues, pendingValue: pendingValue, isPendingNull: isPendingNull, - isPendingDefault: isPendingDefault + isPendingDefault: isPendingDefault, + isAbsent: isAbsent, + isPendingRemoval: isPendingRemoval ) if let preservedId { newField.id = preservedId @@ -220,6 +260,7 @@ final class MultiRowEditState: ObservableObject { selectedRowIndices = [displayRow] rowIDs = [] + absentCells = [] columns = names columnTypes = Array(repeating: .text(rawType: nil), count: names.count) allRows = [schemaFields.map(\.value)] @@ -271,15 +312,17 @@ final class MultiRowEditState: ObservableObject { fields[index].pendingValue = pending fields[index].isPendingNull = false fields[index].isPendingDefault = false + fields[index].isPendingRemoval = false if pending != nil { onFieldChanged?(index, PluginCellValue.fromOptional(pending), .typing) } else if hadPendingEdit { - /// `originalValue` is nil for two different situations, and only one of them is a - /// value: a stored NULL, and a selection whose rows do not agree. Sending it as one - /// value wrote NULL into every selected row when the user cleared a field they all - /// disagreed on, which the field then reported as unedited. - if fields[index].hasMultipleValues { - onFieldReverted?(index, configuredValues(atColumn: index)) + /// `originalValue` is nil for three different situations, and only one of them is a + /// value: a stored NULL, a selection whose rows do not agree, and a field the rows do + /// not have. Sending it as one value wrote NULL into every selected row when the user + /// cleared a field they all disagreed on, which the field then reported as unedited. + let absentRows = configuredAbsentRows(atColumn: index) + if fields[index].hasMultipleValues || !absentRows.isEmpty { + onFieldReverted?(index, configuredValues(atColumn: index), absentRows) } else { onFieldChanged?(index, PluginCellValue.fromOptional(original), .typing) } @@ -296,6 +339,10 @@ final class MultiRowEditState: ObservableObject { return values } + private func configuredAbsentRows(atColumn index: Int) -> Set { + Set(rowIDs.indices.filter { absentCells[safe: $0]?.contains(index) == true }.map { rowIDs[$0] }) + } + /// A commit from a detached value window, which outlives the selection it was opened from. /// /// While that selection is still the one on screen this is an ordinary field edit. Once it has @@ -330,6 +377,7 @@ final class MultiRowEditState: ObservableObject { fields[index].pendingValue = encoded fields[index].isPendingNull = false fields[index].isPendingDefault = false + fields[index].isPendingRemoval = false onFieldChanged?(index, .bytes(data), .discrete) } @@ -338,14 +386,30 @@ final class MultiRowEditState: ObservableObject { fields[index].pendingValue = nil fields[index].isPendingNull = true fields[index].isPendingDefault = false + fields[index].isPendingRemoval = false onFieldChanged?(index, .null, .discrete) } + /// Takes the field out of every selected row. The value is gone as well, so it reads as + /// neither NULL nor any value. + /// + /// Rows that had no field for it when the selection was configured are back as they were, + /// which leaves nothing pending, exactly as typing a field's own value back does. + func removeField(at index: Int) { + guard index < fields.count, !rowIDs.isEmpty else { return } + fields[index].pendingValue = nil + fields[index].isPendingNull = false + fields[index].isPendingDefault = false + fields[index].isPendingRemoval = !fields[index].isAbsent + onFieldRemoved?(index, rowIDs) + } + func setFieldToDefault(at index: Int) { guard index < fields.count else { return } fields[index].pendingValue = nil fields[index].isPendingNull = false fields[index].isPendingDefault = true + fields[index].isPendingRemoval = false onFieldChanged?(index, .text("__DEFAULT__"), .discrete) } @@ -354,6 +418,7 @@ final class MultiRowEditState: ObservableObject { fields[index].pendingValue = function fields[index].isPendingNull = false fields[index].isPendingDefault = false + fields[index].isPendingRemoval = false onFieldChanged?(index, .text(function), .discrete) } @@ -367,6 +432,7 @@ final class MultiRowEditState: ObservableObject { } fields[index].isPendingNull = false fields[index].isPendingDefault = false + fields[index].isPendingRemoval = false if fields[index].pendingValue != nil || hadPendingEdit { onFieldChanged?(index, .text(""), .discrete) } @@ -378,6 +444,7 @@ final class MultiRowEditState: ObservableObject { fields[i].pendingValue = nil fields[i].isPendingNull = false fields[i].isPendingDefault = false + fields[i].isPendingRemoval = false } } @@ -386,19 +453,26 @@ final class MultiRowEditState: ObservableObject { fields = [] onFieldChanged = nil onFieldReverted = nil + onFieldRemoved = nil onDetachedFieldChanged = nil selectedRowIndices = [] rowIDs = [] allRows = [] + absentCells = [] columns = [] columnTypes = [] } /// Get all edited fields with their new values - func getEditedFields() -> [(columnIndex: Int, columnName: String, newValue: String?)] { + func getEditedFields() -> [InspectorFieldEdit] { fields.compactMap { field in guard field.hasEdit else { return nil } - return (field.columnIndex, field.columnName, field.effectiveValue) + return InspectorFieldEdit( + columnIndex: field.columnIndex, + columnName: field.columnName, + newValue: field.effectiveValue, + removesField: field.isPendingRemoval + ) } } } diff --git a/TablePro/Resources/Localizable.xcstrings b/TablePro/Resources/Localizable.xcstrings index ed906a32e1..3b7b621393 100644 --- a/TablePro/Resources/Localizable.xcstrings +++ b/TablePro/Resources/Localizable.xcstrings @@ -184481,16 +184481,22 @@ "This row's _id is binary data whose subtype is not known, so TablePro cannot tell which document it names." : { }, - "The MongoDB client library will not insert a field whose name is empty. Set it to NULL, then set it on the saved document." : { + "%@ holds an empty key, which the MongoDB client library will not insert. Set it to NULL, then set it on the saved document." : { }, - "%@ holds an empty key, which the MongoDB client library will not insert. Set it to NULL, then set it on the saved document." : { + "%@ holds text in some rows and documents or arrays in others, so this value could be either. Change it with a query." : { }, - "The shell drops a field named __proto__ from a new document. Set it to NULL, then set it on the saved document." : { + "No Field" : { }, - "%@ holds text in some rows and documents or arrays in others, so this value could be either. Change it with a query." : { + "Remove Field" : { + + }, + "The MongoDB client library will not insert a field whose name is empty. Remove the field, then set it on the saved document." : { + + }, + "The shell drops a field named __proto__ from a new document. Remove the field, then set it on the saved document." : { } }, diff --git a/TablePro/Views/Main/Extensions/MainContentCoordinator+QueryHelpers.swift b/TablePro/Views/Main/Extensions/MainContentCoordinator+QueryHelpers.swift index 3825099717..f1ead3409a 100644 --- a/TablePro/Views/Main/Extensions/MainContentCoordinator+QueryHelpers.swift +++ b/TablePro/Views/Main/Extensions/MainContentCoordinator+QueryHelpers.swift @@ -130,7 +130,8 @@ extension MainContentCoordinator { anchor: StatementAnchor? = nil, timing: PluginQueryTiming? = nil, viewport: GridReloadIntent = .firstRow, - serverOutput: PluginServerOutput = .none + serverOutput: PluginServerOutput = .none, + absentCells: [Int: Set] = [:] ) { queryExecutionCoordinator.applyPhase1Result( tabId: tabId, @@ -152,7 +153,8 @@ extension MainContentCoordinator { anchor: anchor, timing: timing, viewport: viewport, - serverOutput: serverOutput + serverOutput: serverOutput, + absentCells: absentCells ) } diff --git a/TablePro/Views/Main/Extensions/MainContentCoordinator+RowOperations.swift b/TablePro/Views/Main/Extensions/MainContentCoordinator+RowOperations.swift index 9f51b6510a..2f1adc30d5 100644 --- a/TablePro/Views/Main/Extensions/MainContentCoordinator+RowOperations.swift +++ b/TablePro/Views/Main/Extensions/MainContentCoordinator+RowOperations.swift @@ -57,8 +57,18 @@ extension MainContentCoordinator { rowEditingCoordinator.endInspectorEditRun() } - func revertInspectorFieldEdit(columnIndex: Int, valuesByRow: [RowID: PluginCellValue]) { - rowEditingCoordinator.revertInspectorFieldEdit(columnIndex: columnIndex, valuesByRow: valuesByRow) + func revertInspectorFieldEdit( + columnIndex: Int, + valuesByRow: [RowID: PluginCellValue], + absentRowIDs: Set = [] + ) { + rowEditingCoordinator.revertInspectorFieldEdit( + columnIndex: columnIndex, valuesByRow: valuesByRow, absentRowIDs: absentRowIDs + ) + } + + func stageInspectorFieldRemoval(columnIndex: Int, rowIDs: [RowID]) { + rowEditingCoordinator.stageInspectorFieldRemoval(columnIndex: columnIndex, rowIDs: rowIDs) } func copySelectedRowsToClipboard(indices: Set) { diff --git a/TablePro/Views/Main/Extensions/MainContentCoordinator+SidebarSave.swift b/TablePro/Views/Main/Extensions/MainContentCoordinator+SidebarSave.swift index 72ca2f3851..723b04e954 100644 --- a/TablePro/Views/Main/Extensions/MainContentCoordinator+SidebarSave.swift +++ b/TablePro/Views/Main/Extensions/MainContentCoordinator+SidebarSave.swift @@ -21,8 +21,10 @@ extension MainContentCoordinator { runQuery(viewport: .keepPlace) } + /// Each row keeps which fields it has none of, so a removal is written as one and a value + /// typed into a missing field is compared against a field that is not there. func sidebarEditStatements( - editedFields: [(columnIndex: Int, columnName: String, newValue: String?)] + editedFields: [InspectorFieldEdit] ) throws -> [ParameterizedStatement] { guard let tab = tabManager.selectedTab, !selectionState.indices.isEmpty, @@ -57,10 +59,13 @@ extension MainContentCoordinator { columnIndex: field.columnIndex, columnName: field.columnName, oldValue: oldValue, - newValue: PluginCellValue.fromOptional(field.newValue) + newValue: field.removesField ? .null : PluginCellValue.fromOptional(field.newValue), + oldIsAbsent: resolvedRow.isAbsent(field.columnIndex), + newIsAbsent: field.removesField ) }, - originalRow: originalRow + originalRow: originalRow, + absentColumns: resolvedRow.absentColumns ) } diff --git a/TablePro/Views/Main/Extensions/MainContentView+EventHandlers.swift b/TablePro/Views/Main/Extensions/MainContentView+EventHandlers.swift index 327878e390..6b907f53c2 100644 --- a/TablePro/Views/Main/Extensions/MainContentView+EventHandlers.swift +++ b/TablePro/Views/Main/Extensions/MainContentView+EventHandlers.swift @@ -210,12 +210,13 @@ extension MainContentView { selectedRowIndices: selectedIndices, rowIDs: selectedRows.map(\.id), allRows: stringRows, + absentCells: selectedRows.map(\.absentColumns), columns: tableRows.columns, columnTypes: columnTypes, externallyModifiedColumns: modifiedColumns, primaryKeyColumns: pkColumns, foreignKeyColumns: fkColumns, - serverOwnedColumns: tableRows.generatedColumns, + serverOwnedColumns: tableRows.generatedColumns.union(changeManager.unwritableColumns(among: tableRows.columns)), displayFormats: inspectorDisplayFormats(for: tab, columns: tableRows.columns, types: tableRows.columnTypes) ) @@ -234,8 +235,13 @@ extension MainContentView { continuity: continuity ) } - trailingPaneState.inspector.editState.onFieldReverted = { columnIndex, valuesByRow in - capturedCoordinator.revertInspectorFieldEdit(columnIndex: columnIndex, valuesByRow: valuesByRow) + trailingPaneState.inspector.editState.onFieldReverted = { columnIndex, valuesByRow, absentRowIDs in + capturedCoordinator.revertInspectorFieldEdit( + columnIndex: columnIndex, valuesByRow: valuesByRow, absentRowIDs: absentRowIDs + ) + } + trailingPaneState.inspector.editState.onFieldRemoved = { columnIndex, rowIDs in + capturedCoordinator.stageInspectorFieldRemoval(columnIndex: columnIndex, rowIDs: rowIDs) } trailingPaneState.inspector.editState.onDetachedFieldChanged = { columnIndex, newValue, rowIDs in /// A value window commits on every keystroke exactly as the field it detached from @@ -295,6 +301,7 @@ extension MainContentView { private func clearSidebarEditHandlers() { trailingPaneState.inspector.editState.onFieldChanged = nil trailingPaneState.inspector.editState.onFieldReverted = nil + trailingPaneState.inspector.editState.onFieldRemoved = nil trailingPaneState.inspector.editState.onDetachedFieldChanged = nil } @@ -318,6 +325,7 @@ extension MainContentView { let capturedCoordinator = coordinator trailingPaneState.inspector.editState.onFieldReverted = nil + trailingPaneState.inspector.editState.onFieldRemoved = nil trailingPaneState.inspector.editState.onDetachedFieldChanged = nil trailingPaneState.inspector.editState.onFieldChanged = { fieldIndex, newValue, _ in capturedCoordinator.inspectorRowSource?.commitInspectorField( diff --git a/TablePro/Views/Main/MainContentCoordinator.swift b/TablePro/Views/Main/MainContentCoordinator.swift index 7314d6977b..67dec3313a 100644 --- a/TablePro/Views/Main/MainContentCoordinator.swift +++ b/TablePro/Views/Main/MainContentCoordinator.swift @@ -1408,7 +1408,8 @@ final class MainContentCoordinator: ObservableObject { anchor: anchor, timing: fetchResult.resolvedTiming, viewport: viewport, - serverOutput: fetchResult.serverOutput + serverOutput: fetchResult.serverOutput, + absentCells: fetchResult.absentCells ) scheduleTraceCompletion(traceToken, outcome: .completed) diff --git a/TablePro/Views/Results/Cells/DataGridCellAppearance.swift b/TablePro/Views/Results/Cells/DataGridCellAppearance.swift index f3b63769bc..67f173144c 100644 --- a/TablePro/Views/Results/Cells/DataGridCellAppearance.swift +++ b/TablePro/Views/Results/Cells/DataGridCellAppearance.swift @@ -48,7 +48,7 @@ struct DataGridCellAppearance: Equatable { case .none: font = palette.regularFont baseColor = deletedTextColor ?? valueTextColor(kind: kind, rawValue: content.rawValue, palette: palette) - case .null, .empty: + case .null, .empty, .absent: font = palette.italicFont baseColor = deletedTextColor ?? palette.placeholderText case .defaultMarker: diff --git a/TablePro/Views/Results/Cells/DataGridCellContent.swift b/TablePro/Views/Results/Cells/DataGridCellContent.swift index 47be2fa1ad..f61ae1cc24 100644 --- a/TablePro/Views/Results/Cells/DataGridCellContent.swift +++ b/TablePro/Views/Results/Cells/DataGridCellContent.swift @@ -10,6 +10,8 @@ enum DataGridCellPlaceholder: Equatable { case null case empty case defaultMarker + /// The row has no field for the column, which a document store tells apart from NULL. + case absent } struct DataGridCellContent { @@ -17,7 +19,8 @@ struct DataGridCellContent { let rawValue: String? let placeholder: DataGridCellPlaceholder? - static func placeholder(for rawValue: PluginCellValue) -> DataGridCellPlaceholder? { + static func placeholder(for rawValue: PluginCellValue, isAbsent: Bool = false) -> DataGridCellPlaceholder? { + guard !isAbsent else { return .absent } switch rawValue { case .null: return .null @@ -44,8 +47,14 @@ struct DataGridCellContent { return isLargeDataset ? "" : String(localized: "Empty") case .defaultMarker: return isLargeDataset ? "" : String(localized: "DEFAULT") + case .absent: + return isLargeDataset ? "" : absentFieldText } } + + static var absentFieldText: String { + String(localized: "No Field") + } } struct DataGridCellState { diff --git a/TablePro/Views/Results/DataGridCellFactory.swift b/TablePro/Views/Results/DataGridCellFactory.swift index fb7af408d7..2628affa55 100644 --- a/TablePro/Views/Results/DataGridCellFactory.swift +++ b/TablePro/Views/Results/DataGridCellFactory.swift @@ -140,7 +140,9 @@ final class DataGridCellFactory { ) ?? "" let value = DataGridCellContent.resolvedDisplayText( formattedValue, - placeholder: DataGridCellContent.placeholder(for: rawValue), + placeholder: DataGridCellContent.placeholder( + for: rawValue, isAbsent: tableRows.isAbsent(row: i, column: columnIndex) + ), isLargeDataset: isLargeDataset, nullDisplayString: resolvedNullDisplayString ) diff --git a/TablePro/Views/Results/DataGridRowView.swift b/TablePro/Views/Results/DataGridRowView.swift index 6774eb0f0a..cf510b43bf 100644 --- a/TablePro/Views/Results/DataGridRowView.swift +++ b/TablePro/Views/Results/DataGridRowView.swift @@ -525,13 +525,7 @@ class DataGridRowView: NSTableRowView { menu.addItem(NSMenuItem.separator()) } - let namesWritableColumn = dataColumnIndex >= 0 && dataColumnIndex < tableRows.columns.count - && coordinator.isColumnWritable(tableRows.columns[dataColumnIndex]) - if coordinator.isEditable && namesWritableColumn { - let setValueItem = NSMenuItem(title: String(localized: "Set Value"), action: nil, keyEquivalent: "") - setValueItem.submenu = buildSetValueMenu(dataColumnIndex: dataColumnIndex, tableRows: tableRows) - menu.addItem(setValueItem) - } + addValueEditingItems(to: menu, dataColumnIndex: dataColumnIndex, tableRows: tableRows, coordinator: coordinator) menu.addItem(NSMenuItem.separator()) @@ -607,6 +601,31 @@ class DataGridRowView: NSTableRowView { } } + /// Set Value, and Remove Field on an engine that tells a missing field from NULL while the cell + /// still has a field to remove. + private func addValueEditingItems( + to menu: NSMenu, + dataColumnIndex: Int, + tableRows: TableRows, + coordinator: TableViewCoordinator + ) { + let namesWritableColumn = dataColumnIndex >= 0 && dataColumnIndex < tableRows.columns.count + && coordinator.isColumnWritable(tableRows.columns[dataColumnIndex]) + guard coordinator.isEditable, namesWritableColumn else { return } + + let setValueItem = NSMenuItem(title: String(localized: "Set Value"), action: nil, keyEquivalent: "") + setValueItem.submenu = buildSetValueMenu(dataColumnIndex: dataColumnIndex, tableRows: tableRows) + menu.addItem(setValueItem) + + guard coordinator.supportsFieldRemoval, + coordinator.displayRow(at: rowIndex)?.isAbsent(dataColumnIndex) == false else { return } + let removeFieldItem = NSMenuItem( + title: String(localized: "Remove Field"), action: #selector(removeFieldValue(_:)), keyEquivalent: "") + removeFieldItem.representedObject = dataColumnIndex + removeFieldItem.target = self + menu.addItem(removeFieldItem) + } + private func buildSetValueMenu(dataColumnIndex: Int, tableRows: TableRows) -> NSMenu { let setValueMenu = NSMenu() @@ -718,6 +737,11 @@ class DataGridRowView: NSTableRowView { coordinator?.setCellValueAtColumn(nil, at: rowIndex, columnIndex: columnIndex) } + @objc private func removeFieldValue(_ sender: NSMenuItem) { + guard let columnIndex = sender.representedObject as? Int else { return } + coordinator?.removeField(row: rowIndex, columnIndex: columnIndex) + } + @objc private func setEmptyValue(_ sender: NSMenuItem) { guard let columnIndex = sender.representedObject as? Int else { return } coordinator?.setCellValueAtColumn("", at: rowIndex, columnIndex: columnIndex) diff --git a/TablePro/Views/Results/DataGridView+RowActions.swift b/TablePro/Views/Results/DataGridView+RowActions.swift index 42156c0f4b..ab480ccd17 100644 --- a/TablePro/Views/Results/DataGridView+RowActions.swift +++ b/TablePro/Views/Results/DataGridView+RowActions.swift @@ -25,31 +25,6 @@ extension TableViewCoordinator { delegate?.dataGridAddRow() } - func copyRows(at indices: Set) { - let sortedIndices = indices.sorted() - let tableRows = tableRowsProvider() - let projection = visibleColumnProjection - let columnTypes = projection.columnTypes(tableRows.columnTypes) - let columns = projection.columns(tableRows.columns) - var tsvRows: [String] = [] - var htmlRows: [[String]] = [] - var structuredRows: [[PluginCellValue]] = [] - - for index in sortedIndices { - guard let values = displayRow(at: index)?.values else { continue } - let projected = projection.values(Array(values)) - let formatted = formatRowValues(values: projected, columnTypes: columnTypes) - tsvRows.append(formatted.joined(separator: "\t")) - htmlRows.append(formatted) - structuredRows.append(projected) - } - - let tsv = tsvRows.joined(separator: "\n") - let html = HtmlTableEncoder.encode(rows: htmlRows) - let payload = GridRowsClipboardPayload(columns: columns, rows: structuredRows) - ClipboardService.shared.writeRows(tsv: tsv, html: html, gridRows: payload) - } - func copyRowsWithHeaders(at indices: Set) { let sortedIndices = indices.sorted() let tableRows = tableRowsProvider() @@ -58,20 +33,19 @@ extension TableViewCoordinator { let columns = projection.columns(tableRows.columns) var tsvRows: [String] = [columns.joined(separator: "\t")] var htmlRows: [[String]] = [] - var structuredRows: [[PluginCellValue]] = [] + var copiedRows: [Row] = [] for index in sortedIndices { - guard let values = displayRow(at: index)?.values else { continue } - let projected = projection.values(Array(values)) - let formatted = formatRowValues(values: projected, columnTypes: columnTypes) + guard let row = displayRow(at: index) else { continue } + let formatted = formatRowValues(values: projection.values(Array(row.values)), columnTypes: columnTypes) tsvRows.append(formatted.joined(separator: "\t")) htmlRows.append(formatted) - structuredRows.append(projected) + copiedRows.append(row) } let tsv = tsvRows.joined(separator: "\n") let html = HtmlTableEncoder.encode(rows: htmlRows, headers: columns) - let payload = GridRowsClipboardPayload(columns: columns, rows: structuredRows) + let payload = GridRowsClipboardPayload(columns: columns, copying: copiedRows, projection: projection) ClipboardService.shared.writeRows(tsv: tsv, html: html, gridRows: payload) } diff --git a/TablePro/Views/Results/Extensions/DataGridView+CellCommit.swift b/TablePro/Views/Results/Extensions/DataGridView+CellCommit.swift index a4df2f94d9..8a54ac28a0 100644 --- a/TablePro/Views/Results/Extensions/DataGridView+CellCommit.swift +++ b/TablePro/Views/Results/Extensions/DataGridView+CellCommit.swift @@ -16,7 +16,18 @@ extension TableViewCoordinator { func commitTypedCellEdit(row: Int, columnIndex: Int, newValue typedNewValue: PluginCellValue) { guard recordCellEdit(row: row, columnIndex: columnIndex, newValue: typedNewValue) != nil else { return } + repaintEditedCell(row: row, columnIndex: columnIndex) + } + + /// Takes the field out of the row, for an engine that tells a missing field from NULL. + func removeField(row: Int, columnIndex: Int) { + guard supportsFieldRemoval, + recordCellEdit(row: row, columnIndex: columnIndex, newValue: .null, removesField: true) != nil + else { return } + repaintEditedCell(row: row, columnIndex: columnIndex) + } + private func repaintEditedCell(row: Int, columnIndex: Int) { invalidateDisplayCache() updateVisualIndex(forDisplayRow: row) @@ -56,16 +67,21 @@ extension TableViewCoordinator { let tableRows = tableRowsProvider() guard let storageRow = tableRows.index(of: rowID), storageRow < tableRows.rows.count else { return nil } return recordCellEdit( - rowID: rowID, - values: Array(tableRows.rows[storageRow].values), + in: tableRows.rows[storageRow], columnIndex: columnIndex, newValue: typedNewValue, + removesField: false, displayRow: DisplayRowMapping.displayIndex(forRowID: rowID, displayIDs: displayIDs, in: tableRows) ) } @discardableResult - func recordCellEdit(row: Int, columnIndex: Int, newValue typedNewValue: PluginCellValue) -> Delta? { + func recordCellEdit( + row: Int, + columnIndex: Int, + newValue typedNewValue: PluginCellValue, + removesField: Bool = false + ) -> Delta? { cellCommitLogger.debug("recordCellEdit(row: \(row, privacy: .public), columnIndex: \(columnIndex, privacy: .public)) isCommitting=\(self.isCommittingCellEdit, privacy: .public) delegate=\(self.delegate == nil ? "nil" : "present", privacy: .public)") guard !isCommittingCellEdit else { return nil } let tableRows = tableRowsProvider() @@ -76,31 +92,39 @@ extension TableViewCoordinator { guard isColumnWritable(tableRows.columns[columnIndex]) else { return nil } guard let displayRowValues = displayRow(at: row) else { return nil } return recordCellEdit( - rowID: displayRowValues.id, - values: Array(displayRowValues.values), + in: displayRowValues, columnIndex: columnIndex, newValue: typedNewValue, + removesField: removesField, displayRow: row ) } /// The one place a cell edit becomes a pending change and a new value in the shared row buffer. /// `displayRow` is only what the delegate is told; the record and the write are by identity. + /// + /// A value written into a missing field brings the field back, and `removesField` takes it out, + /// so NULL over a missing field is an edit even though both read `.null`. @discardableResult private func recordCellEdit( - rowID: RowID, - values: [PluginCellValue], + in row: Row, columnIndex: Int, newValue typedNewValue: PluginCellValue, + removesField: Bool, displayRow: Int? ) -> Delta? { guard !isCommittingCellEdit else { return nil } let tableRows = tableRowsProvider() guard columnIndex >= 0, columnIndex < tableRows.columns.count else { return nil } guard isColumnWritable(tableRows.columns[columnIndex]) else { return nil } - guard columnIndex < values.count else { return nil } - let oldValue = values[columnIndex] - guard oldValue != typedNewValue else { + guard columnIndex < row.values.count else { return nil } + let oldValue = row.values[columnIndex] + let absence = FieldAbsence( + wasAbsent: row.isAbsent(columnIndex), + isAbsent: removesField, + originalRow: row.absentColumns + ) + guard oldValue != typedNewValue || absence.wasAbsent != absence.isAbsent else { cellCommitLogger.debug("recordCellEdit - value unchanged, guard returned") return nil } @@ -108,19 +132,21 @@ extension TableViewCoordinator { isCommittingCellEdit = true defer { isCommittingCellEdit = false } + let rowID = row.id changeManager.recordCellChange( rowID: rowID, columnIndex: columnIndex, columnName: tableRows.columns[columnIndex], oldValue: oldValue, newValue: typedNewValue, - originalRow: values + originalRow: Array(row.values), + absence: absence ) var delta: Delta = .none if let storageRow = tableRows.index(of: rowID) { delta = tableRowsMutator { tableRows in - tableRows.edit(row: storageRow, column: columnIndex, value: typedNewValue) + tableRows.edit(row: storageRow, column: columnIndex, value: typedNewValue, isAbsent: removesField) } } /// A record the display order is hiding has no row to report, and every delegate reads that diff --git a/TablePro/Views/Results/Extensions/DataGridView+Columns.swift b/TablePro/Views/Results/Extensions/DataGridView+Columns.swift index 3fe119a492..9af1e3b37e 100644 --- a/TablePro/Views/Results/Extensions/DataGridView+Columns.swift +++ b/TablePro/Views/Results/Extensions/DataGridView+Columns.swift @@ -89,7 +89,7 @@ extension TableViewCoordinator { content: DataGridCellContent( displayText: formattedValue ?? "", rawValue: rawValue.asText, - placeholder: DataGridCellContent.placeholder(for: rawValue) + placeholder: DataGridCellContent.placeholder(for: rawValue, isAbsent: displayRow.isAbsent(columnIndex)) ), state: DataGridCellState( visualState: visualState(of: displayRow, atDisplayRow: row), @@ -124,10 +124,11 @@ extension TableViewCoordinator { ) ?? "" guard text.isEmpty else { return text } - switch DataGridCellContent.placeholder(for: rawValue) { + switch DataGridCellContent.placeholder(for: rawValue, isAbsent: displayRow.isAbsent(columnIndex)) { case .null: return String(localized: "NULL") case .empty: return String(localized: "Empty") case .defaultMarker: return String(localized: "DEFAULT") + case .absent: return DataGridCellContent.absentFieldText case .none: return text } } diff --git a/TablePro/Views/Results/Extensions/DataGridView+Editing.swift b/TablePro/Views/Results/Extensions/DataGridView+Editing.swift index fb8266e6d8..90d3d9a091 100644 --- a/TablePro/Views/Results/Extensions/DataGridView+Editing.swift +++ b/TablePro/Views/Results/Extensions/DataGridView+Editing.swift @@ -53,6 +53,12 @@ extension TableViewCoordinator { return !immutable.contains(columnName) } + /// Whether a row here can lack a field, apart from holding NULL, which is what offers Remove + /// Field. Only a data grid's rows can; a structure grid edits definitions. + var supportsFieldRemoval: Bool { + changeManager.supportsFieldRemoval + } + func canStartInlineEdit(row: Int, columnIndex: Int) -> Bool { if case .editable = editEligibility(row: row, columnIndex: columnIndex) { return true diff --git a/TablePro/Views/RowInspector/FieldEditors/BooleanPickerView.swift b/TablePro/Views/RowInspector/FieldEditors/BooleanPickerView.swift index 5d36632f3d..ec7100e7ec 100644 --- a/TablePro/Views/RowInspector/FieldEditors/BooleanPickerView.swift +++ b/TablePro/Views/RowInspector/FieldEditors/BooleanPickerView.swift @@ -38,11 +38,8 @@ internal struct BooleanPickerView: View { ?? Self.normalized(context.valueState.editableText) }, set: { newValue in - switch newValue { - case FieldPickerSentinel.null: onSetNull?() - case FieldPickerSentinel.defaultValue: onSetDefault?() - case FieldPickerSentinel.multiple: break - default: context.value.wrappedValue = newValue + FieldPickerSentinel.choose(newValue, onSetNull: onSetNull, onSetDefault: onSetDefault) { + context.value.wrappedValue = $0 } } ) diff --git a/TablePro/Views/RowInspector/FieldEditors/EnumPickerView.swift b/TablePro/Views/RowInspector/FieldEditors/EnumPickerView.swift index ef3e320cfd..0949739b12 100644 --- a/TablePro/Views/RowInspector/FieldEditors/EnumPickerView.swift +++ b/TablePro/Views/RowInspector/FieldEditors/EnumPickerView.swift @@ -36,11 +36,8 @@ internal struct EnumPickerView: View { Binding( get: { FieldPickerSentinel.tag(for: context.valueState) ?? context.valueState.editableText }, set: { newValue in - switch newValue { - case FieldPickerSentinel.null: onSetNull?() - case FieldPickerSentinel.defaultValue: onSetDefault?() - case FieldPickerSentinel.multiple: break - default: context.value.wrappedValue = newValue + FieldPickerSentinel.choose(newValue, onSetNull: onSetNull, onSetDefault: onSetDefault) { + context.value.wrappedValue = $0 } } ) diff --git a/TablePro/Views/RowInspector/FieldEditors/FieldMenuContent.swift b/TablePro/Views/RowInspector/FieldEditors/FieldMenuContent.swift index 9e20c24b8a..27622d078c 100644 --- a/TablePro/Views/RowInspector/FieldEditors/FieldMenuContent.swift +++ b/TablePro/Views/RowInspector/FieldEditors/FieldMenuContent.swift @@ -17,10 +17,14 @@ internal struct FieldMenuContent: View { let canMutate: Bool let isPendingNull: Bool let isPendingDefault: Bool + var isPendingRemoval = false let onSetNull: () -> Void let onSetDefault: () -> Void let onSetEmpty: () -> Void let onSetFunction: (String) -> Void + /// Offered only on an engine that tells a missing field from NULL, and only while the field is + /// there to remove. + var onRemoveField: (() -> Void)? let onClear: () -> Void var body: some View { @@ -28,6 +32,9 @@ internal struct FieldMenuContent: View { Button("Set NULL") { onSetNull() } Button("Set DEFAULT") { onSetDefault() } Button("Set EMPTY") { onSetEmpty() } + if let onRemoveField { + Button("Remove Field") { onRemoveField() } + } Divider() } @@ -61,7 +68,7 @@ internal struct FieldMenuContent: View { } } - if isPendingNull || isPendingDefault { + if isPendingNull || isPendingDefault || isPendingRemoval { Divider() Button("Clear") { onClear() } } diff --git a/TablePro/Views/RowInspector/FieldEditors/FieldPickerSentinel.swift b/TablePro/Views/RowInspector/FieldEditors/FieldPickerSentinel.swift index b2c5f420b1..239bad6896 100644 --- a/TablePro/Views/RowInspector/FieldEditors/FieldPickerSentinel.swift +++ b/TablePro/Views/RowInspector/FieldEditors/FieldPickerSentinel.swift @@ -15,6 +15,7 @@ internal enum FieldPickerSentinel { internal static let null = "\u{FFFE}NULL" internal static let defaultValue = "\u{FFFE}DEFAULT" internal static let multiple = "\u{FFFE}MULTIPLE" + internal static let absent = "\u{FFFE}ABSENT" /// The tag that stands for a state, or nil where the state is an ordinary value the column /// already offers. @@ -23,12 +24,31 @@ internal enum FieldPickerSentinel { case .null, .pendingNull: return null case .pendingDefault: return defaultValue case .multipleValues: return multiple + case .absent, .pendingRemoval: return absent case .value: return nil } } internal static func isSentinel(_ tag: String) -> Bool { - tag == null || tag == defaultValue || tag == multiple + tag == null || tag == defaultValue || tag == multiple || tag == absent + } + + /// What choosing a tag does, for every picker. A row that only says where the field stands, + /// the rows disagreeing or the field missing, changes nothing when it is chosen again, so its + /// tag never reaches the field as a value. + internal static func choose( + _ tag: String, + onSetNull: (() -> Void)?, + onSetDefault: (() -> Void)?, + onValue: (String) -> Void + ) { + switch tag { + case null: onSetNull?() + case defaultValue: onSetDefault?() + default: + guard !isSentinel(tag) else { return } + onValue(tag) + } } } diff --git a/TablePro/Views/RowInspector/InspectorFieldListView.swift b/TablePro/Views/RowInspector/InspectorFieldListView.swift index 5f8eb38a94..f48bb2fe6d 100644 --- a/TablePro/Views/RowInspector/InspectorFieldListView.swift +++ b/TablePro/Views/RowInspector/InspectorFieldListView.swift @@ -17,6 +17,10 @@ internal struct InspectorFieldListView: View { internal let databaseType: DatabaseType internal let userDefinedTypeScope: DatabaseScope? internal var offersDatabaseValues = true + + private var offersFieldRemoval: Bool { + offersDatabaseValues && PluginManager.shared.supportsFieldRemoval(for: databaseType) + } internal var onPopOut: ((FieldEditState, String, FieldEditorKind) -> Void)? @State private var searchText = "" @@ -153,6 +157,9 @@ internal struct InspectorFieldListView: View { onSetDefault: { editState.setFieldToDefault(at: field.columnIndex) }, onSetEmpty: { editState.setFieldToEmpty(at: field.columnIndex) }, onSetFunction: { editState.setFieldToFunction(at: field.columnIndex, function: $0) }, + onRemoveField: offersFieldRemoval && !field.isSchemaField + ? { editState.removeField(at: field.columnIndex) } + : nil, onToggleExpand: FieldEditorContent.canExpand(kind: kind, state: FieldValueState.resolve(field)) ? { expandedFieldID = expandedFieldID == field.id ? nil : field.id } : nil, diff --git a/TablePro/Views/RowInspector/InspectorFieldRow.swift b/TablePro/Views/RowInspector/InspectorFieldRow.swift index af27eda1b3..5bd66c5446 100644 --- a/TablePro/Views/RowInspector/InspectorFieldRow.swift +++ b/TablePro/Views/RowInspector/InspectorFieldRow.swift @@ -26,6 +26,7 @@ internal struct InspectorFieldRow: View { internal let onSetDefault: () -> Void internal let onSetEmpty: () -> Void internal let onSetFunction: (String) -> Void + internal var onRemoveField: (() -> Void)? internal var onToggleExpand: (() -> Void)? internal var onPopOut: ((String) -> Void)? @@ -187,10 +188,13 @@ internal struct InspectorFieldRow: View { canMutate: context.canMutate, isPendingNull: context.valueState == .pendingNull, isPendingDefault: context.valueState == .pendingDefault, + isPendingRemoval: context.valueState == .pendingRemoval, onSetNull: onSetNull, onSetDefault: onSetDefault, onSetEmpty: onSetEmpty, onSetFunction: onSetFunction, + onRemoveField: context.valueState == .absent || context.valueState == .pendingRemoval + ? nil : onRemoveField, onClear: { context.value.wrappedValue = context.originalValue ?? "" } ) } diff --git a/TableProTests/Core/ChangeTracking/FieldAbsenceTrackingTests.swift b/TableProTests/Core/ChangeTracking/FieldAbsenceTrackingTests.swift new file mode 100644 index 0000000000..5d96a38241 --- /dev/null +++ b/TableProTests/Core/ChangeTracking/FieldAbsenceTrackingTests.swift @@ -0,0 +1,286 @@ +// +// FieldAbsenceTrackingTests.swift +// TableProTests +// + +import Foundation +@testable import TablePro +import TableProPluginKit +import Testing + +struct FieldAbsencePendingChangesTests { + private let row = RowID.existing(0) + + @Test("NULL over a missing field is a change, although both read NULL") + func nullOverMissingFieldIsAChange() throws { + var pending = PendingChanges() + + let recorded = pending.recordCellChange( + rowID: row, columnIndex: 1, columnName: "deletedAt", oldValue: .null, newValue: .null, + absence: FieldAbsence(wasAbsent: true, isAbsent: false, originalRow: [1]) + ) + + #expect(recorded) + let cell = try #require(pending.changes.first?.cellChanges.first) + #expect(cell.oldIsAbsent && !cell.newIsAbsent) + #expect(pending.changes.first?.absentColumns == [1]) + } + + @Test("Removing a field that held NULL is a change") + func removingANullFieldIsAChange() throws { + var pending = PendingChanges() + + let recorded = pending.recordCellChange( + rowID: row, columnIndex: 1, columnName: "deletedAt", oldValue: .null, newValue: .null, + absence: FieldAbsence(wasAbsent: false, isAbsent: true) + ) + + #expect(recorded) + #expect(try #require(pending.changes.first?.cellChanges.first).newIsAbsent) + } + + @Test("Removing a field and then typing its value back leaves nothing pending") + func removeThenRestoreCollapses() { + var pending = PendingChanges() + pending.recordCellChange( + rowID: row, columnIndex: 1, columnName: "nick", oldValue: "Ada", newValue: .null, + absence: FieldAbsence(wasAbsent: false, isAbsent: true) + ) + pending.recordCellChange( + rowID: row, columnIndex: 1, columnName: "nick", oldValue: .null, newValue: "Ada", + absence: FieldAbsence(wasAbsent: true, isAbsent: false) + ) + + #expect(pending.isEmpty) + } + + @Test("Removing a NULL field and setting NULL again leaves nothing pending") + func removeThenNullOnANullFieldCollapses() { + var pending = PendingChanges() + pending.recordCellChange( + rowID: row, columnIndex: 1, columnName: "nick", oldValue: .null, newValue: .null, + absence: FieldAbsence(wasAbsent: false, isAbsent: true) + ) + pending.recordCellChange( + rowID: row, columnIndex: 1, columnName: "nick", oldValue: .null, newValue: .null, + absence: FieldAbsence(wasAbsent: true, isAbsent: false) + ) + + #expect(pending.isEmpty) + } + + @Test("Setting NULL on a field that was removed keeps the field, with the stored value as the old side") + func nullAfterRemovalIsStillAChange() throws { + var pending = PendingChanges() + pending.recordCellChange( + rowID: row, columnIndex: 1, columnName: "nick", oldValue: "Ada", newValue: .null, + absence: FieldAbsence(wasAbsent: false, isAbsent: true) + ) + pending.recordCellChange( + rowID: row, columnIndex: 1, columnName: "nick", oldValue: .null, newValue: .null, + absence: FieldAbsence(wasAbsent: true, isAbsent: false) + ) + + let cell = try #require(pending.changes.first?.cellChanges.first) + #expect(cell.oldValue == "Ada" && !cell.oldIsAbsent) + #expect(cell.newValue == .null && !cell.newIsAbsent) + } + + @Test("A missing field on a new row follows its edits") + func insertedRowTracksItsMissingFields() { + var pending = PendingChanges() + let inserted = RowID.inserted(UUID()) + pending.recordRowInsertion(rowID: inserted, values: ["__DEFAULT__", .null, .null], absentColumns: [1, 2]) + + pending.recordCellChange( + rowID: inserted, columnIndex: 1, columnName: "name", oldValue: .null, newValue: "Ada", + absence: FieldAbsence(wasAbsent: true, isAbsent: false) + ) + pending.recordCellChange( + rowID: inserted, columnIndex: 2, columnName: "deletedAt", oldValue: .null, newValue: .null, + absence: FieldAbsence(wasAbsent: true, isAbsent: false) + ) + + #expect(pending.insertedAbsentColumns(forRow: inserted).isEmpty) + #expect(pending.insertedRowData[inserted] == ["__DEFAULT__", "Ada", .null]) + } + + @Test("A snapshot carries the fields a deleted row did not have") + func snapshotKeepsDeletedRowAbsence() { + var pending = PendingChanges() + pending.recordRowDeletion(rowID: row, originalRow: ["1", .null], absentColumns: [1]) + + var restored = PendingChanges() + restored.restore(from: pending.snapshot(primaryKeyColumns: ["_id"], columns: ["_id", "nick"])) + + #expect(restored.change(forRow: row, type: .delete)?.absentColumns == [1]) + } +} + +struct FieldAbsenceTableRowsTests { + private func rows(absentCells: [Int: Set]) -> TableRows { + TableRows.from( + queryRows: [["1", .null, "x"], ["2", .null, .null]], + columns: ["_id", "deletedAt", "nick"], + columnTypes: [.text(rawType: nil), .text(rawType: nil), .text(rawType: nil)], + absentCells: absentCells + ) + } + + @Test("Rows keep the fields each result row lacked, and nothing past the last column") + func rowsKeepAbsence() { + let tableRows = rows(absentCells: [0: [1], 1: [2, 9]]) + + #expect(tableRows.isAbsent(row: 0, column: 1)) + #expect(!tableRows.isAbsent(row: 1, column: 1)) + #expect(tableRows.rows[1].absentColumns == [2]) + } + + @Test("A value written into a missing field brings the field back, and removal takes it out") + func editMovesAbsence() { + var tableRows = rows(absentCells: [0: [1]]) + + #expect(tableRows.edit(row: 0, column: 1, value: .null) == .cellChanged(row: 0, column: 1)) + #expect(!tableRows.isAbsent(row: 0, column: 1)) + + #expect(tableRows.edit(row: 0, column: 2, value: .null, isAbsent: true) == .cellChanged(row: 0, column: 2)) + #expect(tableRows.isAbsent(row: 0, column: 2)) + #expect(tableRows.value(at: 0, column: 2) == .null) + #expect(tableRows.edit(row: 0, column: 2, value: .null, isAbsent: true) == .none) + } + + @Test("Several cells can be put back missing at once") + func editManyRestoresAbsence() { + var tableRows = rows(absentCells: [:]) + + let delta = tableRows.editMany( + [(row: 0, column: 2, value: .null), (row: 1, column: 1, value: "d")], + absentCells: [CellPosition(row: 0, column: 2)] + ) + + #expect(delta == .cellsChanged([CellPosition(row: 0, column: 2), CellPosition(row: 1, column: 1)])) + #expect(tableRows.isAbsent(row: 0, column: 2)) + #expect(!tableRows.isAbsent(row: 1, column: 1)) + } + + @Test("Fetching every row keeps what each row lacks") + func replaceKeepsAbsence() { + var tableRows = rows(absentCells: [:]) + tableRows.replace(rows: [["3", .null, "z"]], absentCells: [0: [1]]) + + #expect(tableRows.isAbsent(row: 0, column: 1)) + } +} + +@MainActor +struct FieldAbsenceUndoTests { + private func manager() -> (DataChangeManager, UndoManager) { + let manager = DataChangeManager() + manager.configureForTable( + tableName: "items", + columns: ["_id", "nick"], + primaryKeyColumns: ["_id"], + databaseType: .mongodb, + generatedColumns: [] + ) + let undoManager = UndoManager() + undoManager.groupsByEvent = false + manager.undoManagerProvider = { undoManager } + return (manager, undoManager) + } + + @Test("Undoing Remove Field brings the value back, and redoing it takes the field out again") + func undoAndRedoOfRemoval() throws { + let (manager, undoManager) = manager() + var captured: UndoResult? + manager.onUndoApplied = { captured = $0 } + var tableRows = TableRows.from( + queryRows: [["1", "Ada"]], columns: ["_id", "nick"], + columnTypes: [.text(rawType: nil), .text(rawType: nil)] + ) + let operations = RowOperationsManager(changeManager: manager) + + manager.recordCellChange( + rowID: .existing(0), columnIndex: 1, columnName: "nick", oldValue: "Ada", newValue: .null, + originalRow: ["1", "Ada"], absence: FieldAbsence(wasAbsent: false, isAbsent: true) + ) + tableRows.edit(row: 0, column: 1, value: .null, isAbsent: true) + + undoManager.undo() + _ = operations.applyUndoResult(try #require(captured), tableRows: &tableRows) + #expect(tableRows.value(at: 0, column: 1) == "Ada") + #expect(!tableRows.isAbsent(row: 0, column: 1)) + #expect(!manager.hasChanges) + + undoManager.redo() + _ = operations.applyUndoResult(try #require(captured), tableRows: &tableRows) + #expect(tableRows.isAbsent(row: 0, column: 1)) + let cell = try #require(manager.changes.first?.cellChanges.first) + #expect(cell.newIsAbsent && !cell.oldIsAbsent) + #expect(cell.oldValue == "Ada") + } + + @Test("Redoing a new row's insertion brings back the fields it had none of") + func redoOfInsertionKeepsMissingFields() throws { + let (manager, undoManager) = manager() + var captured: UndoResult? + manager.onUndoApplied = { captured = $0 } + var tableRows = TableRows.from( + queryRows: [], columns: ["_id", "nick"], + columnTypes: [.text(rawType: nil), .text(rawType: nil)] + ) + let operations = RowOperationsManager(changeManager: manager) + let added = try #require(operations.addNewRow(tableRows: &tableRows)) + let missing = manager.pending.insertedAbsentColumns(forRow: added.rowID) + #expect(!missing.isEmpty) + + undoManager.undo() + _ = operations.applyUndoResult(try #require(captured), tableRows: &tableRows) + #expect(tableRows.rows.isEmpty) + + undoManager.redo() + _ = operations.applyUndoResult(try #require(captured), tableRows: &tableRows) + #expect(manager.pending.insertedAbsentColumns(forRow: added.rowID) == missing) + #expect(tableRows.row(withID: added.rowID)?.absentColumns == missing) + #expect(manager.pending.insertedRowData[added.rowID] == added.values) + } + + @Test("Undoing a redone insertion takes the row out again") + func undoAfterRedoRemovesTheRow() throws { + let (manager, undoManager) = manager() + var captured: UndoResult? + manager.onUndoApplied = { captured = $0 } + var tableRows = TableRows.from( + queryRows: [], columns: ["_id", "nick"], + columnTypes: [.text(rawType: nil), .text(rawType: nil)] + ) + let operations = RowOperationsManager(changeManager: manager) + let added = try #require(operations.addNewRow(tableRows: &tableRows)) + + undoManager.undo() + _ = operations.applyUndoResult(try #require(captured), tableRows: &tableRows) + undoManager.redo() + _ = operations.applyUndoResult(try #require(captured), tableRows: &tableRows) + undoManager.undo() + _ = operations.applyUndoResult(try #require(captured), tableRows: &tableRows) + + #expect(!manager.pending.isRowInserted(added.rowID)) + #expect(tableRows.row(withID: added.rowID) == nil) + #expect(!manager.hasChanges) + } + + @Test("Discarding puts a removed field back and a filled-in missing field back to missing") + func originalValuesCarryAbsence() { + let (manager, _) = manager() + manager.recordCellChange( + rowID: .existing(0), columnIndex: 1, columnName: "nick", oldValue: .null, newValue: "Ada", + originalRow: ["1", .null], absence: FieldAbsence(wasAbsent: true, isAbsent: false, originalRow: [1]) + ) + + let originals = manager.getOriginalValues() + + #expect(originals.count == 1) + #expect(originals.first?.isAbsent == true) + #expect(originals.first?.value == .null) + } +} diff --git a/TableProTests/Core/DataWrite/FieldAbsenceWriteTests.swift b/TableProTests/Core/DataWrite/FieldAbsenceWriteTests.swift new file mode 100644 index 0000000000..296d0040f7 --- /dev/null +++ b/TableProTests/Core/DataWrite/FieldAbsenceWriteTests.swift @@ -0,0 +1,471 @@ +// +// FieldAbsenceWriteTests.swift +// TableProTests +// + +import Foundation +@testable import TablePro +import TableProPluginKit +import Testing + +/// Writes through the MongoDB plugin's own generator, so a test sees the statement the plugin +/// would send for what the host handed it. +private final class MongoGeneratorStubDriver: PluginDatabaseDriver, @unchecked Sendable { + func generateRowWrites( + table: String, + schema: String?, + columns: [String], + primaryKeyColumns: [String], + changes: [PluginRowChange], + insertedRowData: [Int: [PluginCellValue]], + deletedRowIndices: Set, + insertedRowIndices: Set + ) throws -> [PluginRowWrite]? { + try MongoDBStatementGenerator(collectionName: table, columns: columns).generateRowWrites( + from: changes, + insertedRowData: insertedRowData, + deletedRowIndices: deletedRowIndices, + insertedRowIndices: insertedRowIndices + ) + } + + func generateIdentityPreservingInsert( + table: String, + schema: String?, + columns: [String], + primaryKeyColumns: [String], + rows: [[PluginCellValue]], + absentCells: [Int: Set] + ) -> [(statement: String, parameters: [PluginCellValue])]? { + MongoDBStatementGenerator(collectionName: table, columns: columns) + .generateRestore(rows: rows, absentCells: absentCells) + } + + func quoteIdentifier(_ name: String) -> String { name } + func connect() async throws {} + func disconnect() {} + + func execute(query: String) async throws -> PluginQueryResult { + PluginQueryResult(columns: [], columnTypeNames: [], rows: [], rowsAffected: 0, executionTime: 0) + } + + func fetchTables(schema: String?) async throws -> [PluginTableInfo] { [] } + func fetchColumns(table: String, schema: String?) async throws -> [PluginColumnInfo] { [] } + func fetchIndexes(table: String, schema: String?) async throws -> [PluginIndexInfo] { [] } + func fetchForeignKeys(table: String, schema: String?) async throws -> [PluginForeignKeyInfo] { [] } + func fetchTableDDL(table: String, schema: String?) async throws -> String { "" } + func fetchViewDefinition(view: String, schema: String?) async throws -> String { "" } + + func fetchTableMetadata(table: String, schema: String?) async throws -> PluginTableMetadata { + PluginTableMetadata(tableName: table) + } + + func fetchDatabases() async throws -> [String] { [] } + + func fetchDatabaseMetadata(_ database: String) async throws -> PluginDatabaseMetadata { + PluginDatabaseMetadata(name: database) + } +} + +@MainActor +struct FieldAbsenceStatementTests { + private static let identity: PluginCellValue = "507f1f77bcf86cd799439011" + private let columns = ["_id", "nick", "deletedAt"] + + private func factory() -> RowChangeStatementFactory { + RowChangeStatementFactory( + tableName: "items", + schemaName: nil, + columns: columns, + primaryKeyColumns: ["_id"], + databaseType: .mongodb, + pluginDriver: MongoGeneratorStubDriver() + ) + } + + @Test("Remove Field reaches the driver as a missing field, and Set NULL as a value") + func removalAndNullReachTheDriverApart() throws { + let change = RowChange( + rowID: .existing(0), + type: .update, + cellChanges: [ + CellChange(columnIndex: 1, columnName: "nick", oldValue: "Ada", newValue: .null, newIsAbsent: true), + CellChange(columnIndex: 2, columnName: "deletedAt", oldValue: "2024-05-01", newValue: .null) + ], + originalRow: [Self.identity, "Ada", "2024-05-01"] + ) + + let statements = try factory().statements(for: [change]).map(\.sql) + + #expect(statements == [ + #"db.items.updateOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}}, {"$set": {"deletedAt": null}, "$unset": {"nick": ""}})"# + ]) + } + + @Test("A new row's missing fields are left out and its NULL fields are written") + func insertKeepsNullAndLeavesOutMissing() throws { + let inserted = RowID.inserted(UUID()) + let change = RowChange(rowID: inserted, type: .insert, absentColumns: [1]) + + let statements = try factory().statements( + for: [change], + insertedRowData: [inserted: ["__DEFAULT__", .null, .null]], + insertedRowIDs: [inserted] + ).map(\.sql) + + #expect(statements == [#"db.items.insertOne({"deletedAt": null})"#]) + } + + @Test("Every other engine is handed no missing fields") + func otherEnginesSeeNoAbsence() { + let keyed = PluginKeyedChanges( + changes: [ + RowChange( + rowID: .existing(0), type: .update, + cellChanges: [CellChange(columnIndex: 1, columnName: "name", oldValue: "a", newValue: .null)], + originalRow: ["1", "a"] + ), + RowChange(rowID: .existing(1), type: .delete, originalRow: ["2", "b"]) + ], + insertedRowData: [:], + deletedRowIDs: [.existing(1)], + insertedRowIDs: [] + ) + + #expect(keyed.changes.allSatisfy { $0.absentColumns == nil }) + } +} + +struct FieldAbsenceRewindRecordTests { + private let target = DataWriteTarget(database: "shop", schema: nil, table: "items") + private let columns = ["_id", "nick", "deletedAt"] + + private func operations( + changes: [RowChange], + insertedRowData: [RowID: [PluginCellValue]] = [:], + deletedRowIDs: Set = [], + insertedRowIDs: Set = [] + ) -> [RowWriteOperation] { + RowWriteOperationBuilder.operations( + from: changes, + insertedRowData: insertedRowData, + deletedRowIDs: deletedRowIDs, + insertedRowIDs: insertedRowIDs, + target: target, + columns: columns, + primaryKeyColumns: ["_id"], + generatedColumns: [], + containsTableOperation: false + ) + } + + @Test("An edit records which fields the row lacked before and after it") + func updateRecordsAbsenceOnBothSides() throws { + let change = RowChange( + rowID: .existing(0), + type: .update, + cellChanges: [ + CellChange(columnIndex: 1, columnName: "nick", oldValue: .null, newValue: "Ada", oldIsAbsent: true), + CellChange(columnIndex: 2, columnName: "deletedAt", oldValue: "2024", newValue: .null, newIsAbsent: true) + ], + originalRow: ["1", .null, "2024"], + absentColumns: [1] + ) + + let operation = try #require(operations(changes: [change]).first) + + #expect(operation.preImageAbsentColumns == [1]) + #expect(operation.postImageAbsentColumns == [2]) + } + + @Test("A delete records what the row lacked, and a new row what it was saved without") + func deleteAndInsertRecordAbsence() { + let inserted = RowID.inserted(UUID()) + let result = operations( + changes: [ + RowChange(rowID: .existing(0), type: .delete, originalRow: ["1", .null, .null], absentColumns: [2]), + RowChange(rowID: inserted, type: .insert, absentColumns: [1]) + ], + insertedRowData: [inserted: ["2", .null, .null]], + deletedRowIDs: [.existing(0)], + insertedRowIDs: [inserted] + ) + + #expect(result.first?.preImageAbsentColumns == [2]) + #expect(result.last?.postImageAbsentColumns == [1]) + } + + @Test("A record saved before absence was captured reads every NULL as a missing field, as it was rewound then") + func legacyRecordReadsNullsAsMissing() throws { + let current = RowWriteOperation( + kind: .update, target: target, columns: columns, primaryKeyColumns: ["_id"], + preImage: ["1", .null, "x"], postImage: ["1", "Ada", .null], + writtenColumns: ["nick", "deletedAt"], refusal: nil + ) + var json = try #require(JSONSerialization.jsonObject(with: JSONEncoder().encode(current)) as? [String: Any]) + json.removeValue(forKey: "preImageAbsentColumns") + json.removeValue(forKey: "postImageAbsentColumns") + + let legacy = try JSONDecoder().decode(RowWriteOperation.self, from: JSONSerialization.data(withJSONObject: json)) + + #expect(legacy.preImageAbsentColumns == nil) + #expect(legacy.absentColumnsBeforeWrite == [1]) + #expect(legacy.absentColumnsAfterWrite == [2]) + } +} + +@MainActor +struct FieldAbsenceRewindTests { + private static let identity: PluginCellValue = "507f1f77bcf86cd799439011" + private let columns = ["_id", "nick", "deletedAt"] + private let target = DataWriteTarget(database: "shop", schema: nil, table: "items") + + private func planner(_ operation: RowWriteOperation) -> RewindPlanner { + RewindPlanner( + record: RewindRecord( + id: UUID(), historyId: nil, connectionId: UUID(), databaseType: .mongodb, + target: target, capturedAt: Date(timeIntervalSince1970: 0), + generatedColumns: [], operations: [operation] + ), + factory: RowChangeStatementFactory( + tableName: target.table, schemaName: nil, columns: columns, + primaryKeyColumns: ["_id"], databaseType: .mongodb, pluginDriver: MongoGeneratorStubDriver() + ), + queryBuilder: TableQueryBuilder(databaseType: .mongodb, pagination: .offset) + ) + } + + private func update( + pre: [PluginCellValue], post: [PluginCellValue], + preAbsent: Set?, postAbsent: Set?, written: [String] + ) -> RowWriteOperation { + RowWriteOperation( + kind: .update, target: target, columns: columns, primaryKeyColumns: ["_id"], + preImage: pre, postImage: post, writtenColumns: written, refusal: nil, + preImageAbsentColumns: preAbsent, postImageAbsentColumns: postAbsent + ) + } + + private func inverse(of operation: RowWriteOperation, current: RewindCurrentRow?) throws -> [String] { + try planner(operation).plan(currentRows: current.map { [$0] } ?? []).statements.map(\.sql) + } + + private func outcome(of operation: RowWriteOperation, current: RewindCurrentRow) throws -> RewindRowOutcome? { + try planner(operation).plan(currentRows: [current]).rows.first?.outcome + } + + @Test("Rewinding Remove Field puts the value back") + func rewindOfRemovalSetsTheValue() throws { + let operation = update( + pre: [Self.identity, "Ada", .null], post: [Self.identity, .null, .null], + preAbsent: [], postAbsent: [1], written: ["nick"] + ) + + let current = RewindCurrentRow(values: [Self.identity, .null, .null], absentColumns: [1]) + #expect(try inverse(of: operation, current: current) == [ + #"db.items.updateOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}}, {"$set": {"nick": "Ada"}})"# + ]) + } + + @Test("Rewinding a value typed into a missing field removes the field again") + func rewindOfFillRemovesTheField() throws { + let operation = update( + pre: [Self.identity, .null, .null], post: [Self.identity, "Ada", .null], + preAbsent: [1], postAbsent: [], written: ["nick"] + ) + + #expect(try inverse(of: operation, current: RewindCurrentRow(values: [Self.identity, "Ada", .null])) == [ + #"db.items.updateOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}}, {"$unset": {"nick": ""}})"# + ]) + } + + @Test("Rewinding an edit of a field that held NULL stores NULL again") + func rewindOfNullFieldStoresNull() throws { + let operation = update( + pre: [Self.identity, .null, .null], post: [Self.identity, "Ada", .null], + preAbsent: [], postAbsent: [], written: ["nick"] + ) + + #expect(try inverse(of: operation, current: RewindCurrentRow(values: [Self.identity, "Ada", .null])) == [ + #"db.items.updateOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}}, {"$set": {"nick": null}})"# + ]) + } + + @Test("Rewinding a delete puts NULL fields back as null and leaves missing fields out") + func rewindOfDeleteKeepsNullAndMissingApart() throws { + let operation = RowWriteOperation( + kind: .delete, target: target, columns: columns, primaryKeyColumns: ["_id"], + preImage: [Self.identity, .null, .null], postImage: nil, writtenColumns: columns, refusal: nil, + preImageAbsentColumns: [1] + ) + + #expect(try inverse(of: operation, current: nil) == [ + #"db.items.insertOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}, "deletedAt": null})"# + ]) + } + + @Test("Rewinding Set NULL on a missing field removes the field again, though the value never changed") + func rewindOfNullOverMissingFieldRemovesIt() throws { + let operation = update( + pre: [Self.identity, .null, .null], post: [Self.identity, .null, .null], + preAbsent: [1], postAbsent: [], written: ["nick"] + ) + + #expect(try inverse(of: operation, current: RewindCurrentRow(values: [Self.identity, .null, .null])) == [ + #"db.items.updateOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}}, {"$unset": {"nick": ""}})"# + ]) + } + + @Test("Rewinding Remove Field on a NULL field stores null again, though the value never changed") + func rewindOfRemovalOfNullStoresNull() throws { + let operation = update( + pre: [Self.identity, .null, .null], post: [Self.identity, .null, .null], + preAbsent: [], postAbsent: [1], written: ["nick"] + ) + let current = RewindCurrentRow(values: [Self.identity, .null, .null], absentColumns: [1]) + + #expect(try inverse(of: operation, current: current) == [ + #"db.items.updateOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}}, {"$set": {"nick": null}})"# + ]) + } + + @Test("A field set to NULL since the save is a change, not the missing field the rewind would restore") + func presenceChangedSinceTheSaveIsAConflict() throws { + let filled = update( + pre: [Self.identity, .null, .null], post: [Self.identity, "Ada", .null], + preAbsent: [1], postAbsent: [], written: ["nick"] + ) + + let nulled = RewindCurrentRow(values: [Self.identity, .null, .null]) + let missing = RewindCurrentRow(values: [Self.identity, .null, .null], absentColumns: [1]) + #expect(try outcome(of: filled, current: nulled) == .changedSinceSave) + #expect(try outcome(of: filled, current: missing) == .alreadyRestored) + } + + @Test("A new document whose left-out field has been added since is a change") + func insertComparesPresence() throws { + let inserted = RowWriteOperation( + kind: .insert, target: target, columns: columns, primaryKeyColumns: ["_id"], + preImage: nil, postImage: [Self.identity, .null, .null], writtenColumns: columns, refusal: nil, + postImageAbsentColumns: [1] + ) + + let untouched = RewindCurrentRow(values: [Self.identity, .null, .null], absentColumns: [1]) + let nickAdded = RewindCurrentRow(values: [Self.identity, .null, .null]) + #expect(try outcome(of: inserted, current: untouched) == .willRestore) + #expect(try outcome(of: inserted, current: nickAdded) == .changedSinceSave) + } +} + +struct RewindCurrentRowTests { + private static let identity: PluginCellValue = "507f1f77bcf86cd799439011" + + private func result(columns: [String], rows: [[PluginCellValue]], absentCells: [Int: Set] = [:]) -> QueryResult { + var result = QueryResult( + columns: columns, + columnTypes: columns.map { _ in .text(rawType: nil) }, + rows: rows, + rowsAffected: 0, + executionTime: 0, + error: nil + ) + result.absentCells = absentCells + return result + } + + @Test("Documents read back line up with the record by field name, and a field none returned is missing") + func documentsAlignByName() { + let read = result( + columns: ["_id", "deletedAt"], + rows: [[Self.identity, .null], [Self.identity, .null]], + absentCells: [1: [1]] + ) + + let rows = RewindCurrentRow.rows(of: read, alignedTo: ["_id", "nick", "deletedAt"], matchingByName: true) + + #expect(rows == [ + RewindCurrentRow(values: [Self.identity, .null, .null], absentColumns: [1]), + RewindCurrentRow(values: [Self.identity, .null, .null], absentColumns: [1, 2]) + ]) + } + + @Test("Fields read back in another order are put in the recorded order") + func documentsAreReordered() { + let read = result(columns: ["nick", "_id"], rows: [["Ada", Self.identity]]) + + let rows = RewindCurrentRow.rows(of: read, alignedTo: ["_id", "nick"], matchingByName: true) + + #expect(rows == [RewindCurrentRow(values: [Self.identity, "Ada"])]) + } + + @Test("A SQL read is already in the recorded order and has no missing fields") + func sqlRowsStayAsRead() { + let read = result(columns: ["ID", "NAME"], rows: [["7", .null]]) + + let rows = RewindCurrentRow.rows(of: read, alignedTo: ["id", "name"], matchingByName: false) + + #expect(rows == [RewindCurrentRow(values: ["7", .null])]) + } +} + +@MainActor +struct FieldAbsenceSidebarSaveTests { + private static let identity: PluginCellValue = "507f1f77bcf86cd799439011" + + private func makeCoordinator() -> MainContentCoordinator { + let tabManager = QueryTabManager() + let coordinator = MainContentCoordinator( + connection: TestFixtures.makeConnection(type: .mongodb), + tabManager: tabManager, + changeManager: DataChangeManager(), + toolbarState: ConnectionToolbarState() + ) + var tab = QueryTab(title: "items", query: "db.items.find({})", tabType: .table, tableName: "items") + tab.execution.lastExecutedAt = Date() + tabManager.tabs.append(tab) + tabManager.selectedTabId = tab.id + + coordinator.setActiveTableRows( + TableRows.from( + queryRows: [[Self.identity, "Ada", .null]], + columns: ["_id", "nick", "deletedAt"], + columnTypes: Array(repeating: .text(rawType: nil), count: 3), + hasAuthoritativeSchema: true, + absentCells: [0: [2]] + ), + for: tab.id + ) + coordinator.changeManager.configureForTable( + tableName: "items", + columns: ["_id", "nick", "deletedAt"], + primaryKeyColumns: ["_id"], + databaseType: .mongodb, + generatedColumns: [] + ) + coordinator.changeManager.pluginDriver = MongoGeneratorStubDriver() + coordinator.selectionState.indices = [0] + return coordinator + } + + @Test("An inspector save of Remove Field removes the field rather than storing null in it") + func removalIsWrittenAsARemoval() throws { + let statements = try makeCoordinator().sidebarEditStatements(editedFields: [ + InspectorFieldEdit(columnIndex: 1, columnName: "nick", newValue: nil, removesField: true) + ]) + + #expect(statements.map(\.sql) == [ + #"db.items.updateOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}}, {"$unset": {"nick": ""}})"# + ]) + } + + @Test("An inspector save of NULL into a missing field stores null in it") + func nullIntoMissingFieldIsStored() throws { + let statements = try makeCoordinator().sidebarEditStatements(editedFields: [ + InspectorFieldEdit(columnIndex: 2, columnName: "deletedAt", newValue: nil) + ]) + + #expect(statements.map(\.sql) == [ + #"db.items.updateOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}}, {"$set": {"deletedAt": null}})"# + ]) + } +} diff --git a/TableProTests/Core/DataWrite/RewindPlannerTests.swift b/TableProTests/Core/DataWrite/RewindPlannerTests.swift index 9df0e8188b..e20d9efed1 100644 --- a/TableProTests/Core/DataWrite/RewindPlannerTests.swift +++ b/TableProTests/Core/DataWrite/RewindPlannerTests.swift @@ -4,9 +4,9 @@ // import Foundation +@testable import TablePro import TableProPluginKit import Testing -@testable import TablePro @MainActor struct RewindPlannerTests { @@ -61,7 +61,7 @@ struct RewindPlannerTests { func unchangedRowRestores() throws { let plan = try planner(operations: [ operation(kind: .update, preImage: ["7", "Ada"], postImage: ["7", "Grace"], writtenColumns: ["name"]), - ]).plan(currentRows: [["7", "Grace"]]) + ]).plan(currentRows: [RewindCurrentRow(values: ["7", "Grace"])]) #expect(plan.rows.first?.outcome == .willRestore) #expect(plan.restorableCount == 1) @@ -72,7 +72,7 @@ struct RewindPlannerTests { func changedRowIsSkipped() throws { let plan = try planner(operations: [ operation(kind: .update, preImage: ["7", "Ada"], postImage: ["7", "Grace"], writtenColumns: ["name"]), - ]).plan(currentRows: [["7", "Hopper"]]) + ]).plan(currentRows: [RewindCurrentRow(values: ["7", "Hopper"])]) #expect(plan.rows.first?.outcome == .changedSinceSave) #expect(plan.restorableCount == 0) @@ -84,7 +84,7 @@ struct RewindPlannerTests { func alreadyRestoredRowIsIdempotent() throws { let plan = try planner(operations: [ operation(kind: .update, preImage: ["7", "Ada"], postImage: ["7", "Grace"], writtenColumns: ["name"]), - ]).plan(currentRows: [["7", "Ada"]]) + ]).plan(currentRows: [RewindCurrentRow(values: ["7", "Ada"])]) #expect(plan.rows.first?.outcome == .alreadyRestored) #expect(plan.statements.isEmpty) @@ -116,7 +116,7 @@ struct RewindPlannerTests { queryBuilder: TableQueryBuilder(databaseType: .sqlite, pagination: .offset) ) - let plan = try planner.plan(currentRows: [["7", "Grace", "2026-06-30"]]) + let plan = try planner.plan(currentRows: [RewindCurrentRow(values: ["7", "Grace", "2026-06-30"])]) #expect(plan.rows.first?.outcome == .willRestore) } @@ -138,7 +138,8 @@ struct RewindPlannerTests { #expect(restored.rows.first?.outcome == .willRestore) #expect(restored.statements.first?.sql.hasPrefix("INSERT") == true) - let occupied = try planner(operations: [deleted]).plan(currentRows: [["7", "Someone else"]]) + let occupied = try planner(operations: [deleted]) + .plan(currentRows: [RewindCurrentRow(values: ["7", "Someone else"])]) #expect(occupied.rows.first?.outcome == .rowAlreadyPresent) #expect(occupied.statements.isEmpty) } @@ -147,7 +148,7 @@ struct RewindPlannerTests { func insertInverse() throws { let plan = try planner(operations: [ operation(kind: .insert, preImage: nil, postImage: ["7", "Ada"], writtenColumns: columns), - ]).plan(currentRows: [["7", "Ada"]]) + ]).plan(currentRows: [RewindCurrentRow(values: ["7", "Ada"])]) #expect(plan.rows.first?.outcome == .willRestore) #expect(plan.statements.first?.sql.hasPrefix("DELETE") == true) @@ -160,7 +161,7 @@ struct RewindPlannerTests { kind: .update, preImage: ["7", "Ada"], postImage: ["7", "Grace"], writtenColumns: ["name"], refusal: .noPrimaryKey ), - ]).plan(currentRows: [["7", "Grace"]]) + ]).plan(currentRows: [RewindCurrentRow(values: ["7", "Grace"])]) #expect(plan.rows.first?.outcome == .notReversible(.noPrimaryKey)) #expect(plan.statements.isEmpty) @@ -215,7 +216,8 @@ struct RewindPlannerTests { let first = operation(kind: .delete, preImage: ["7", "Ada"], postImage: nil, writtenColumns: columns) let second = operation(kind: .insert, preImage: nil, postImage: ["8", "Grace"], writtenColumns: columns) - let plan = try planner(operations: [first, second]).plan(currentRows: [["8", "Grace"]]) + let plan = try planner(operations: [first, second]) + .plan(currentRows: [RewindCurrentRow(values: ["8", "Grace"])]) #expect(plan.statements.count == 2) #expect(plan.statements.first?.sql.hasPrefix("DELETE") == true) diff --git a/TableProTests/Core/MongoDB/BsonDocumentFlattenerTests.swift b/TableProTests/Core/MongoDB/BsonDocumentFlattenerTests.swift index a0e0bc565f..ea41397038 100644 --- a/TableProTests/Core/MongoDB/BsonDocumentFlattenerTests.swift +++ b/TableProTests/Core/MongoDB/BsonDocumentFlattenerTests.swift @@ -17,6 +17,20 @@ struct BsonDocumentFlattenerTests { #expect(BsonDocumentFlattener.uniformKind(of: "_id", in: [], representation: .unspecified) == nil) } + @Test("A field a document does not have is absent, and a field holding null is not") + func absentCellsTellMissingFromNull() { + let documents: [[String: Any]] = [ + ["_id": "a", "deletedAt": NSNull(), "nick": "x"], + ["_id": "b", "nick": "y"], + ["_id": "c", "deletedAt": NSNull(), "nick": "z"] + ] + + let absent = BsonDocumentFlattener.absentCells(of: documents, columns: ["_id", "deletedAt", "nick", "declared"]) + + #expect(absent == [0: [3], 1: [1, 3], 2: [3]]) + #expect(BsonDocumentFlattener.absentCells(of: [["_id": "a"]], columns: ["_id"]).isEmpty) + } + // MARK: - unionColumns(from:) @Suite("unionColumns") @@ -382,7 +396,7 @@ struct BsonDocumentFlattenerTests { @Test("Double never gains binary floating point noise") func doubleHasNoExcessDigits() { - #expect(BsonDocumentFlattener.stringValue(for: 1847.27, representation: .unspecified) == "1847.27") + #expect(BsonDocumentFlattener.stringValue(for: 1_847.27, representation: .unspecified) == "1847.27") #expect(BsonDocumentFlattener.stringValue(for: 0.1, representation: .unspecified) == "0.1") } @@ -394,7 +408,7 @@ struct BsonDocumentFlattenerTests { @Test("Double nested in a sub-document keeps its digits and stays a floating point value") func nestedDoubleMatchesScalar() { - let document: [String: Any] = ["rate": 0.1, "total": 1847.27, "qty": 3.0] + let document: [String: Any] = ["rate": 0.1, "total": 1_847.27, "qty": 3.0] let result = BsonDocumentFlattener.stringValue(for: document, representation: .unspecified) #expect(result == #"{"qty":3.0,"rate":0.1,"total":1847.27}"#) } @@ -442,7 +456,7 @@ struct BsonDocumentFlattenerTests { @Test("Double nested in an array keeps the same digits as a top level one") func nestedArrayDoubleMatchesScalar() { - let values: [Any] = [0.1, 1847.27, 1.0 / 3.0] + let values: [Any] = [0.1, 1_847.27, 1.0 / 3.0] let result = BsonDocumentFlattener.stringValue(for: values, representation: .unspecified) #expect(result == "[0.1,1847.27,0.3333333333333333]") } diff --git a/TableProTests/Core/Plugins/PluginFieldRemovalCapabilityTests.swift b/TableProTests/Core/Plugins/PluginFieldRemovalCapabilityTests.swift new file mode 100644 index 0000000000..53ecb9861b --- /dev/null +++ b/TableProTests/Core/Plugins/PluginFieldRemovalCapabilityTests.swift @@ -0,0 +1,82 @@ +// +// PluginFieldRemovalCapabilityTests.swift +// TableProTests +// + +import Foundation +@testable import TablePro +import TableProPluginKit +import Testing + +private final class FieldRemovalMongoDBPlugin: NSObject, TableProPlugin, DriverPlugin { + static let pluginName = "Mock MongoDB" + static let pluginVersion = "1.0.0" + static let pluginDescription = "A MongoDB plugin built with field removal" + static let capabilities: [PluginCapability] = [.databaseDriver] + + static let databaseTypeId = "MongoDB" + static let databaseDisplayName = "MongoDB" + static let iconName = "mongodb-icon" + static let defaultPort = 27_017 + static let supportsFieldRemoval = true + + func createDriver(config: DriverConnectionConfig) -> any PluginDatabaseDriver { + fatalError("Not used in tests") + } +} + +private final class PreFieldRemovalMongoDBPlugin: NSObject, TableProPlugin, DriverPlugin { + static let pluginName = "Mock MongoDB" + static let pluginVersion = "1.0.0" + static let pluginDescription = "A MongoDB plugin built before field removal existed" + static let capabilities: [PluginCapability] = [.databaseDriver] + + static let databaseTypeId = "MongoDB" + static let databaseDisplayName = "MongoDB" + static let iconName = "mongodb-icon" + static let defaultPort = 27_017 + + func createDriver(config: DriverConnectionConfig) -> any PluginDatabaseDriver { + fatalError("Not used in tests") + } +} + +@MainActor +struct PluginFieldRemovalCapabilityTests { + @Test("A plugin that removes fields offers Remove Field once it loads") + func declaredCapabilityReachesTheSnapshot() { + let built = PluginMetadataRegistry.shared.buildMetadataSnapshot(from: FieldRemovalMongoDBPlugin.self) + #expect(built.capabilities.supportsFieldRemoval) + } + + @Test("A plugin built before field removal existed offers none of it, whatever the curated entry says") + func olderPluginOffersNothing() { + let built = PluginMetadataRegistry.shared.buildMetadataSnapshot(from: PreFieldRemovalMongoDBPlugin.self) + #expect(!built.capabilities.supportsFieldRemoval) + } + + @Test("Only a document store tells a missing field from NULL") + func onlyDocumentStoresOffer() { + #expect(PluginManager.shared.supportsFieldRemoval(for: .mongodb)) + #expect(!PluginManager.shared.supportsFieldRemoval(for: .postgresql)) + #expect(!PluginManager.shared.supportsFieldRemoval(for: .mysql)) + #expect(!PluginManager.shared.supportsFieldRemoval(for: .sqlite)) + } + + @Test("A result's missing cells survive encoding, and a result from an older release decodes without them") + func absentCellsRoundTrip() throws { + var result = PluginQueryResult( + columns: ["_id", "nick"], columnTypeNames: ["ObjectId", "String"], + rows: [["1", .null]], rowsAffected: 0, timing: PluginQueryTiming(total: 0) + ) + result.absentCells = [0: [1]] + + let decoded = try JSONDecoder().decode(PluginQueryResult.self, from: JSONEncoder().encode(result)) + #expect(decoded.absentCells == [0: [1]]) + + var json = try #require(JSONSerialization.jsonObject(with: JSONEncoder().encode(result)) as? [String: Any]) + json.removeValue(forKey: "absentCells") + let older = try JSONDecoder().decode(PluginQueryResult.self, from: JSONSerialization.data(withJSONObject: json)) + #expect(older.absentCells == nil) + } +} diff --git a/TableProTests/Core/Services/RowOperationsFieldAbsenceTests.swift b/TableProTests/Core/Services/RowOperationsFieldAbsenceTests.swift new file mode 100644 index 0000000000..e78fa0f5a0 --- /dev/null +++ b/TableProTests/Core/Services/RowOperationsFieldAbsenceTests.swift @@ -0,0 +1,154 @@ +// +// RowOperationsFieldAbsenceTests.swift +// TableProTests +// + +import AppKit +import Foundation +@testable import TablePro +import TableProPluginKit +import Testing + +private final class AbsenceClipboard: ClipboardProvider { + var written: GridRowsClipboardPayload? + var gridRowsToRead: GridRowsClipboardPayload? + var textToRead: String? + + func readText() -> String? { textToRead } + func readGridRows() -> GridRowsClipboardPayload? { gridRowsToRead } + func writeText(_ text: String) {} + func writeCsv(_ csv: String) {} + func writeImage(_ image: NSImage) {} + func writeRows(tsv: String, html: String?, gridRows: GridRowsClipboardPayload) { written = gridRows } + var hasText: Bool { textToRead != nil } + var hasGridRows: Bool { gridRowsToRead != nil } +} + +@MainActor +struct RowOperationsFieldAbsenceTests { + private static let columns = ["_id", "name", "deletedAt"] + + private func makeManager(_ databaseType: DatabaseType = .mongodb) -> (RowOperationsManager, DataChangeManager) { + let changeManager = DataChangeManager() + changeManager.configureForTable( + tableName: "items", + columns: Self.columns, + primaryKeyColumns: ["_id"], + databaseType: databaseType, + generatedColumns: [] + ) + return (RowOperationsManager(changeManager: changeManager), changeManager) + } + + private func rows(_ queryRows: [[PluginCellValue]] = [], absentCells: [Int: Set] = [:]) -> TableRows { + TableRows.from( + queryRows: queryRows, + columns: Self.columns, + columnTypes: Array(repeating: .text(rawType: nil), count: Self.columns.count), + hasAuthoritativeSchema: true, + absentCells: absentCells + ) + } + + private func paste( + _ clipboard: AbsenceClipboard, + into manager: RowOperationsManager, + parser: RowDataParser? = nil + ) -> (TableRows, RowOperationsManager.PasteRowsResult) { + var tableRows = rows() + let result = manager.pasteRowsFromClipboard( + columns: Self.columns, + primaryKeyColumns: ["_id"], + tableRows: &tableRows, + clipboard: clipboard, + parser: parser + ) + return (tableRows, result) + } + + @Test("A new MongoDB row starts with its fields missing, and a SQL row with NULLs") + func newRowStartsMissingOnlyWhereFieldsCanBe() throws { + let (mongo, mongoChanges) = makeManager() + var mongoRows = rows() + let added = try #require(mongo.addNewRow(tableRows: &mongoRows)) + + #expect(mongoRows.rows[0].absentColumns == [0, 1, 2]) + #expect(mongoChanges.changes.first?.absentColumns == [0, 1, 2]) + #expect(added.values == [.null, .null, .null]) + + let (sql, sqlChanges) = makeManager(.mysql) + var sqlRows = rows() + _ = sql.addNewRow(tableRows: &sqlRows) + + #expect(sqlRows.rows[0].absentColumns.isEmpty) + #expect(sqlChanges.changes.first?.absentColumns.isEmpty == true) + } + + @Test("A duplicated document keeps its NULL fields and leaves its missing fields missing") + func duplicateCopiesAbsence() throws { + let (manager, changes) = makeManager() + var tableRows = rows([["1", "Ada", .null], ["2", "Bo", .null]], absentCells: [1: [2]]) + + let nullKept = try #require(manager.duplicateRow(sourceRowIndex: 0, tableRows: &tableRows)) + let missingKept = try #require(manager.duplicateRow(sourceRowIndex: 1, tableRows: &tableRows)) + + #expect(tableRows.row(withID: nullKept.rowID)?.absentColumns.isEmpty == true) + #expect(tableRows.row(withID: missingKept.rowID)?.absentColumns == [2]) + #expect(changes.pending.insertedAbsentColumns(forRow: missingKept.rowID) == [2]) + #expect(missingKept.values[0] == .text("__DEFAULT__")) + } + + @Test("Copying rows records which fields each lacked, in the copied column order") + func copyWritesAbsence() throws { + let (manager, _) = makeManager() + let clipboard = AbsenceClipboard() + ClipboardService.shared = clipboard + let tableRows = rows([["1", "Ada", .null], ["2", "Bo", .null]], absentCells: [1: [2]]) + + manager.copySelectedRowsToClipboard(selectedIndices: [0, 1], tableRows: tableRows, visibleColumnIndices: [2, 1]) + + let payload = try #require(clipboard.written) + #expect(payload.columns == ["deletedAt", "name"]) + #expect(payload.absentCells == [1: [0]]) + } + + @Test("Pasting copied rows leaves missing what the copy lacked, and what it did not carry") + func structuredPasteCopiesAbsence() throws { + let (manager, _) = makeManager() + let clipboard = AbsenceClipboard() + clipboard.gridRowsToRead = GridRowsClipboardPayload( + columns: ["_id", "name"], + rows: [["1", "Ada"], ["2", .null]], + absentCells: [1: [1]] + ) + + let (tableRows, result) = paste(clipboard, into: manager) + + #expect(result.pastedRows.count == 2) + #expect(tableRows.rows[0].absentColumns == [2]) + #expect(tableRows.rows[1].absentColumns == [1, 2]) + } + + @Test("A SQL paste leaves a column the copy did not carry NULL, as before") + func sqlPasteHasNoAbsence() { + let (manager, _) = makeManager(.mysql) + let clipboard = AbsenceClipboard() + clipboard.gridRowsToRead = GridRowsClipboardPayload(columns: ["_id", "name"], rows: [["1", "Ada"]]) + + let (tableRows, result) = paste(clipboard, into: manager) + + #expect(result.pastedRows.first?.values == [.text("__DEFAULT__"), "Ada", .null]) + #expect(tableRows.rows[0].absentColumns.isEmpty) + } + + @Test("Pasted text says nothing about fields, so a NULL in it leaves the field out of a document") + func textPasteReadsNullAsMissing() { + let (manager, _) = makeManager() + let clipboard = AbsenceClipboard() + clipboard.textToRead = "1\tAda\tNULL" + + let (tableRows, _) = paste(clipboard, into: manager, parser: TSVRowParser()) + + #expect(tableRows.rows[0].absentColumns == [2]) + } +} diff --git a/TableProTests/Models/FieldValueStateTests.swift b/TableProTests/Models/FieldValueStateTests.swift index 6106e2c069..df1f0f319e 100644 --- a/TableProTests/Models/FieldValueStateTests.swift +++ b/TableProTests/Models/FieldValueStateTests.swift @@ -135,4 +135,36 @@ struct FieldValueStateTests { #expect(FieldPickerSentinel.isSentinel(FieldPickerSentinel.defaultValue)) #expect(FieldPickerSentinel.isSentinel(FieldPickerSentinel.multiple)) } + + @Test("Choosing a row that only says where the field stands writes nothing") + func stateRowsWriteNothing() { + for tag in [FieldPickerSentinel.absent, FieldPickerSentinel.multiple] { + var written: [String] = [] + var commands = 0 + FieldPickerSentinel.choose(tag, onSetNull: { commands += 1 }, onSetDefault: { commands += 1 }) { + written.append($0) + } + #expect(written.isEmpty) + #expect(commands == 0) + } + } + + @Test("Choosing NULL, DEFAULT or a value does what it names") + func commandsAndValuesAreRouted() { + var calls: [String] = [] + let choose = { (tag: String) in + FieldPickerSentinel.choose( + tag, + onSetNull: { calls.append("null") }, + onSetDefault: { calls.append("default") }, + onValue: { calls.append("value \($0)") } + ) + } + + choose(FieldPickerSentinel.null) + choose(FieldPickerSentinel.defaultValue) + choose("1") + + #expect(calls == ["null", "default", "value 1"]) + } } diff --git a/TableProTests/Models/MultiRowEditStateFieldAbsenceTests.swift b/TableProTests/Models/MultiRowEditStateFieldAbsenceTests.swift new file mode 100644 index 0000000000..9331385fee --- /dev/null +++ b/TableProTests/Models/MultiRowEditStateFieldAbsenceTests.swift @@ -0,0 +1,158 @@ +// +// MultiRowEditStateFieldAbsenceTests.swift +// TableProTests +// + +import Foundation +@testable import TablePro +import TableProPluginKit +import Testing + +@MainActor +struct MultiRowEditStateFieldAbsenceTests { + private let rowA = RowID.existing(0) + private let rowB = RowID.existing(1) + + private func configured(rows: [[String?]], absentCells: [Set], modified: Set = []) -> MultiRowEditState { + let state = MultiRowEditState() + state.configure( + selectedRowIndices: Set(rows.indices), + rowIDs: Array([rowA, rowB].prefix(rows.count)), + allRows: rows, + absentCells: absentCells, + columns: ["_id", "nick"], + columnTypes: [.text(rawType: nil), .text(rawType: nil)], + externallyModifiedColumns: modified + ) + return state + } + + @Test("A field no selected row has reads No Field, and one they disagree on reads as multiple values") + func missingFieldHasItsOwnState() { + let missing = configured(rows: [["1", nil]], absentCells: [[1]]) + #expect(FieldValueState.resolve(missing.fields[1]) == .absent) + #expect(FieldValueState.resolve(missing.fields[1]).placeholder == "No Field") + + let null = configured(rows: [["1", nil]], absentCells: [[]]) + #expect(FieldValueState.resolve(null.fields[1]) == .null) + + let mixed = configured(rows: [["1", nil], ["2", nil]], absentCells: [[1], []]) + #expect(FieldValueState.resolve(mixed.fields[1]) == .multipleValues) + } + + @Test("Remove Field asks for the field to go from every selected row") + func removeFieldReportsTheRows() { + let state = configured(rows: [["1", "Ada"], ["2", "Bo"]], absentCells: [[], []]) + var removed: (Int, [RowID])? + state.onFieldRemoved = { removed = ($0, $1) } + + state.removeField(at: 1) + + #expect(removed?.0 == 1) + #expect(removed?.1 == [rowA, rowB]) + #expect(FieldValueState.resolve(state.fields[1]) == .pendingRemoval) + #expect(state.fields[1].hasEdit) + } + + @Test("A field the grid removed shows as a pending removal rather than an empty value") + func externallyRemovedFieldIsPendingRemoval() { + let state = configured(rows: [["1", nil]], absentCells: [[1]], modified: [1]) + + #expect(FieldValueState.resolve(state.fields[1]) == .pendingRemoval) + } + + @Test("Removing a field no selected row had, after typing into it, leaves nothing pending") + func removalBackToMissingIsNoEdit() { + let state = configured(rows: [["1", nil]], absentCells: [[1]]) + var removed: [RowID]? + state.onFieldChanged = { _, _, _ in } + state.onFieldRemoved = { removed = $1 } + + state.updateField(at: 1, value: "Ada") + state.removeField(at: 1) + + #expect(removed == [rowA]) + #expect(!state.hasEdits) + #expect(FieldValueState.resolve(state.fields[1]) == .absent) + #expect(state.getEditedFields().isEmpty) + } + + @Test("Removing a field only some selected rows had is still a pending removal") + func removalOnMixedRowsIsPending() { + let state = configured(rows: [["1", nil], ["2", "Bo"]], absentCells: [[1], []]) + state.onFieldRemoved = { _, _ in } + + state.removeField(at: 1) + + #expect(state.hasEdits) + #expect(FieldValueState.resolve(state.fields[1]) == .pendingRemoval) + } + + @Test("A pending removal reaches the save as a removal, not as NULL") + func editedFieldsCarryRemoval() { + let state = configured(rows: [["1", "Ada"]], absentCells: [[]]) + state.onFieldRemoved = { _, _ in } + + state.removeField(at: 1) + + #expect(state.getEditedFields() == [ + InspectorFieldEdit(columnIndex: 1, columnName: "nick", newValue: nil, removesField: true) + ]) + } + + @Test("Clearing a value typed into a missing field puts the field back to missing") + func revertOfMissingFieldRemovesItAgain() { + let state = configured(rows: [["1", nil]], absentCells: [[1]]) + var reverted: Set? + state.onFieldChanged = { _, _, _ in } + state.onFieldReverted = { _, _, absentRows in reverted = absentRows } + + state.updateField(at: 1, value: "Ada") + state.updateField(at: 1, value: nil) + + #expect(reverted == [rowA]) + } + + /// Staging refuses a value for a MongoDB `_id`, so the inspector offered Remove Field, Set NULL + /// and typing on a field whose edit then stayed pending and never saved. + @Test("A column the change manager refuses is read-only in the inspector, Remove Field included") + func unwritableColumnOffersNoEdit() throws { + let manager = DataChangeManager() + manager.configureForTable( + tableName: "items", columns: ["_id", "nick"], primaryKeyColumns: ["_id"], + databaseType: .mongodb, generatedColumns: [] + ) + let state = MultiRowEditState() + state.configure( + selectedRowIndices: [0], + rowIDs: [rowA], + allRows: [["1", "Ada"]], + absentCells: [[]], + columns: ["_id", "nick"], + columnTypes: [.text(rawType: nil), .text(rawType: nil)], + externallyModifiedColumns: [], + serverOwnedColumns: manager.unwritableColumns(among: ["_id", "nick"]) + ) + + let identity = try #require(state.fields.first) + let nick = try #require(state.fields.last) + #expect(!InspectorFieldListView.isFieldEditable(identity, kind: FieldEditorResolver.resolve(field: identity), rowIsEditable: true)) + #expect(InspectorFieldListView.isFieldEditable(nick, kind: FieldEditorResolver.resolve(field: nick), rowIsEditable: true)) + } + + @Test("The inspector's read-only columns are the generated ones and the ones the driver declares immutable") + func unwritableColumnsJoinGeneratedAndImmutable() { + let manager = DataChangeManager() + manager.configureForTable( + tableName: "items", columns: ["_id", "nick", "total"], primaryKeyColumns: ["_id"], + databaseType: .mongodb, generatedColumns: ["total"] + ) + #expect(manager.unwritableColumns(among: ["_id", "nick", "total"]) == ["_id", "total"]) + + manager.configureForTable( + tableName: "items", columns: ["_id", "nick"], primaryKeyColumns: ["_id"], + databaseType: .mysql, generatedColumns: [] + ) + #expect(manager.unwritableColumns(among: ["_id", "nick"]).isEmpty) + } +} diff --git a/TableProTests/Plugins/MongoDBCollectionSchemaTests.swift b/TableProTests/Plugins/MongoDBCollectionSchemaTests.swift index 14d10629d9..6b5b8ecad6 100644 --- a/TableProTests/Plugins/MongoDBCollectionSchemaTests.swift +++ b/TableProTests/Plugins/MongoDBCollectionSchemaTests.swift @@ -53,6 +53,110 @@ struct MongoDBCollectionSchemaTests { #expect(schema.valueKinds["score"] == nil) } + @Test("A field takes null only when its declared type lists null, whether or not it is required") + func nullIsDecidedByTheTypeNotTheRequiredList() throws { + let schema = MongoDBCollectionSchema.parse(jsonSchema: """ + {"bsonType": "object", "required": ["deletedAt", "title"], "properties": { + "deletedAt": {"bsonType": ["date", "null"]}, "title": {"bsonType": "string"}, + "nick": {"bsonType": "string"}, "note": {"type": ["string", "null"]}, "free": {"description": "any"}, + "level": {"enum": [1, 2]}, "mode": {"enum": ["a", null]}, "code": {"anyOf": [{"bsonType": "string"}]} + }} + """) + + #expect(try #require(schema.field(named: "deletedAt")).admitsNull) + #expect(try #require(schema.field(named: "title")).admitsNull == false) + #expect(try #require(schema.field(named: "nick")).admitsNull == false) + #expect(try #require(schema.field(named: "note")).admitsNull) + #expect(try #require(schema.field(named: "free")).admitsNull) + #expect(try #require(schema.field(named: "level")).admitsNull == false) + #expect(try #require(schema.field(named: "mode")).admitsNull) + #expect(try #require(schema.field(named: "code")).admitsNull == false) + } + + /// Measured on 7.0.43: `{s: null}` is refused with 121 under `bsonType: ["string", "null"]` plus + /// `enum: ["draft"]` and plus `not: {bsonType: "null"}`, and stored under `enum: ["draft", null]` + /// and under `minLength: 3` with `pattern: "^x"`. + @Test("Every keyword of a field's rule decides null together, not the type alone") + func siblingKeywordsDecideNullTogether() throws { + let schema = MongoDBCollectionSchema.parse(jsonSchema: """ + {"properties": { + "status": {"bsonType": ["string", "null"], "enum": ["draft"]}, + "stage": {"bsonType": ["string", "null"], "enum": ["draft", null]}, + "code": {"bsonType": ["string", "null"], "minLength": 3, "pattern": "^x"}, + "mode": {"enum": ["a", null], "not": {"bsonType": "null"}}, + "kind": {"type": ["string", "null"], "enum": ["a"]} + }} + """) + + #expect(try #require(schema.field(named: "status")).admitsNull == false) + #expect(try #require(schema.field(named: "stage")).admitsNull) + #expect(try #require(schema.field(named: "code")).admitsNull) + #expect(try #require(schema.field(named: "mode")).admitsNull == false) + #expect(try #require(schema.field(named: "kind")).admitsNull == false) + } + + /// Measured on 7.0.43: a top-level `anyOf` or `patternProperties` refuses null in a field whose + /// own rule takes it, and an `additionalProperties` schema refuses it in every undeclared field. + @Test("A rule over the whole document can refuse null in any field, so none is taken to admit it") + func documentWideRulesDecideNullForEveryField() { + for rule in [#""anyOf": [{"properties": {"s": {"bsonType": "string"}}}]"#, + #""patternProperties": {"^s": {"bsonType": "string"}}"#, + #""dependencies": {"a": {"properties": {"s": {"bsonType": "string"}}}}"#] { + let schema = MongoDBCollectionSchema.parse(jsonSchema: #"{"properties": {"s": {"bsonType": ["string", "null"]}}, "# + rule + "}") + #expect(schema.admitsNull(fieldNamed: "s") == false, "\(rule)") + #expect(schema.admitsNull(fieldNamed: "other") == false, "\(rule)") + } + + let plain = MongoDBCollectionSchema.parse(jsonSchema: #"{"properties": {"s": {"bsonType": ["string", "null"]}, "t": {"bsonType": "string"}}}"#) + #expect(plain.admitsNull(fieldNamed: "s")) + #expect(plain.admitsNull(fieldNamed: "t") == false) + #expect(plain.admitsNull(fieldNamed: "other")) + } + + @Test("additionalProperties decides null for the fields the validator does not declare") + func additionalPropertiesDecidesUndeclaredFields() { + let typed = MongoDBCollectionSchema.parse(jsonSchema: """ + {"additionalProperties": {"bsonType": "string"}, "properties": {"_id": {}, "t": {"bsonType": ["string", "null"]}}} + """) + #expect(typed.admitsNull(fieldNamed: "t")) + #expect(typed.admitsNull(fieldNamed: "u") == false) + + let nullable = MongoDBCollectionSchema.parse(jsonSchema: #"{"additionalProperties": {"bsonType": ["string", "null"]}}"#) + #expect(nullable.admitsNull(fieldNamed: "u")) + + let closed = MongoDBCollectionSchema.parse(jsonSchema: #"{"additionalProperties": false, "properties": {"_id": {}}}"#) + #expect(closed.admitsNull(fieldNamed: "u") == false) + } + + /// Measured on 7.0.43: `{$jsonSchema: {…s: ["string", "null"]…}, s: {$type: "string"}}` refuses + /// `{s: null}` with 121, and `validationAction: "warn"` stores it. + @Test("A query operator beside $jsonSchema refuses null everywhere, and a validator that only warns refuses nothing") + func validatorSettingsDecideNull() { + let besideSchema = MongoDBCollectionSchema.parse(listCollectionsReply: reply(options: """ + { "validator" : { "$jsonSchema" : { "properties" : { "s" : { "bsonType" : [ "string", "null" ] } } }, "s" : { "$type" : "string" } } } + """)) + #expect(besideSchema.field(named: "s") != nil) + #expect(besideSchema.admitsNull(fieldNamed: "s") == false) + + let queryOnly = MongoDBCollectionSchema.parse(listCollectionsReply: reply(options: """ + { "validator" : { "age" : { "$gte" : { "$numberInt" : "0" } } } } + """)) + #expect(queryOnly.isEmpty) + #expect(queryOnly.admitsNull(fieldNamed: "age") == false) + + for setting in [#""validationAction" : "warn""#, #""validationLevel" : "off""#] { + let unenforced = MongoDBCollectionSchema.parse(listCollectionsReply: reply(options: """ + { "validator" : { "$jsonSchema" : { "properties" : { "t" : { "bsonType" : "string" } } } }, \(setting) } + """)) + #expect(unenforced.admitsNull(fieldNamed: "t"), "\(setting)") + } + + let enforced = MongoDBCollectionSchema.parse(listCollectionsReply: reply(options: articlesOptions)) + #expect(enforced.admitsNull(fieldNamed: "title") == false) + #expect(enforced.admitsNull(fieldNamed: "date")) + #expect(MongoDBCollectionSchema.parse(listCollectionsReply: reply(options: "{ }")).admitsNull(fieldNamed: "any")) + } + @Test("A string enum is kept as the field's allowed values") func enumsAreKept() { let schema = MongoDBCollectionSchema.parse(listCollectionsReply: reply(options: articlesOptions)) diff --git a/TableProTests/Plugins/MongoDBFieldNameUpdateTests.swift b/TableProTests/Plugins/MongoDBFieldNameUpdateTests.swift index 6bc111f63d..b7fb9173b2 100644 --- a/TableProTests/Plugins/MongoDBFieldNameUpdateTests.swift +++ b/TableProTests/Plugins/MongoDBFieldNameUpdateTests.swift @@ -14,6 +14,7 @@ struct MongoDBFieldNameUpdateTests { private func update( _ cells: [(column: String, old: PluginCellValue, new: PluginCellValue)], columns: [String], + removing removed: Set = [], kinds: [String: BsonValueKind] = [:], version: String? = "7.0.43" ) throws -> String { @@ -24,7 +25,7 @@ struct MongoDBFieldNameUpdateTests { fieldKinds: MongoDBFieldKinds(kinds.mapValues { [$0] }), capabilities: { MongoDBCapabilities.parse(version) } ) - let change = PluginRowChange( + var change = PluginRowChange( rowIndex: 0, type: .update, cellChanges: cells.map { cell in @@ -32,6 +33,7 @@ struct MongoDBFieldNameUpdateTests { }, originalRow: [.text("1")] + columns.dropFirst().map { _ in PluginCellValue.null } ) + change.absentColumns = Set(removed.compactMap { columns.firstIndex(of: $0) }) let writes = try gen.generateRowWrites( from: [change], insertedRowData: [:], deletedRowIndices: [], insertedRowIndices: [] ) @@ -70,7 +72,7 @@ struct MongoDBFieldNameUpdateTests { @Test("Removing a field with a dotted name uses $unsetField") func dottedNameRemovalUsesUnsetField() throws { - let statement = try update([("price.usd", "10", nil)], columns: ["_id", "price.usd"]) + let statement = try update([("price.usd", "10", nil)], columns: ["_id", "price.usd"], removing: ["price.usd"]) #expect(statement.contains(#"{"$replaceWith": {"$unsetField": {"field": {"$literal": "price.usd"}, "input": "$$ROOT"}}}"#)) #expect(!statement.contains(#""$unset": {"#)) @@ -80,7 +82,8 @@ struct MongoDBFieldNameUpdateTests { func mixedRowIsOnePipeline() throws { let statement = try update( [("name", "a", "b"), ("price.usd", "10", "12"), ("note", "x", nil)], - columns: ["_id", "name", "price.usd", "note"] + columns: ["_id", "name", "price.usd", "note"], + removing: ["note"] ) let stages = try stages(of: statement) @@ -92,7 +95,9 @@ struct MongoDBFieldNameUpdateTests { @Test("A row of ordinary fields keeps the classic $set document, which every server version reads") func ordinaryRowStaysClassic() throws { - let statement = try update([("name", "a", "b"), ("note", "x", nil)], columns: ["_id", "name", "note"], version: "4.4.0") + let statement = try update( + [("name", "a", "b"), ("note", "x", nil)], columns: ["_id", "name", "note"], removing: ["note"], version: "4.4.0" + ) #expect(statement == #"db.items.updateOne({"_id": 1}, {"$set": {"name": "b"}, "$unset": {"note": ""}})"#) } diff --git a/TableProTests/Plugins/MongoDBFieldPresenceWriteTests.swift b/TableProTests/Plugins/MongoDBFieldPresenceWriteTests.swift new file mode 100644 index 0000000000..5db3558b3f --- /dev/null +++ b/TableProTests/Plugins/MongoDBFieldPresenceWriteTests.swift @@ -0,0 +1,137 @@ +// +// MongoDBFieldPresenceWriteTests.swift +// TableProTests +// + +import Foundation +import TableProPluginKit +import Testing + +/// Measured on MongoDB 7.0.43 with a validator of `required: ["deletedAt"]` and `bsonType: +/// ["date", "null"]`: `$unset` of `deletedAt` fails with 121 "Document failed validation", and +/// `$set: {deletedAt: null}` saves and still matches `{deletedAt: {$exists: true}}`. +struct MongoDBFieldPresenceWriteTests { + private static let identity: PluginCellValue = "507f1f77bcf86cd799439011" + + private func update( + _ cells: [(column: Int, old: PluginCellValue, new: PluginCellValue)], + columns: [String], + removing removed: Set = [] + ) throws -> String { + let gen = MongoDBStatementGenerator(collectionName: "items", columns: columns) + var change = PluginRowChange( + rowIndex: 0, + type: .update, + cellChanges: cells.map { (columnIndex: $0.column, columnName: columns[$0.column], oldValue: $0.old, newValue: $0.new) }, + originalRow: [Self.identity] + columns.dropFirst().map { _ in PluginCellValue.null } + ) + if !removed.isEmpty { + change.absentColumns = removed + } + let writes = try gen.generateRowWrites( + from: [change], insertedRowData: [:], deletedRowIndices: [], insertedRowIndices: [] + ) + return try #require(writes.first?.statement) + } + + @Test("Set NULL stores null and leaves the field in the document") + func setNullStoresNull() throws { + let statement = try update([(1, "2024-05-01T10:00:00Z", .null)], columns: ["_id", "deletedAt"]) + + #expect(statement == #"db.items.updateOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}}, {"$set": {"deletedAt": null}})"#) + #expect(!statement.contains("$unset")) + } + + @Test("Remove Field writes $unset, and only for the field it names") + func removeFieldWritesUnset() throws { + let statement = try update( + [(1, "a", .null), (2, "b", .null)], + columns: ["_id", "nick", "deletedAt"], + removing: [1] + ) + + #expect(statement.contains(#""$unset": {"nick": ""}"#)) + #expect(statement.contains(#""$set": {"deletedAt": null}"#)) + } + + @Test("A value written into a field the document lacked is a $set") + func valueIntoMissingFieldIsSet() throws { + let statement = try update([(1, .null, "Ada")], columns: ["_id", "nick"]) + + #expect(statement == #"db.items.updateOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}}, {"$set": {"nick": "Ada"}})"#) + } + + @Test("A field named with a dot takes null through $setField and is removed through $unsetField") + func specialNamesKeepNullAndRemovalApart() throws { + let nulled = try update([(1, "10", .null)], columns: ["_id", "price.usd"]) + #expect(nulled.contains(#""$setField": {"field": {"$literal": "price.usd"}, "input": "$$ROOT", "value": {"$literal": null}}"#)) + #expect(!nulled.contains("$unsetField")) + + let removed = try update([(1, "10", .null)], columns: ["_id", "price.usd"], removing: [1]) + #expect(removed.contains(#""$unsetField": {"field": {"$literal": "price.usd"}, "input": "$$ROOT"}"#)) + } + + @Test("A new row keeps its null fields and leaves out the ones it does not have") + func insertTellsNullFromMissing() throws { + let gen = MongoDBStatementGenerator(collectionName: "items", columns: ["_id", "name", "deletedAt", "nick"]) + var change = PluginRowChange(rowIndex: 0, type: .insert, cellChanges: [], originalRow: nil) + change.absentColumns = [3] + + let writes = try gen.generateRowWrites( + from: [change], + insertedRowData: [0: ["__DEFAULT__", "Ada", .null, .null]], + deletedRowIndices: [], + insertedRowIndices: [0] + ) + + #expect(writes.map(\.statement) == [#"db.items.insertOne({"name": "Ada", "deletedAt": null})"#]) + } + + @Test("A NULL _id on a new row is the server's to generate, not a null key") + func nullIdentityOnInsertIsLeftOut() throws { + let gen = MongoDBStatementGenerator(collectionName: "items", columns: ["_id", "name"]) + let change = PluginRowChange(rowIndex: 0, type: .insert, cellChanges: [], originalRow: nil) + + let writes = try gen.generateRowWrites( + from: [change], insertedRowData: [0: [.null, "Ada"]], deletedRowIndices: [], insertedRowIndices: [0] + ) + + #expect(writes.map(\.statement) == [#"db.items.insertOne({"name": "Ada"})"#]) + } + + /// Measured on 7.0.43: insertOne fails with "[22] invalid document for insert: empty key" for an + /// empty name whatever it holds, null included, and the shell drops `__proto__` whatever it + /// holds. Leaving the field out is the only route, and it is what the refusal names. + @Test("A new row with an empty or __proto__ field refuses null there and leaves the field out once removed") + func unwritableNamesAreLeftOutOnlyWhenMissing() throws { + for name in ["", "__proto__"] { + let gen = MongoDBStatementGenerator(collectionName: "items", columns: ["_id", "name", name]) + var change = PluginRowChange(rowIndex: 0, type: .insert, cellChanges: [], originalRow: nil) + let row: [PluginCellValue] = [.null, "Ada", .null] + + #expect(throws: PluginRowWriteRefusal.self, "\(name)") { + try gen.generateRowWrites( + from: [change], insertedRowData: [0: row], deletedRowIndices: [], insertedRowIndices: [0] + ) + } + + change.absentColumns = [2] + let writes = try gen.generateRowWrites( + from: [change], insertedRowData: [0: row], deletedRowIndices: [], insertedRowIndices: [0] + ) + #expect(writes.map(\.statement) == [#"db.items.insertOne({"name": "Ada"})"#], "\(name)") + } + } + + @Test("Putting a deleted document back keeps its null fields and leaves missing ones missing") + func restoreTellsNullFromMissing() throws { + let gen = MongoDBStatementGenerator(collectionName: "items", columns: ["_id", "name", "deletedAt", "nick"]) + + let restored = try #require(gen.generateRestore( + rows: [[Self.identity, "Ada", .null, .null]], + absentCells: [0: [3]] + )?.first) + + #expect(restored.statement == #"db.items.insertOne({"_id": {"$oid": "507f1f77bcf86cd799439011"}, "name": "Ada", "deletedAt": null})"#) + } +} diff --git a/TableProTests/Plugins/MongoDBStatementGeneratorTests.swift b/TableProTests/Plugins/MongoDBStatementGeneratorTests.swift index 569d47f349..df632e6c54 100644 --- a/TableProTests/Plugins/MongoDBStatementGeneratorTests.swift +++ b/TableProTests/Plugins/MongoDBStatementGeneratorTests.swift @@ -77,19 +77,20 @@ struct MongoDBStatementGeneratorTests { #expect(!stmt.contains("\"age\"")) } - @Test("Insert with nil values are excluded from document") - func insertNilValuesExcluded() throws { + @Test("An insert leaves out a field the new row does not have") + func insertLeavesOutAbsentFields() throws { let gen = MongoDBStatementGenerator( collectionName: "users", columns: ["_id", "name", "email"] ) - let change = PluginRowChange( + var change = PluginRowChange( rowIndex: 0, type: .insert, cellChanges: [], originalRow: nil ) + change.absentColumns = [2] let insertedData: [Int: [PluginCellValue]] = [ 0: [nil, "Carol", nil] @@ -102,10 +103,27 @@ struct MongoDBStatementGeneratorTests { insertedRowIndices: [0] ) - #expect(results.count == 1) - let stmt = results[0].statement - #expect(stmt.contains("\"name\": \"Carol\"")) - #expect(!stmt.contains("\"email\"")) + #expect(results.map(\.statement) == [#"db.users.insertOne({"name": "Carol"})"#]) + } + + @Test("An insert writes NULL as null, so a duplicated null field keeps its null") + func insertWritesNullAsNull() throws { + let gen = MongoDBStatementGenerator( + collectionName: "users", + columns: ["_id", "name", "deletedAt"] + ) + + var change = PluginRowChange(rowIndex: 0, type: .insert, cellChanges: [], originalRow: nil) + change.absentColumns = [] + + let results = try gen.generateRowWrites( + from: [change], + insertedRowData: [0: ["__DEFAULT__", "Carol", nil]], + deletedRowIndices: [], + insertedRowIndices: [0] + ) + + #expect(results.map(\.statement) == [#"db.users.insertOne({"name": "Carol", "deletedAt": null})"#]) } @Test("A new row with every cell empty inserts the empty document") @@ -412,7 +430,7 @@ struct MongoDBStatementGeneratorTests { columns: ["_id", "name", "bio"] ) - let change = PluginRowChange( + var change = PluginRowChange( rowIndex: 0, type: .update, cellChanges: [ @@ -421,6 +439,7 @@ struct MongoDBStatementGeneratorTests { ], originalRow: ["507f1f77bcf86cd799439011", "Alice", "Some bio"] ) + change.absentColumns = [2] let results = try gen.generateRowWrites( from: [change], diff --git a/TableProTests/Plugins/MongoDBWriteBackBinaryTests.swift b/TableProTests/Plugins/MongoDBWriteBackBinaryTests.swift index 63f8bbbaec..62362ec56a 100644 --- a/TableProTests/Plugins/MongoDBWriteBackBinaryTests.swift +++ b/TableProTests/Plugins/MongoDBWriteBackBinaryTests.swift @@ -35,18 +35,22 @@ struct MongoDBWriteBackBinaryTests { return recorded } - /// A new row whose `typed` columns the user filled in; every other value was copied into it. + /// A new row whose `typed` columns the user filled in and whose `missing` columns it lacks; + /// every other value was copied into it. private func insert( _ values: [PluginCellValue], typed: Set = [], + missing absentColumns: Set = [], columns: [String] = ["_id", "name", "thumbnail"], with gen: MongoDBStatementGenerator ) throws -> [PluginRowWrite] { let filledIn = typed.sorted().map { index in (columnIndex: index, columnName: columns[index], oldValue: PluginCellValue.null, newValue: values[index]) } + var change = PluginRowChange(rowIndex: 0, type: .insert, cellChanges: filledIn, originalRow: nil) + change.absentColumns = absentColumns return try gen.generateRowWrites( - from: [PluginRowChange(rowIndex: 0, type: .insert, cellChanges: filledIn, originalRow: nil)], + from: [change], insertedRowData: [0: values], deletedRowIndices: [], insertedRowIndices: [0] @@ -136,13 +140,13 @@ struct MongoDBWriteBackBinaryTests { @Test("A row empty apart from a value that cannot be written is refused, not inserted as {}") func refusedValueNeverBecomesEmptyDocument() { #expect(throws: MongoDBWriteRefusal.binarySubtypeUnknown(field: "thumbnail").refusal(ofRow: 0)) { - try insert([nil, nil, .bytes(Self.png)], with: generator()) + try insert([nil, nil, .bytes(Self.png)], missing: [0, 1], with: generator()) } } @Test("An _id typed into a new row is kept, typed the way the filters type it") func typedIdIsKept() throws { - let writes = try insert(["1001", "a", nil], with: generator(identityKind: .int32)) + let writes = try insert(["1001", "a", nil], missing: [2], with: generator(identityKind: .int32)) #expect(writes[0].statement == #"db.items.insertOne({"_id": 1001, "name": "a"})"#) } diff --git a/TableProTests/Views/Main/InspectorFieldEditStagingTests.swift b/TableProTests/Views/Main/InspectorFieldEditStagingTests.swift index 7252041319..473595e33b 100644 --- a/TableProTests/Views/Main/InspectorFieldEditStagingTests.swift +++ b/TableProTests/Views/Main/InspectorFieldEditStagingTests.swift @@ -56,8 +56,10 @@ struct InspectorFieldEditStagingTests { columnIndex: columnIndex, value: value, rowIDs: rows, continuity: continuity ) } - state.onFieldReverted = { [coordinator] columnIndex, valuesByRow in - coordinator.revertInspectorFieldEdit(columnIndex: columnIndex, valuesByRow: valuesByRow) + state.onFieldReverted = { [coordinator] columnIndex, valuesByRow, absentRowIDs in + coordinator.revertInspectorFieldEdit( + columnIndex: columnIndex, valuesByRow: valuesByRow, absentRowIDs: absentRowIDs + ) } return state } diff --git a/TableProTests/Views/Main/SidebarSaveCoverageTests.swift b/TableProTests/Views/Main/SidebarSaveCoverageTests.swift index d66256307d..b87147655a 100644 --- a/TableProTests/Views/Main/SidebarSaveCoverageTests.swift +++ b/TableProTests/Views/Main/SidebarSaveCoverageTests.swift @@ -53,7 +53,7 @@ struct SidebarSaveCoverageTests { #expect(throws: DataWriteError.changesNotWritable(table: "items", unwritten: UnwrittenRowCounts(updates: 1))) { _ = try coordinator.sidebarEditStatements( - editedFields: [(columnIndex: 1, columnName: "name", newValue: "z")] + editedFields: [InspectorFieldEdit(columnIndex: 1, columnName: "name", newValue: "z")] ) } } @@ -67,7 +67,7 @@ struct SidebarSaveCoverageTests { coordinator.selectionState.indices = [0, 1] let statements = try coordinator.sidebarEditStatements( - editedFields: [(columnIndex: 1, columnName: "name", newValue: "z")] + editedFields: [InspectorFieldEdit(columnIndex: 1, columnName: "name", newValue: "z")] ) #expect(statements.map(\.sql) == ["updateOne(0)", "updateOne(1)"]) diff --git a/TableProTests/Views/Main/ValueFilterEditedRowTests.swift b/TableProTests/Views/Main/ValueFilterEditedRowTests.swift index 55b84a333a..009647d0ee 100644 --- a/TableProTests/Views/Main/ValueFilterEditedRowTests.swift +++ b/TableProTests/Views/Main/ValueFilterEditedRowTests.swift @@ -179,7 +179,7 @@ struct ValueFilterEditedRowTests { fixture.coordinator.selectionState.indices = [0] let statements = try fixture.coordinator.sidebarEditStatements( - editedFields: [(columnIndex: 1, columnName: "name", newValue: "Yan")] + editedFields: [InspectorFieldEdit(columnIndex: 1, columnName: "name", newValue: "Yan")] ) #expect(statements.count == 1) diff --git a/TableProTests/Views/Results/DataGridFieldRemovalTests.swift b/TableProTests/Views/Results/DataGridFieldRemovalTests.swift new file mode 100644 index 0000000000..f71068ac4a --- /dev/null +++ b/TableProTests/Views/Results/DataGridFieldRemovalTests.swift @@ -0,0 +1,139 @@ +// +// DataGridFieldRemovalTests.swift +// TableProTests +// + +import AppKit +import SwiftUI +import TableProPluginKit +import Testing + +@testable import TablePro + +@MainActor +private final class NoopFieldRemovalLayoutPersister: ColumnLayoutPersisting { + func load(for key: ColumnLayoutTableKey) -> ColumnLayoutState? { nil } + func save(_ layout: ColumnLayoutState, for key: ColumnLayoutTableKey) {} + func clear(for key: ColumnLayoutTableKey) {} +} + +private final class FieldRemovalClipboard: ClipboardProvider { + var written: GridRowsClipboardPayload? + + func readText() -> String? { nil } + func readGridRows() -> GridRowsClipboardPayload? { nil } + func writeText(_ text: String) {} + func writeCsv(_ csv: String) {} + func writeImage(_ image: NSImage) {} + func writeRows(tsv: String, html: String?, gridRows: GridRowsClipboardPayload) { written = gridRows } + var hasText: Bool { false } + var hasGridRows: Bool { written != nil } +} + +@MainActor +private final class FieldRemovalRowStore { + var tableRows: TableRows + + init(_ tableRows: TableRows) { + self.tableRows = tableRows + } +} + +@MainActor +struct DataGridFieldRemovalTests { + private static let columns = ["_id", "nick", "deletedAt"] + + private func makeGrid( + _ databaseType: DatabaseType = .mongodb + ) -> (TableViewCoordinator, FieldRemovalRowStore, DataChangeManager) { + let manager = DataChangeManager() + manager.configureForTable( + tableName: "items", + columns: Self.columns, + primaryKeyColumns: ["_id"], + databaseType: databaseType, + generatedColumns: [] + ) + let coordinator = TableViewCoordinator( + changeManager: AnyChangeManager(manager), + isEditable: true, + selectedRowIndices: .constant([]), + delegate: nil, + layoutPersister: NoopFieldRemovalLayoutPersister() + ) + coordinator.databaseType = databaseType + let store = FieldRemovalRowStore(TableRows.from( + queryRows: [["1", "Ada", .null]], + columns: Self.columns, + columnTypes: Array(repeating: .text(rawType: nil), count: Self.columns.count), + absentCells: [0: [2]] + )) + coordinator.tableRowsProvider = { store.tableRows } + coordinator.tableRowsMutator = { mutation in mutation(&store.tableRows) } + coordinator.updateCache() + return (coordinator, store, manager) + } + + @Test("A missing field reads No Field, not NULL") + func missingFieldIsNamed() { + let (coordinator, _, _) = makeGrid() + + #expect(coordinator.accessibilityText(row: 0, columnIndex: 2) == "No Field") + #expect(DataGridCellContent.placeholder(for: .null, isAbsent: true) == .absent) + #expect(DataGridCellContent.placeholder(for: .null) == .null) + } + + @Test("Remove Field takes the field out of the row and stages the removal") + func removeFieldStagesTheRemoval() throws { + let (coordinator, store, manager) = makeGrid() + + coordinator.removeField(row: 0, columnIndex: 1) + + #expect(store.tableRows.isAbsent(row: 0, column: 1)) + let cell = try #require(manager.changes.first?.cellChanges.first) + #expect(cell.oldValue == "Ada" && !cell.oldIsAbsent) + #expect(cell.newIsAbsent) + #expect(coordinator.accessibilityText(row: 0, columnIndex: 1) == "No Field") + } + + @Test("Set NULL on a missing field puts the field back holding null") + func nullOverMissingFieldIsAnEdit() throws { + let (coordinator, store, manager) = makeGrid() + + coordinator.commitCellEdit(row: 0, columnIndex: 2, newValue: nil) + + #expect(!store.tableRows.isAbsent(row: 0, column: 2)) + let change = try #require(manager.changes.first) + #expect(change.absentColumns == [2]) + let cell = try #require(change.cellChanges.first) + #expect(cell.oldIsAbsent && !cell.newIsAbsent) + #expect(cell.newValue == .null) + #expect(coordinator.accessibilityText(row: 0, columnIndex: 2) == "NULL") + } + + @Test("Copy with Headers records which fields each copied row lacked") + func copyWithHeadersCarriesAbsence() throws { + let (coordinator, _, _) = makeGrid() + let clipboard = FieldRemovalClipboard() + ClipboardService.shared = clipboard + defer { ClipboardService.shared = NSPasteboardClipboardProvider() } + + coordinator.copyRowsWithHeaders(at: [0]) + + let payload = try #require(clipboard.written) + #expect(payload.columns == Self.columns) + #expect(payload.rows == [["1", "Ada", .null]]) + #expect(payload.absentCells == [0: [2]]) + } + + @Test("An engine that cannot tell a missing field from NULL offers no removal") + func removalNeedsTheCapability() { + let (coordinator, store, manager) = makeGrid(.mysql) + + coordinator.removeField(row: 0, columnIndex: 1) + + #expect(!coordinator.supportsFieldRemoval) + #expect(!store.tableRows.isAbsent(row: 0, column: 1)) + #expect(!manager.hasChanges) + } +} diff --git a/docs/databases/mongodb.mdx b/docs/databases/mongodb.mdx index eaf3ca39bd..eccfb96ec0 100644 --- a/docs/databases/mongodb.mdx +++ b/docs/databases/mongodb.mdx @@ -77,7 +77,7 @@ Click a collection to page through its documents. The columns are the fields in An edited cell keeps its field's type. A date stays a date, whether typed as `2024-01-02T03:04:05Z`, as `2024-01-02`, or picked in the date picker, which writes your local time. An ObjectId stays an ObjectId, an integer past 2^53 is sent as a 64-bit integer, and a field the validator declares as `string` is written as text even when it reads `123` or `true`. Where the grid has no type to go on, `123` is a number and `{"a": 1}` a document; text that only looks like JSON is written as a string. -Editing a nested object or array writes only the paths that changed, such as `address.city` or `tags.1`, so every other value in it keeps its type and its place. Adding two keys at once, reordering keys, or changing an array's length writes the whole value instead. A binary cell keeps its BSON subtype when its bytes are edited, and a duplicated or pasted row keeps its binary fields. **Set NULL** removes the field from the document, and a new row saved with every cell empty inserts a document holding only a generated `_id`. +Editing a nested object or array writes only the paths that changed, such as `address.city` or `tags.1`, so every other value in it keeps its type and its place. Adding two keys at once, reordering keys, or changing an array's length writes the whole value instead. A binary cell keeps its BSON subtype when its bytes are edited, and a duplicated or pasted row keeps its binary fields. A field whose own name contains a dot, starts with `$` or is `__proto__` is written with `$setField`, which needs MongoDB 5.0 or later. On an older server the save is refused and nothing is sent. @@ -85,6 +85,14 @@ The filter bar's column picker lists paths inside nested objects and arrays of o The Structure tab lists a collection's indexes; drop one from a query tab with `db.users.dropIndex("email_1")`. **New Database** asks for a database name and a first collection, both required. **New View** opens a query tab holding a `db.createView("view_name", "source_collection", [pipeline])` template, and editing a view pre-fills `db.runCommand({"collMod": …})`. +### Missing fields and null + +A field a document does not have reads **No Field**; a field holding null reads NULL. **Set Value > NULL** stores `null` and keeps the field. To delete a field, right-click the cell and choose **Remove Field**, or choose it from the value menu of the field in the inspector; saving sends `$unset`. A validator that lists the field in `required` refuses the save with `Document failed validation`. + +A new row starts with every field missing, and only the cells you fill in are written, so a row saved untouched inserts a document holding only a generated `_id`. **Duplicate** and **Paste** keep which fields were missing and which held null. Pasted text carries no such distinction, so a `NULL` in it leaves the field out. Undo, discarding, and [restoring a save](/features/change-tracking#restoring-a-save) put a removed field back and take out a field that was filled in. A restore treats a field removed or added since the save as **Changed since the save**, and leaves that document alone. + +**Set Value > NULL** appears only where the validator takes null: the field's type lists `null` or is not declared, and any `enum` on it lists `null` too. A rule over the whole document, such as a top-level `anyOf` or a query operator beside `$jsonSchema`, hides it on every field. A validator set to `validationAction: "warn"` hides it on none. The filter bar's **is NULL** matches a field holding null and a missing field alike, as `{field: null}` does in `mongosh`. To find only documents that lack a field, run `db.users.find({nickname: {$exists: false}})` in a query tab. + ### Binary UUIDs A binary subtype 4 field renders as `UUID("8cd003eb-4a25-4324-9332-88fce2da0d1a")`. Subtype 3 is the legacy format, and its bytes do not say which driver wrote them, so it stays `BinData(3, "…")` until **Legacy UUID Encoding** on the connection is set to Java, C#, or Python. Match it to the driver that wrote the data: the wrong choice shows a valid-looking but wrong UUID. @@ -217,8 +225,8 @@ New connections default to **Disabled**, and the driver has no TLS fallback: **P - A binary value keeps the subtype it was read with. Bytes typed into a new row are saved as subtype 0 where the validator declares the field `binData`. - Other bytes refuse the save when their subtype was never read, or was read as two different ones: bytes typed into an existing document, pasted from another collection, or put back by Data Rewind after TablePro restarts. Write that value with a query. - A field holding text in some documents and objects or arrays in others takes no value that reads as JSON from the grid, since it could be either type: the save is refused. Write it with a query. -- A new row, duplicate or paste cannot hold a field named `__proto__`, or an empty field name at any depth: the save is refused. Set that cell to NULL, save, then set it on the saved document. -- `_id` is read-only in the grid, and a row added with **Add Row** is inserted without one so the server generates it. To choose your own, use **Insert Document…**. +- A new row, duplicate or paste cannot hold a field named `__proto__`, or an empty field name at any depth: the save is refused. Use **Remove Field** on that cell, save, then set it on the saved document. +- `_id` is read-only in the grid and the row inspector, and a row added with **Add Row** is inserted without one so the server generates it. To choose your own, use **Insert Document…**. - Transactions are not exposed. Statements always run standalone, on any topology. - A collection takes one text index. A second **FULLTEXT** row fails after the collection and the indexes before it are created: list every text field in one index instead. - **New Table…** writes the validator with the server's own level and action. To log bad documents instead of refusing them, run `db.runCommand({collMod: "articles", validationAction: "warn"})` after creating the collection. diff --git a/docs/features/change-tracking.mdx b/docs/features/change-tracking.mdx index 9b95cd24e9..e69ef8430a 100644 --- a/docs/features/change-tracking.mdx +++ b/docs/features/change-tracking.mdx @@ -38,6 +38,8 @@ Double-click a cell, or press `Enter` on it. `Enter` commits the edit to the que Right-click an editable cell and open **Set Value** for the common ones without typing: **Empty**, **NULL** on a nullable column, **Default** on a column the server fills in, and date functions such as `NOW()` or `CURRENT_DATE` on date columns. +On MongoDB a document can lack a field altogether, which the grid shows as **No Field**. **Remove Field** on the same right-click menu deletes the field rather than storing NULL. See [missing fields and null](/databases/mongodb#missing-fields-and-null). + ## Adding and deleting rows - **Add Row** (`Cmd+Shift+N`), the **+** button, or **Edit > Add Row**. The row appears at the bottom with the cursor in its first writable cell. Later edits fold into the INSERT rather than queueing as updates. diff --git a/docs/features/table-structure.mdx b/docs/features/table-structure.mdx index e6c13f23b4..a0328736d7 100644 --- a/docs/features/table-structure.mdx +++ b/docs/features/table-structure.mdx @@ -254,7 +254,7 @@ Visual table creation is supported for MySQL, MariaDB, PostgreSQL, PGlite, SQLit ## MongoDB collections -MongoDB structure is read-only, and inferred from the collection's first 200 documents: top-level field names are unioned across the sample, and each field takes its most common BSON type. Fields the collection's `$jsonSchema` validator declares follow, including ones no sampled document holds, and a field in the validator's `required` list is not nullable. `_id` comes first, marked as the primary key. The DDL tab shows indexes as `createIndex()` commands for `mongosh`, plus the validator and capped-collection options where present. +MongoDB structure is read-only, and inferred from the collection's first 200 documents: top-level field names are unioned across the sample, and each field takes its most common BSON type. Fields the collection's `$jsonSchema` validator declares follow, including ones no sampled document holds. A field is nullable when the validator takes null in it, as [missing fields and null](/databases/mongodb#missing-fields-and-null) describes; `required` decides only whether a document must have the field. `_id` comes first, marked as the primary key. The DDL tab shows indexes as `createIndex()` commands for `mongosh`, plus the validator and capped-collection options where present. ## Limitations