diff --git a/CHANGELOG.md b/CHANGELOG.md index 3831e58ce..b1c5ee679 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -75,6 +75,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Import into Table** from a data file window, into an open connection's import sheet. - **Text Encoding** in a data file's Save As panel. - `BSONSymbol()` in the MongoDB shell. +- `MinKey`, `MaxKey`, `Code`, `DBRef` and the legacy UUID helpers in MongoDB shell autocomplete. ### Changed @@ -120,6 +121,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Large data files opened, filtered, sorted and searched in the background, with progress and Cancel. - `.json` and `.ndjson` files opened in the Data Files window rather than as a DuckDB connection. - `$regex` and `$options` objects in MongoDB scripts and **Raw Filter** sent as operator documents, as in mongosh. +- Whole numbers past 2^53 and `-0` stored as doubles by the MongoDB shell, as mongosh stores them. ### Removed @@ -229,6 +231,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `NumberDecimal("NaN")` and `NumberDecimal("Infinity")` refused by the MongoDB shell. - Fields named `__proto__` dropped from documents a MongoDB script writes or reads. - Error text after a carriage return or line separator left uncommented in Edit View Definition's fallback. +- Whole numbers from 2^63 up, such as `1e20`, refused by the MongoDB shell as a document MongoDB cannot read. +- `NumberLong` given a number past 2^53 storing a different integer in the MongoDB shell. +- Wrong values stored by `NumberInt` and `Timestamp` in the MongoDB shell, such as 12 for `NumberInt("12abc")`. +- `Int32`, `Long`, `Decimal128` and `Timestamp({ t, i })` missing from the MongoDB shell although autocomplete offers them. +- MongoDB scripts stopping on a stored decimal `NaN` or a regular expression JavaScript cannot compile. +- Regular expressions changed when a MongoDB script writes back a document it read. - Capped MongoDB collection size shown as 0 in DDL. - **Save** permanently dim on a Custom provider for an OpenAI-compatible server that wants no API key. - Model list not reloading when the API key changes, leaving the picker empty with no way to retry. diff --git a/Plugins/MongoDBDriverPlugin/MongoScriptPrelude.swift b/Plugins/MongoDBDriverPlugin/MongoScriptPrelude.swift index 799d0edf4..a03540b34 100644 --- a/Plugins/MongoDBDriverPlugin/MongoScriptPrelude.swift +++ b/Plugins/MongoDBDriverPlugin/MongoScriptPrelude.swift @@ -65,42 +65,126 @@ enum MongoScriptPrelude { }; ObjectId.prototype.toEJSON = function () { return { "$oid": this.__id }; }; - function __wholeText(value, name) { - var text = value === undefined ? "0" : String(value); - if (!/^[+-]?[0-9]+$/.test(text)) { throw new Error(name + " takes a whole number"); } - return text; + var __numeral = /^[+-]?([0-9]+(\\.[0-9]*)?|\\.[0-9]+)([eE][+-]?[0-9]+)?$/; + + function __wholeNumber(value, name) { + var number = value; + if (typeof value === "string" && __numeral.test(value.trim())) { + number = Number(value); + } else if (value instanceof Int32 || value instanceof Long || value instanceof Double) { + number = value.valueOf(); + } + if (typeof number !== "number" || !isFinite(number)) { throw new Error(name + " takes a number"); } + var whole = Math.trunc(number); + return whole === 0 ? 0 : whole; + } + + // Each legacy name shares its type's prototype, so `NumberInt(5) instanceof Int32` holds both + // ways round. + function __int32(text) { + var made = Object.create(Int32.prototype); + made.__value = text; + return made; + } + + function __checkedInt32(value, name) { + var whole = value === undefined ? 0 : __wholeNumber(value, name); + if (whole < -2147483648 || whole > 2147483647) { + throw new Error(name + " takes a whole number from -2147483648 to 2147483647"); + } + return __int32(String(whole)); + } + + function Int32(value) { return __checkedInt32(value, "Int32"); } + function NumberInt(value) { return __checkedInt32(value, "NumberInt"); } + NumberInt.prototype = Int32.prototype; + Int32.prototype.toString = function () { return this.__value; }; + Int32.prototype.valueOf = function () { return Number(this.__value); }; + Int32.prototype.toEJSON = function () { return { "$numberInt": this.__value }; }; + + var __int64Min = BigInt("-9223372036854775808"); + var __int64Max = BigInt("9223372036854775807"); + var __twoTo32 = BigInt(4294967296); + + function __long(text) { + var made = Object.create(Long.prototype); + made.__value = text; + return made; + } + + // A number is read as the integer it holds, which past 2^53 need not be the one that was + // typed: `9007199254740993` is already `9007199254740992`. Only a string or a Long carries + // every int64 exactly. + function __int64(value, name) { + if (value === undefined) { return BigInt(0); } + if (typeof value === "bigint") { return value; } + if (value instanceof Long) { return BigInt(value.__value); } + if (typeof value !== "string") { return BigInt(__wholeNumber(value, name)); } + var digits = value.trim(); + if (!/^[+-]?[0-9]+$/.test(digits)) { throw new Error(name + " takes a whole number"); } + return BigInt(digits); + } + + function __checkedLong(value, name) { + var whole = __int64(value, name); + if (whole < __int64Min || whole > __int64Max) { + throw new Error(name + " takes a whole number from -9223372036854775808 to 9223372036854775807"); + } + return __long(whole.toString()); } - function NumberLong(value) { - if (!(this instanceof NumberLong)) { return new NumberLong(value); } - this.__value = __wholeText(value, "NumberLong"); + function __isHalfOfLong(value) { + return typeof value === "number" && Math.trunc(value) === value + && value >= -2147483648 && value <= 4294967295; + } + + function __longFromHalves(low, high) { + if (!__isHalfOfLong(low) || !__isHalfOfLong(high)) { + throw new Error("Long takes its low and high halves as whole numbers from -2147483648 to 4294967295"); + } + return __long(BigInt.asIntN(64, BigInt(high >>> 0) * __twoTo32 + BigInt(low >>> 0)).toString()); } - NumberLong.prototype.toString = function () { return this.__value; }; - NumberLong.prototype.valueOf = function () { return Number(this.__value); }; - NumberLong.prototype.toNumber = function () { return Number(this.__value); }; - NumberLong.prototype.toEJSON = function () { return { "$numberLong": this.__value }; }; - function NumberInt(value) { - if (!(this instanceof NumberInt)) { return new NumberInt(value); } - this.__value = __wholeText(value === undefined ? 0 : parseInt(value, 10), "NumberInt"); + function Long(value, high) { + return high === undefined ? __checkedLong(value, "Long") : __longFromHalves(value, high); + } + function NumberLong(value) { return __checkedLong(value, "NumberLong"); } + NumberLong.prototype = Long.prototype; + Long.prototype.toString = function () { return this.__value; }; + Long.prototype.valueOf = function () { return Number(this.__value); }; + Long.prototype.toNumber = function () { return Number(this.__value); }; + Long.prototype.toEJSON = function () { return { "$numberLong": this.__value }; }; + + var __decimalSpecial = /^([+-]?)(inf|infinity|nan)$/i; + + function __decimal(text) { + var made = Object.create(Decimal128.prototype); + made.__value = text; + return made; } - NumberInt.prototype.toString = function () { return this.__value; }; - NumberInt.prototype.valueOf = function () { return Number(this.__value); }; - NumberInt.prototype.toEJSON = function () { return { "$numberInt": this.__value }; }; - function NumberDecimal(value) { - if (!(this instanceof NumberDecimal)) { return new NumberDecimal(value); } - var text = value === undefined ? "0" : String(value); - if (!/^([+-]?([0-9]+(\\.[0-9]*)?|\\.[0-9]+)([eE][+-]?[0-9]+)?|NaN|-?Infinity)$/.test(text)) { - throw new Error("NumberDecimal takes a number"); + function __decimalText(value, name) { + if (value === undefined) { return "0"; } + if (value === 0 && 1 / value < 0) { return "-0"; } + var text = String(value); + var special = __decimalSpecial.exec(text); + if (special) { + if (special[2].toLowerCase() === "nan") { return "NaN"; } + return special[1] === "-" ? "-Infinity" : "Infinity"; } - this.__value = text; + if (!__numeral.test(text)) { throw new Error(name + " takes a number"); } + return text; } - NumberDecimal.prototype.toString = function () { return this.__value; }; - NumberDecimal.prototype.toEJSON = function () { return { "$numberDecimal": this.__value }; }; - // A whole JavaScript number is sent as an integer, so a Double that happens to be whole, 1.0 or - // -0.0, needs its own constructor to stay a Double. Show DDL writes one for every such value. + function Decimal128(value) { return __decimal(__decimalText(value, "Decimal128")); } + function NumberDecimal(value) { return __decimal(__decimalText(value, "NumberDecimal")); } + NumberDecimal.prototype = Decimal128.prototype; + Decimal128.prototype.toString = function () { return this.__value; }; + Decimal128.prototype.toEJSON = function () { return { "$numberDecimal": this.__value }; }; + + // A whole JavaScript number up to 2^53 is sent as an integer, so a Double that happens to be + // whole, such as 1.0, needs its own constructor to stay a Double. Show DDL writes one for + // every such value. function __doubleText(value) { return value === 0 && 1 / value < 0 ? "-0.0" : String(value); } @@ -115,10 +199,38 @@ enum MongoScriptPrelude { Double.prototype.valueOf = function () { return this.__value; }; Double.prototype.toEJSON = function () { return { "$numberDouble": __doubleText(this.__value) }; }; + function __timestampHalf(value, name) { + if (typeof value !== "number" || isNaN(value)) { throw new Error("Timestamp takes " + name + " as a number"); } + if (value < 0 || value >= 4294967296) { throw new Error("Timestamp takes " + name + " from 0 to 4294967295"); } + return Math.trunc(value); + } + + function __timestampFromBits(bits) { + return [Number(bits / __twoTo32), Number(bits % __twoTo32)]; + } + + function __timestampParts(t, i) { + if (i !== undefined) { return [__timestampHalf(t, "t"), __timestampHalf(i, "i")]; } + if (t === undefined) { return [0, 0]; } + if (typeof t === "number") { return [__timestampHalf(t, "t"), 0]; } + if (t instanceof Long) { return __timestampFromBits(BigInt.asUintN(64, BigInt(t.__value))); } + if (typeof t === "bigint") { + if (t < BigInt(0) || t >= __twoTo32 * __twoTo32) { + throw new Error("Timestamp takes a bigint from 0 to 18446744073709551615"); + } + return __timestampFromBits(t); + } + if (t !== null && typeof t === "object" && Object.getPrototypeOf(t) === Object.prototype) { + return [__timestampHalf(t.t, "t"), __timestampHalf(t.i, "i")]; + } + throw new Error("Timestamp takes (t, i), { t, i }, a Long or a bigint"); + } + function Timestamp(t, i) { if (!(this instanceof Timestamp)) { return new Timestamp(t, i); } - this.t = t === undefined ? 0 : t; - this.i = i === undefined ? 0 : i; + var parts = __timestampParts(t, i); + this.t = parts[0]; + this.i = parts[1]; } Timestamp.prototype.toString = function () { return "Timestamp(" + this.t + ", " + this.i + ")"; }; Timestamp.prototype.toEJSON = function () { return { "$timestamp": { t: this.t, i: this.i } }; }; @@ -141,6 +253,24 @@ enum MongoScriptPrelude { return { "$regularExpression": { pattern: this.pattern, options: this.options } }; }; + // A stored expression JavaScript can compile reads as a RegExp, as in mongosh, and keeps the + // text the server holds: `source` escapes a slash and spells an empty pattern `(?:)`, so + // writing the document back would change it. The `l` and `x` options and PCRE-only syntax + // read as a BSONRegExp instead of throwing partway through a loop. + function __storedRegExp(pattern, options) { + if (/^[imsu]*$/.test(options)) { + try { + var compiled = new RegExp(pattern, options); + Object.defineProperty(compiled, "__stored", { value: { pattern: pattern, options: options } }); + return compiled; + } catch (unreadable) {} + } + var kept = Object.create(BSONRegExp.prototype); + kept.pattern = pattern; + kept.options = options; + return kept; + } + function BSONSymbol(value) { if (!(this instanceof BSONSymbol)) { return new BSONSymbol(value); } this.value = String(value); @@ -288,7 +418,7 @@ enum MongoScriptPrelude { if (this.__exhausted) { return false; } this.__started = true; var page = __tp.call({ op: "cursorFetch", handle: this.__handle }); - this.__batch = EJSON.deserialize(page.docs); + this.__batch = EJSON.__fromServer(page.docs); this.__index = 0; this.__exhausted = page.done; return this.__batch.length > 0; @@ -322,7 +452,7 @@ enum MongoScriptPrelude { Cursor.prototype.size = function () { return this.itcount(); }; Cursor.prototype.count = function () { return __tp.call({ op: "cursorCount", handle: this.__handle }); }; Cursor.prototype.explain = function (verbosity) { - return EJSON.deserialize(__tp.call({ + return EJSON.__fromServer(__tp.call({ op: "cursorExplain", handle: this.__handle, verbosity: verbosity === undefined ? "queryPlanner" : String(verbosity) @@ -357,7 +487,7 @@ enum MongoScriptPrelude { return __tp.call(payload); }; DBCollection.prototype.__reply = function (op, payload) { - return EJSON.deserialize(this.__call(op, payload)); + return EJSON.__fromServer(this.__call(op, payload)); }; DBCollection.prototype.find = function (filter, projection) { var handle = this.__call("openCursor", { @@ -589,10 +719,10 @@ enum MongoScriptPrelude { return { getDB: function (name) { return new DB(String(name)); } }; }; DB.prototype.runCommand = function (command) { - return EJSON.deserialize(__tp.call({ op: "command", db: this.__name, command: __ejson(command) })); + return EJSON.__fromServer(__tp.call({ op: "command", db: this.__name, command: __ejson(command) })); }; DB.prototype.adminCommand = function (command) { - return EJSON.deserialize(__tp.call({ op: "command", db: "admin", command: __ejson(command) })); + return EJSON.__fromServer(__tp.call({ op: "command", db: "admin", command: __ejson(command) })); }; DB.prototype.getCollectionNames = function () { return __tp.call({ op: "listCollections", db: this.__name }); @@ -660,7 +790,12 @@ enum MongoScriptPrelude { } if (value instanceof Date) { return { "$date": { "$numberLong": String(value.getTime()) } }; } if (value instanceof RegExp) { - return { "$regularExpression": { pattern: value.source, options: value.flags } }; + var stored = value.__stored; + return { + "$regularExpression": stored + ? { pattern: stored.pattern, options: stored.options } + : { pattern: value.source, options: value.flags } + }; } var document = {}; for (var key in value) { @@ -669,44 +804,90 @@ enum MongoScriptPrelude { return document; } + // mongosh sends every whole number outside int32 as a double. Up to 2^53 it is sent as + // int64 here, the type TablePro always gave it; past 2^53 a number is not an exact integer, + // so it stays the double it is rather than becoming a different int64. function serializeNumber(value) { - if (!isFinite(value)) { return { "$numberDouble": String(value) }; } - if (Math.floor(value) !== value) { return { "$numberDouble": String(value) }; } - return value >= -2147483648 && value <= 2147483647 - ? { "$numberInt": String(value) } - : { "$numberLong": String(value) }; + var whole = Math.trunc(value) === value && !(value === 0 && 1 / value < 0); + if (whole && value >= -2147483648 && value <= 2147483647) { return { "$numberInt": String(value) }; } + if (whole && Number.isSafeInteger(value)) { return { "$numberLong": String(value) }; } + return { "$numberDouble": __doubleText(value) }; } - function deserialize(value) { - if (value === null || typeof value !== "object") { return value; } - if (Array.isArray(value)) { - var list = []; - for (var i = 0; i < value.length; i++) { list.push(deserialize(value[i])); } - return list; + function deserializer(reviveWrapper) { + function deserialize(value) { + if (value === null || typeof value !== "object") { return value; } + if (Array.isArray(value)) { + var list = []; + for (var i = 0; i < value.length; i++) { list.push(deserialize(value[i])); } + return list; + } + var keys = Object.keys(value); + if (keys.length === 1 || (keys.length === 2 && keys[0] === "$code")) { + var revived = reviveWrapper(value, keys[0]); + if (revived !== undefined) { return revived; } + } + var document = {}; + for (var key in value) { + if (Object.prototype.hasOwnProperty.call(value, key)) { + __setMember(document, key, deserialize(value[key])); + } + } + return document; } - var keys = Object.keys(value); - if (keys.length === 1 || (keys.length === 2 && keys[0] === "$code")) { - var revived = revive(value, keys[0]); - if (revived !== undefined) { return revived; } + return deserialize; + } + + // What a script hands EJSON.parse is checked the way the constructors check it. What the + // server sends is not: it is already a value MongoDB holds, and reading a decimal NaN or a + // PCRE-only pattern back unchanged is what lets a script write the document back as it was. + function reviveChecked(value, key) { + switch (key) { + case "$numberInt": return checkedInt32(value.$numberInt); + case "$numberDouble": return checkedDouble(value.$numberDouble); + case "$numberLong": return __checkedLong(value.$numberLong, "$numberLong"); + case "$numberDecimal": return __decimal(__decimalText(value.$numberDecimal, "$numberDecimal")); + case "$regularExpression": return checkedRegExp(value.$regularExpression); + default: return revive(value, key); } - var document = {}; - for (var key in value) { - if (Object.prototype.hasOwnProperty.call(value, key)) { __setMember(document, key, deserialize(value[key])); } + } + + function checkedInt32(text) { + if (typeof text !== "string" || !/^[+-]?[0-9]+$/.test(text)) { + throw new Error("$numberInt takes a whole number"); } - return document; + return __checkedInt32(text, "$numberInt").valueOf(); } + function checkedDouble(text) { + if (typeof text !== "string" || !(__numeral.test(text) || /^(NaN|-?Infinity)$/.test(text))) { + throw new Error("$numberDouble takes a number"); + } + return Number(text); + } + + function checkedRegExp(wrapper) { + if (wrapper === null || typeof wrapper !== "object") { + throw new Error("$regularExpression takes a pattern and options"); + } + var checked = BSONRegExp(wrapper.pattern, wrapper.options); + return __storedRegExp(checked.pattern, checked.options); + } + + var deserializeChecked = deserializer(reviveChecked); + var deserializeFromServer = deserializer(revive); + function revive(value, key) { switch (key) { case "$oid": return new ObjectId(value.$oid); case "$numberInt": return parseInt(value.$numberInt, 10); case "$numberDouble": return Number(value.$numberDouble); - case "$numberLong": return new NumberLong(value.$numberLong); - case "$numberDecimal": return new NumberDecimal(value.$numberDecimal); + case "$numberLong": return __long(value.$numberLong); + case "$numberDecimal": return __decimal(value.$numberDecimal); case "$date": return new Date(typeof value.$date === "object" ? Number(value.$date.$numberLong) : value.$date); case "$regularExpression": - return new RegExp(value.$regularExpression.pattern, value.$regularExpression.options); + return __storedRegExp(value.$regularExpression.pattern, value.$regularExpression.options); case "$binary": return new BinData(parseInt(value.$binary.subType, 16), value.$binary.base64); case "$timestamp": return new Timestamp(value.$timestamp.t, value.$timestamp.i); case "$minKey": return MinKey; @@ -719,12 +900,13 @@ enum MongoScriptPrelude { return { serialize: serialize, - deserialize: deserialize, + deserialize: deserializeChecked, + __fromServer: deserializeFromServer, stringify: function (value, indent) { return JSON.stringify(serialize(value), null, indent === undefined ? 0 : indent); }, parse: function (text) { - return deserialize(typeof text === "string" ? JSON.parse(text) : text); + return deserializeChecked(typeof text === "string" ? JSON.parse(text) : text); } }; })(); diff --git a/TablePro/Core/Autocomplete/Mongo/MongoVocabulary.swift b/TablePro/Core/Autocomplete/Mongo/MongoVocabulary.swift index 8c82b2442..caa6876ad 100644 --- a/TablePro/Core/Autocomplete/Mongo/MongoVocabulary.swift +++ b/TablePro/Core/Autocomplete/Mongo/MongoVocabulary.swift @@ -349,17 +349,30 @@ enum MongoVocabulary { ("ObjectId", "12-byte ObjectId, from a 24-character hex string"), ("ISODate", "Date from an ISO 8601 string"), ("UUID", "UUID from a 36-character string, BSON binary subtype 4"), + ("LegacyJavaUUID", "UUID in the Java driver's legacy byte order, BSON binary subtype 3"), + ("LegacyCSharpUUID", "UUID in the C# driver's legacy byte order, BSON binary subtype 3"), + ("LegacyPythonUUID", "UUID in the Python driver's legacy byte order, BSON binary subtype 3"), + ("JUUID", "LegacyJavaUUID, legacy shell name"), + ("CSUUID", "LegacyCSharpUUID, legacy shell name"), + ("NUUID", "LegacyCSharpUUID, legacy shell name"), + ("PYUUID", "LegacyPythonUUID, legacy shell name"), + ("LUUID", "LegacyPythonUUID, legacy shell name"), ("BinData", "Binary value from a subtype and a base64 string"), ("HexData", "Binary value from a subtype and a hex string"), - ("Timestamp", "BSON timestamp from { t, i }"), + ("Timestamp", "BSON timestamp from (t, i), { t, i } or a Long"), ("Int32", "32-bit integer"), - ("Long", "64-bit integer, from a string"), + ("Long", "64-bit integer, from a string, a whole number or (low, high)"), ("Double", "64-bit floating point number"), ("Decimal128", "IEEE 754 decimal128, from a string"), ("NumberInt", "32-bit integer, legacy shell name"), - ("NumberLong", "64-bit integer, legacy shell name, string argument only"), + ("NumberLong", "64-bit integer, legacy shell name"), ("NumberDecimal", "Decimal128, legacy shell name"), - ("BSONRegExp", "Regular expression with options") + ("BSONRegExp", "Regular expression from a pattern and options i, l, m, s, u, x"), + ("BSONSymbol", "Deprecated BSON symbol, read back as a string"), + ("MinKey", "Compares lower than every other BSON value"), + ("MaxKey", "Compares higher than every other BSON value"), + ("Code", "JavaScript code, with an optional scope document"), + ("DBRef", "Reference to a document by collection, _id and database") ] static let shellCommands: [(name: String, detail: String)] = [ diff --git a/TableProTests/Core/Autocomplete/MongoVocabularyConstructorTests.swift b/TableProTests/Core/Autocomplete/MongoVocabularyConstructorTests.swift new file mode 100644 index 000000000..2bcc353f0 --- /dev/null +++ b/TableProTests/Core/Autocomplete/MongoVocabularyConstructorTests.swift @@ -0,0 +1,36 @@ +// +// MongoVocabularyConstructorTests.swift +// TableProTests +// + +import Foundation +import JavaScriptCore +@testable import TablePro +import Testing + +/// Autocomplete offered `Int32`, `Long`, `Decimal128` and `BSONRegExp` while the shell defined none +/// of them, so accepting the suggestion ended the statement in a ReferenceError, and it never +/// offered `MinKey`, `Code` or the legacy UUID helpers the shell does define. +struct MongoVocabularyConstructorTests { + private static let shellObjectTypes: Set = ["Cursor", "DB", "DBCollection"] + + private func globalNames(of context: JSContext) -> Set { + let names = context.evaluateScript("Object.getOwnPropertyNames(this)")?.toArray() as? [String] + return Set(names ?? []) + } + + @Test("Autocomplete offers exactly the value constructors the shell defines") + func offeredConstructorsAreTheShellsOwn() throws { + let plain = try #require(JSContext()) + let shell = try MongoScriptContext.make(execute: { _ in #"{"ok":true,"v":"shop"}"# }, emit: { _ in true }) + + let defined = globalNames(of: shell).subtracting(globalNames(of: plain)).filter { name in + guard name.first?.isUppercase == true, !Self.shellObjectTypes.contains(name) else { return false } + return shell.evaluateScript("typeof \(name)")?.toString() == "function" + } + let offered = Set(MongoVocabulary.bsonConstructors.map(\.name)) + + #expect(defined.count > 20) + #expect(offered == defined) + } +} diff --git a/TableProTests/Core/MongoDB/MongoScriptPreludeTests.swift b/TableProTests/Core/MongoDB/MongoScriptPreludeTests.swift index 7026a5712..0063c9ef7 100644 --- a/TableProTests/Core/MongoDB/MongoScriptPreludeTests.swift +++ b/TableProTests/Core/MongoDB/MongoScriptPreludeTests.swift @@ -421,6 +421,62 @@ struct MongoScriptPreludeTests { #expect(value?.toString() == "507f1f77bcf86cd799439011") } + @Test( + "EJSON.parse refuses a wrapper its constructor would refuse", + arguments: [ + #"{"n": {"$numberLong": "abc"}}"#, + #"{"n": {"$numberLong": "9223372036854775808"}}"#, + #"{"n": {"$numberDecimal": "abc"}}"#, + #"{"n": {"$numberInt": "12abc"}}"#, + #"{"n": {"$numberInt": "2147483648"}}"#, + #"{"n": {"$numberDouble": "abc"}}"#, + #"{"r": {"$regularExpression": {"pattern": "a", "options": "z"}}}"# + ] + ) + func ejsonParseChecksWrappers(json: String) throws { + let host = RecordingHost() + let context = try makeContext(host) + + context.evaluateScript("EJSON.parse(\(Self.javaScriptString(json)))") + #expect(context.exception != nil) + } + + @Test("EJSON.parse still reads every well-formed wrapper, NaN and Infinity included") + func ejsonParseReadsWellFormedWrappers() throws { + let host = RecordingHost() + let context = try makeContext(host) + let json = #"{"l": {"$numberLong": "9007199254740993"}, "d": {"$numberDecimal": "NaN"}, "# + + #""i": {"$numberInt": "-7"}, "f": {"$numberDouble": "-Infinity"}, "# + + #""r": {"$regularExpression": {"pattern": "^a", "options": "xi"}}}"# + + let value = context.evaluateScript(""" + var parsed = EJSON.parse(\(Self.javaScriptString(json))); + [String(parsed.l), String(parsed.d), parsed.i, parsed.f, parsed.r.options].join("|") + """) + #expect(context.exception == nil) + #expect(value?.toString() == "9007199254740993|NaN|-7|-Infinity|ix") + } + + @Test("A value the server sends is read back as stored, without the parse-time checks") + func serverRepliesAreNotRechecked() throws { + let host = RecordingHost() + host.replies = [ + "1", + #"{"docs": [{"_id": 1, "r": {"$regularExpression": {"pattern": "a", "options": "g"}}}], "done": true}"# + ] + let context = try makeContext(host) + + let value = context.evaluateScript("db.orders.findOne({}).r.options") + #expect(context.exception == nil) + #expect(value?.toString() == "g") + } + + private static func javaScriptString(_ text: String) -> String { + let data = (try? JSONSerialization.data(withJSONObject: [text])) ?? Data() + let array = String(data: data, encoding: .utf8) ?? "[]" + return String(array.dropFirst().dropLast()) + } + @Test("A write with no document is refused rather than sent as null") func writesNeedADocument() throws { let host = RecordingHost() diff --git a/TableProTests/Core/MongoDB/MongoShellValueTests.swift b/TableProTests/Core/MongoDB/MongoShellValueTests.swift new file mode 100644 index 000000000..715fc1b15 --- /dev/null +++ b/TableProTests/Core/MongoDB/MongoShellValueTests.swift @@ -0,0 +1,230 @@ +// +// MongoShellValueTests.swift +// TableProTests +// +// Expected values were measured against mongosh 2.10.0 on MongoDB 7.0.43. Where the shell refuses +// what mongosh wraps, clamps or zeroes, the test says so. +// + +import Foundation +import JavaScriptCore +import Testing + +/// Drives the real prelude's number rules and value constructors. +/// +/// A whole number past int64 went out as `$numberLong` in its shortest spelling, such as +/// `"9223372036854776000"`, which libbson refuses, so `insertOne({n: 1e20})` failed as a document +/// MongoDB could not read. `Int32`, `Long`, `Decimal128` and `BSONRegExp` were offered by +/// autocomplete and undefined, and a stored `NaN` decimal or an `x` regular expression stopped +/// `forEach` partway through. +struct MongoShellValueTests { + private typealias RecordingHost = MongoScriptPreludeTests.RecordingHost + + private func makeContext(_ host: RecordingHost = RecordingHost()) throws -> JSContext { + try MongoScriptContext.make(execute: { host.handle($0) }, emit: { host.record(printed: $0) }) + } + + private func extendedJson(of expression: String, in context: JSContext) -> String? { + context.exception = nil + let value = context.evaluateScript("EJSON.stringify(\(expression))") + return context.exception == nil ? value?.toString() : nil + } + + private func refusal(of statement: String, in context: JSContext) -> String? { + context.exception = nil + context.evaluateScript(statement) + let message = context.exception?.objectForKeyedSubscript("message")?.toString() + context.exception = nil + return message + } + + @Test("A whole number past 2^53 is sent as the double it is, never as a different int64") + func wholeNumbersPastTheExactRangeAreDoubles() throws { + let host = RecordingHost() + host.replies = [#"{"insertedIds": [1], "insertedCount": 1}"#] + let context = try makeContext(host) + + context.evaluateScript(""" + db.n.insertOne({a: 1e20, b: 9223372036854775807, c: -9223372036854775808, d: 1e21, e: -0, \ + f: 2147483648, g: 9007199254740993, h: 9007199254740991, i: -2147483648, j: 5}) + """) + #expect(context.exception == nil) + + let document = try #require(host.requests(op: "insertOne").first?["document"] as? String) + #expect(document == """ + {"a":{"$numberDouble":"100000000000000000000"},"b":{"$numberDouble":"9223372036854776000"},\ + "c":{"$numberDouble":"-9223372036854776000"},"d":{"$numberDouble":"1e+21"},\ + "e":{"$numberDouble":"-0.0"},"f":{"$numberLong":"2147483648"},\ + "g":{"$numberDouble":"9007199254740992"},"h":{"$numberLong":"9007199254740991"},\ + "i":{"$numberInt":"-2147483648"},"j":{"$numberInt":"5"}} + """) + #expect(Double("9223372036854776000") == 0x1p63) + #expect(Double("-9223372036854776000") == -0x1p63) + #expect(Double("9007199254740992") == 0x1p53) + } + + @Test("A Long holds exactly the integer it was given") + func longKeepsTheExactInteger() throws { + let context = try makeContext() + let expected: [(statement: String, digits: String)] = [ + ("NumberLong(2 ** 62)", "4611686018427387904"), + ("Long(2 ** 62)", "4611686018427387904"), + ("NumberLong(9007199254740993)", "9007199254740992"), + ("NumberLong(-(2 ** 63))", "-9223372036854775808"), + (#"Long("9223372036854775807")"#, "9223372036854775807"), + (#"NumberLong(" 5")"#, "5"), + (#"Long("+007")"#, "7"), + ("NumberLong(5.9)", "5"), + ("Long(-5.9)", "-5"), + ("Long(5, 1)", "4294967301"), + ("Long(-1, -1)", "-1"), + ("NumberLong(NumberInt(3))", "3"), + (#"Long(NumberLong("9007199254740993"))"#, "9007199254740993"), + ("NumberLong()", "0") + ] + for (statement, digits) in expected { + #expect( + extendedJson(of: statement, in: context) == #"{"$numberLong":"\#(digits)"}"#, + "\(statement)" + ) + } + } + + @Test("Int32, Decimal128, Timestamp, BSONRegExp and MinKey build the values mongosh builds") + func constructorsMatchMongosh() throws { + let context = try makeContext() + let expected: [(statement: String, json: String)] = [ + ("Int32(5)", #"{"$numberInt":"5"}"#), + (#"new Int32("7")"#, #"{"$numberInt":"7"}"#), + ("NumberInt(5.9)", #"{"$numberInt":"5"}"#), + ("NumberInt(0.0000005)", #"{"$numberInt":"0"}"#), + (#"NumberInt(" 5 ")"#, #"{"$numberInt":"5"}"#), + (#"Decimal128("1.50")"#, #"{"$numberDecimal":"1.50"}"#), + (#"NumberDecimal("NaN")"#, #"{"$numberDecimal":"NaN"}"#), + (#"Decimal128("-inf")"#, #"{"$numberDecimal":"-Infinity"}"#), + ("NumberDecimal(1.5)", #"{"$numberDecimal":"1.5"}"#), + ("Timestamp(1, 2)", #"{"$timestamp":{"t":1,"i":2}}"#), + ("Timestamp({t: 3, i: 4})", #"{"$timestamp":{"t":3,"i":4}}"#), + (#"Timestamp(NumberLong("8589934593"))"#, #"{"$timestamp":{"t":2,"i":1}}"#), + ("Timestamp(1.5, 2.7)", #"{"$timestamp":{"t":1,"i":2}}"#), + ("Timestamp(1)", #"{"$timestamp":{"t":1,"i":0}}"#), + ("Timestamp()", #"{"$timestamp":{"t":0,"i":0}}"#), + (#"BSONRegExp("a", "mi")"#, #"{"$regularExpression":{"pattern":"a","options":"im"}}"#), + ("MinKey()", #"{"$minKey":1}"#), + ("new MinKey()", #"{"$minKey":1}"#) + ] + for (statement, json) in expected { + #expect(extendedJson(of: statement, in: context) == json, "\(statement)") + } + } + + @Test("A constructor refuses a value it cannot hold instead of wrapping, clamping or zeroing it") + func constructorsRefuseWhatTheyCannotHold() throws { + let context = try makeContext() + let int32Range = "takes a whole number from -2147483648 to 2147483647" + let int64Range = "takes a whole number from -9223372036854775808 to 9223372036854775807" + let notTimestamp = "Timestamp takes (t, i), { t, i }, a Long or a bigint" + let expected: [(statement: String, message: String)] = [ + ("Int32(2147483648)", "Int32 \(int32Range)"), + (#"NumberInt("12abc")"#, "NumberInt takes a number"), + ("NumberInt(NaN)", "NumberInt takes a number"), + (#"Long("9223372036854775808")"#, "Long \(int64Range)"), + ("NumberLong(1e20)", "NumberLong \(int64Range)"), + ("NumberLong(9223372036854775807)", "NumberLong \(int64Range)"), + (#"NumberLong("1e3")"#, "NumberLong takes a whole number"), + ("Long(5.5, 1)", "Long takes its low and high halves as whole numbers from -2147483648 to 4294967295"), + (#"Decimal128("abc")"#, "Decimal128 takes a number"), + ("Timestamp(4294967296, 0)", "Timestamp takes t from 0 to 4294967295"), + ("Timestamp(-1, 0)", "Timestamp takes t from 0 to 4294967295"), + (#"Timestamp("5", "6")"#, "Timestamp takes t as a number"), + ("Timestamp(NumberInt(5), NumberInt(6))", "Timestamp takes t as a number"), + ("Timestamp(Double(5), 1)", "Timestamp takes t as a number"), + ("Timestamp(new Date(1700000000000), 1)", "Timestamp takes t as a number"), + ("Timestamp({})", "Timestamp takes t as a number"), + ("Timestamp({t: 1})", "Timestamp takes i as a number"), + ("Timestamp([1, 2])", notTimestamp), + ("Timestamp(null)", notTimestamp), + (#"BSONRegExp("a", "g")"#, "BSONRegExp takes options from i, l, m, s, u and x"), + ("BSONRegExp(/a/)", "BSONRegExp takes its pattern as a string"), + (#"BSONRegExp("a\u0000")"#, "A BSONRegExp pattern cannot hold a null character") + ] + for (statement, message) in expected { + #expect(refusal(of: statement, in: context) == message, "\(statement)") + } + } + + @Test("A legacy name and its type are one type, whichever built the value") + func aliasesShareOneType() throws { + let context = try makeContext() + + let shared = context.evaluateScript(""" + [NumberInt(5) instanceof Int32, Int32(5) instanceof NumberInt, + NumberLong(5) instanceof Long, Long(5) instanceof NumberLong, + NumberDecimal("1") instanceof Decimal128, Decimal128("1") instanceof NumberDecimal, + NumberLong(5).constructor === Long, NumberInt(5).constructor === Int32].join() + """) + #expect(context.exception == nil) + #expect(shared?.toString() == "true,true,true,true,true,true,true,true") + } + + @Test("A document read and written back keeps what JavaScript can hold, and reads through the rest") + func readBackRoundTrip() throws { + let host = RecordingHost() + host.replies = [ + "1", + """ + {"docs": [{"_id": {"$numberInt": "1"}, "d": {"$numberDouble": "5.0"}, \ + "wide": {"$numberDouble": "3000000000.0"}, "neg": {"$numberDouble": "-0.0"}, \ + "big": {"$numberDouble": "1e+20"}, "nan": {"$numberDouble": "NaN"}, \ + "l": {"$numberLong": "9007199254740993"}, "dec": {"$numberDecimal": "NaN"}, \ + "x": {"$regularExpression": {"pattern": "a b", "options": "x"}}, \ + "pcre": {"$regularExpression": {"pattern": "(?i)a", "options": ""}}, \ + "slash": {"$regularExpression": {"pattern": "a/b", "options": "i"}}, \ + "empty": {"$regularExpression": {"pattern": "", "options": ""}}, \ + "ts": {"$timestamp": {"t": 4294967295, "i": 1}}}], "done": true} + """, + #"{"n": 1, "nModified": 1}"# + ] + let context = try makeContext(host) + + context.evaluateScript("db.t.find().forEach(function (doc) { db.t.replaceOne({_id: doc._id}, doc); })") + #expect(context.exception == nil) + + let replacement = try #require(host.requests(op: "replace").first?["update"] as? String) + #expect(replacement == """ + {"_id":{"$numberInt":"1"},"d":{"$numberInt":"5"},"wide":{"$numberLong":"3000000000"},\ + "neg":{"$numberDouble":"-0.0"},"big":{"$numberDouble":"100000000000000000000"},\ + "nan":{"$numberDouble":"NaN"},"l":{"$numberLong":"9007199254740993"},\ + "dec":{"$numberDecimal":"NaN"},\ + "x":{"$regularExpression":{"pattern":"a b","options":"x"}},\ + "pcre":{"$regularExpression":{"pattern":"(?i)a","options":""}},\ + "slash":{"$regularExpression":{"pattern":"a/b","options":"i"}},\ + "empty":{"$regularExpression":{"pattern":"","options":""}},\ + "ts":{"$timestamp":{"t":4294967295,"i":1}}} + """) + } + + @Test("A value read back compares the way it does in mongosh") + func readBackComparisons() throws { + let host = RecordingHost() + host.replies = [ + "1", + """ + {"docs": [{"d": {"$numberDouble": "5.0"}, "neg": {"$numberDouble": "-0.0"}, \ + "i": {"$numberInt": "7"}, "l": {"$numberLong": "5"}, "dec": {"$numberDecimal": "1.50"}, \ + "slash": {"$regularExpression": {"pattern": "a/b", "options": "i"}}, \ + "x": {"$regularExpression": {"pattern": "a b", "options": "x"}}}], "done": true} + """ + ] + let context = try makeContext(host) + + let seen = context.evaluateScript(""" + var doc = db.t.findOne(); + [doc.d === 5, typeof doc.d, Object.is(doc.neg, -0), doc.i === 7, + doc.l === 5, doc.l == 5, doc.l instanceof Long, String(doc.dec), + doc.slash instanceof RegExp, doc.slash.test("A/B"), doc.x instanceof BSONRegExp, doc.x.options].join() + """) + #expect(context.exception == nil) + #expect(seen?.toString() == "true,number,true,true,false,true,true,1.50,true,true,true,x") + } +} diff --git a/docs/databases/mongodb.mdx b/docs/databases/mongodb.mdx index 54fc1cb7e..30cce1bcd 100644 --- a/docs/databases/mongodb.mdx +++ b/docs/databases/mongodb.mdx @@ -196,6 +196,48 @@ print("stale orders: " + stale.length); returns no documents. A statement that returns documents shows those instead, with the printed lines on the status line under the grid. +### Values + +A number written without a constructor is stored by what it holds: + +| Number | Stored as | +|---|---| +| Whole, from -2147483648 to 2147483647 | int32 | +| Whole, outside int32 up to 9007199254740991 (2^53 - 1) | int64 | +| Whole past 2^53, a fraction, `-0`, `NaN`, `Infinity` | double | + +mongosh stores every whole number outside int32 as a double; write `Double(3000000000)` for the +same result here. Past 2^53 a JavaScript number cannot hold every integer, so `9007199254740993` +is already `9007199254740992` when the statement runs. Pass an exact 64-bit integer as a string: +`Long("9007199254740993")`. Because `-0` is a double, `{$inc: {n: Math.round(-0.2)}}` turns an +int32 field into a double, the same as in mongosh. + +| Constructor | Stores | Takes | +|---|---|---| +| `Int32`, `NumberInt` | int32 | A number or numeric string, cut toward zero | +| `Long`, `NumberLong` | int64 | A string of digits, a number cut toward zero, or `Long(low, high)` from two 32-bit halves | +| `Double` | double | A number | +| `Decimal128`, `NumberDecimal` | decimal128 | A numeric string, `NaN`, `Infinity` or `-Infinity` | +| `Timestamp` | timestamp | `(t, i)`, `{t, i}` or a `Long`, each part from 0 to 4294967295 | +| `BSONRegExp` | regular expression | A pattern string, and options from `i`, `l`, `m`, `s`, `u` and `x` | + +A value out of range throws when the constructor runs. `Int32(2147483648)` and +`NumberLong("9223372036854775808")` throw where mongosh wraps them round to a negative number. + +A document read into a script holds these: + +| Stored | Reads as | Compares | +|---|---|---| +| double | a number | A stored `5.0` makes `doc.qty === 5` true | +| int32 | a number | A stored `5` makes `doc.qty === 5` true | +| int64 | a `Long` | `doc.views == 5` is true and `doc.views === 5` is false; `.toNumber()` gives a number | +| decimal128 | a `Decimal128` | `String(doc.price)` gives the stored digits | +| regular expression | a `RegExp`, or a `BSONRegExp` for the `l` and `x` options and PCRE-only syntax such as `(?i)` | `.test()` works on a `RegExp` | + +Written back, a whole double is stored by the rules for a bare number: `5.0` becomes int32, as in +mongosh, and `3000000000.0` becomes int64 where mongosh keeps a double. Wrap it in `Double()` to +keep it a double. A regular expression goes back exactly as the server sent it, options included. + ### Collection references `db.users`, `db["users"]`, or `db.getCollection("users")`. Use `getCollection` for names with dots @@ -264,6 +306,7 @@ New connections default to **Disabled**, and the driver has no TLS fallback: **P - A `Code` scope holding an object that opens with `$type`, `$regex` or `$options`, such as `Code("f", {a: {$type: "binData"}})`, fails with `This is not a document MongoDB can read`. List another key of that object first. - A script that loops without touching the database cannot be stopped: JavaScriptCore has no public way to interrupt one. `Cmd+.` stops anything that reads, writes or prints, which covers every query. A script silent for 120 seconds is abandoned and the shell restarts. - Field names that are whole numbers up to 4294967294 (`"0"`, `"12"`) sort ahead of the rest in a document literal, which is what JavaScript does with them. A nested object with such a key after another key, or with a key named `__proto__`, refuses the save when it is duplicated or written whole. Use **Insert Document…** or a query for it. +- `Decimal128("…")` with more than 34 significant digits, or an exponent outside the decimal128 range, fails when the statement runs with "This is not a document MongoDB can read". Round a long value to 34 digits. For an exponent out of range, store the value scaled to a unit that fits, or as a double or a string. - An MQL export of a view holds the view's documents, not the view, so restoring the file creates a collection of that name. Drop that collection and run the view's **Show DDL** text to get the view back. - A time-series collection takes inserts and deletes from the grid, but refuses an edited cell and a rename, and the server's error is shown. Change its documents from a query tab with `updateMany` filtered on the meta field. - A filter or validator with a regular expression under `$regex`, such as `{email: {$regex: /@/i}}`, is refused as a document MongoDB cannot read. Write `{email: /@/i}` or `{email: {$regex: "@", $options: "i"}}` instead. **Show DDL** writes such a validator the way the server holds it, which runs in mongosh but not in a query tab.