You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
SQLite treats most of its keywords as non-reserved: they act as keywords
where the grammar expects one and as ordinary identifiers everywhere else. We
reserve all 146 words in our keyword table unconditionally, so any of them
used as a column, table or alias name is a syntax error.
This blocks the SQE integration outright, independently of the embedding
API. SQE's iceberg_namespace_properties maps a namespace property key to
a value. Measured against the pinned 3.53.4 oracle:
We cannot create that table, and cannot read one the oracle created. No amount
of facade work (#695) changes that.
Scope: 89 of our 146 keywords
Tested every word in src/parser/tokenizer.rs's keyword table as a column
name, in both engines:
count
oracle accepts, we accept
0
oracle rejects, we reject
57
oracle accepts, we reject
89
Every keyword we tokenize is fully reserved. SQLite reserves 57.
ABORT ACTION AFTER ALWAYS ANALYZE ASC ATTACH BEFORE BEGIN BY CASCADE CAST
COLUMN CONFLICT CROSS CURRENT CURRENT_DATE CURRENT_TIME CURRENT_TIMESTAMP
DATABASE DEFERRED DESC DETACH DO EACH END EXCLUDE EXCLUSIVE EXPLAIN FAIL
FILTER FIRST FOLLOWING FOR FULL GENERATED GLOB GROUPS IF IGNORE IMMEDIATE
INDEXED INITIALLY INNER INSTEAD KEY LAST LEFT LIKE MATCH MATERIALIZED
NATURAL NO NULLS OF OFFSET OTHERS OUTER OVER PARTITION PLAN PRAGMA
PRECEDING QUERY RAISE RANGE RECURSIVE REGEXP REINDEX RELEASE RENAME
REPLACE RESTRICT RIGHT ROLLBACK ROW ROWS SAVEPOINT TEMP TEMPORARY TIES
TRIGGER UNBOUNDED VACUUM VIEW VIRTUAL WINDOW WITH WITHOUT
KEY, VALUE(-adjacent), MATCH, FIRST, LAST, ROW, RANGE, FILTER
— these are ordinary column names in real schemas, not exotic edge cases.
The mechanism is explicit in the pinned grammar
This is not something to infer. parse.y:272 (3.53.4, the pinned oracle)
declares it directly:
%fallback ID
ABORT ACTION AFTER ANALYZE ASC ATTACH BEFORE BEGIN BY CASCADE CAST COLUMNKW
CONFLICT DATABASE DEFERRED DESC DETACH DO
EACH END EXCLUSIVE EXPLAIN FAIL FOR
IGNORE IMMEDIATE INITIALLY INSTEAD LIKE_KW MATCH NO PLAN
QUERY KEY OF OFFSET PRAGMA RAISE RECURSIVE RELEASE REPLACE RESTRICT ROW ROWS
ROLLBACK SAVEPOINT TEMP TRIGGER VACUUM VIEW VIRTUAL WITH WITHOUT
NULLS FIRST LAST
...
Lemon's %fallback means: when the parser cannot shift this token, retry it
as ID. 85 tokens are listed (more under %ifndef SQLITE_OMIT_WINDOWFUNC
etc.), which matches the 89 measured above once token aliases like COLUMNKW/LIKE_KW are expanded.
Our tokenizer has no equivalent concept — ("KEY", Keyword::KEY)
(tokenizer.rs:523) is unconditional.
Two ways to fix it, and they are not equivalent
A. Fallback at the parser. Mirror Lemon: where a rule wants an identifier,
accept a fallback-eligible keyword token too. Faithful to SQLite, because it
keeps the word working as a keyword in keyword position — SELECT key FROM t ORDER BY key and PRIMARY KEY(a) both keep working in the same statement.
Touches every identifier-accepting production.
B. Fallback at the tokenizer. Emit Identifier instead of Keyword for
the 89. Much smaller, and wrong: PRIMARY KEY(a) would stop parsing, because KEY there is a keyword.
So A, despite the size. The grammar convention applies — .openspec/grammar/sqlite.ebnf needs the fallback set expressed, with make check-grammar-drift clean afterwards.
Non-goals
Quoted identifiers ("key", [key], `key`) already work and are not
affected.
The 57 genuinely-reserved keywords stay reserved. CREATE TABLE t(select)
must keep failing, as it does in the oracle.
Making sqlite.ebnf model Lemon's conflict resolution generally. Only the
fallback set.
Acceptance Criteria
All 89 words above work as column names, table names and aliases, with
the same statements accepted as the pinned 3.53.4 oracle accepts
The 57 reserved words still fail, matching the oracle
SQE's actual table works end to end: CREATE TABLE p(namespace TEXT, key TEXT, value TEXT, PRIMARY KEY(namespace, key)),
then INSERT, then SELECT key, value FROM p — byte-identical to the
oracle
A keyword used as both in one statement still parses: SELECT key FROM t ORDER BY key and a table declared with PRIMARY KEY(key)
Corpus test enumerating the fallback set against the oracle, so the
set cannot silently drift from parse.y
sqlite.ebnf expresses the fallback set with a [parse.y:272 fallback]
origin annotation; make check-grammar-drift clean
Full suite green, make lint both clippy passes, make assurance with
no dead links
Complexity
Estimate: medium Reasoning: The set is not guesswork — parse.y:272 enumerates it and the
89 are measured, so there is no discovery left. The work is threading
fallback acceptance through the parser's identifier positions (option A), plus
the grammar annotation. The risk is regression in keyword position — a
fallback that is too eager turns PRIMARY KEY(a) or ORDER BY x DESC into a
parse of something else, which is what the both-in-one-statement criterion
exists to catch.
Found while writing tests for #695: a test using AS first failed, which
turned out not to be a bad test.
Description
SQLite treats most of its keywords as non-reserved: they act as keywords
where the grammar expects one and as ordinary identifiers everywhere else. We
reserve all 146 words in our keyword table unconditionally, so any of them
used as a column, table or alias name is a syntax error.
This blocks the SQE integration outright, independently of the embedding
API. SQE's
iceberg_namespace_propertiesmaps a namespace property key toa value. Measured against the pinned 3.53.4 oracle:
We cannot create that table, and cannot read one the oracle created. No amount
of facade work (#695) changes that.
Scope: 89 of our 146 keywords
Tested every word in
src/parser/tokenizer.rs's keyword table as a columnname, in both engines:
Every keyword we tokenize is fully reserved. SQLite reserves 57.
KEY,VALUE(-adjacent),MATCH,FIRST,LAST,ROW,RANGE,FILTER— these are ordinary column names in real schemas, not exotic edge cases.
The mechanism is explicit in the pinned grammar
This is not something to infer.
parse.y:272(3.53.4, the pinned oracle)declares it directly:
Lemon's
%fallbackmeans: when the parser cannot shift this token, retry itas
ID. 85 tokens are listed (more under%ifndef SQLITE_OMIT_WINDOWFUNCetc.), which matches the 89 measured above once token aliases like
COLUMNKW/LIKE_KWare expanded.Our tokenizer has no equivalent concept —
("KEY", Keyword::KEY)(
tokenizer.rs:523) is unconditional.Two ways to fix it, and they are not equivalent
A. Fallback at the parser. Mirror Lemon: where a rule wants an identifier,
accept a fallback-eligible keyword token too. Faithful to SQLite, because it
keeps the word working as a keyword in keyword position —
SELECT key FROM t ORDER BY keyandPRIMARY KEY(a)both keep working in the same statement.Touches every identifier-accepting production.
B. Fallback at the tokenizer. Emit
Identifierinstead ofKeywordforthe 89. Much smaller, and wrong:
PRIMARY KEY(a)would stop parsing, becauseKEYthere is a keyword.So A, despite the size. The grammar convention applies —
.openspec/grammar/sqlite.ebnfneeds the fallback set expressed, withmake check-grammar-driftclean afterwards.Non-goals
"key",[key],`key`) already work and are notaffected.
CREATE TABLE t(select)must keep failing, as it does in the oracle.
sqlite.ebnfmodel Lemon's conflict resolution generally. Only thefallback set.
Acceptance Criteria
the same statements accepted as the pinned 3.53.4 oracle accepts
CREATE TABLE p(namespace TEXT, key TEXT, value TEXT, PRIMARY KEY(namespace, key)),then
INSERT, thenSELECT key, value FROM p— byte-identical to theoracle
SELECT key FROM t ORDER BY keyand a table declared withPRIMARY KEY(key)set cannot silently drift from
parse.ysqlite.ebnfexpresses the fallback set with a[parse.y:272 fallback]origin annotation;
make check-grammar-driftcleanmake lintboth clippy passes,make assurancewithno dead links
Complexity
Estimate: medium
Reasoning: The set is not guesswork —
parse.y:272enumerates it and the89 are measured, so there is no discovery left. The work is threading
fallback acceptance through the parser's identifier positions (option A), plus
the grammar annotation. The risk is regression in keyword position — a
fallback that is too eager turns
PRIMARY KEY(a)orORDER BY x DESCinto aparse of something else, which is what the both-in-one-statement criterion
exists to catch.
Found while writing tests for #695: a test using
AS firstfailed, whichturned out not to be a bad test.
Refs: 002/Req-2, #678, #695