🤖 Name your tables on every engine: table_names across the multi-engine backends

README.md

@@ -77,6 +77,9 @@ rel = con.sql('SELECT time, AVG("air") AS air FROM air GROUP BY time ORDER BY ti
 xql.to_dataset(rel, template=ds)   # any engine's Arrow result round-trips
 ```
 
+`table_names` (below) works the same way on every engine, so a query written
+against `era5.surface` is not tied to the engine it was written for.
+
 See [Engines](https://xqlsystems.github.io/xarray-sql/latest/engines/) for the support matrix, DuckDB/Polars details,
 and the lazy chunked round-trip.
 

benchmarks/geospatial/_engines.py

@@ -18,9 +18,11 @@
 :meth:`EngineContext.sql_to_dataset`. On the ``datafusion`` path this
 is byte-for-byte the original behavior (``from_dataset`` + ``sql`` +
 ``XarrayDataFrame.to_dataset``); the other engines register one pyarrow
-dataset per dimension group under flattened table names
-(``era5.surface`` → ``era5_surface`` — rewritten in the SQL text) and the
-result rows are round-tripped to an ``xr.Dataset`` through pandas.
+dataset per dimension group under the flat table names
+``xql.arrow_datasets`` returns (``era5.surface`` → ``era5_surface`` —
+rewritten in the SQL text, since these paths register frames one at a
+time rather than through a schema) and the result rows are
+round-tripped to an ``xr.Dataset`` through pandas.
 
 The DataFusion-only UDF cases (07 and the UDF half of 09) build
 ``xql.XarrayContext`` directly rather than through this layer; the suite
@@ -47,24 +49,6 @@ def engine_name() -> str:
     return engine
 
 
-def _group_tables(name, ds, table_names):
-    """Split ``ds`` into per-dimension-group tables like XarrayContext does.
-
-    Returns ``[(flat_name, dotted_name, sub_dataset)]``; a uniform dataset
-    keeps its plain name (flat == dotted == name).
-    """
-    groups: dict[tuple, list] = {}
-    for var, v in ds.data_vars.items():
-        groups.setdefault(tuple(v.dims), []).append(var)
-    if len(groups) == 1:
-        return [(name, name, ds)]
-    out = []
-    for dims, variables in groups.items():
-        sub = (table_names or {}).get(dims) or "_".join(dims)
-        out.append((f"{name}_{sub}", f"{name}.{sub}", ds[variables]))
-    return out
-
-
 def _literal(value: Any) -> str:
     """Render a parameter value as a SQL literal (for engines without binds)."""
     if isinstance(value, (datetime.datetime, pd.Timestamp, np.datetime64)):
@@ -134,15 +118,16 @@ def from_dataset(self, name, ds, *, chunks=None, table_names=None):
         """Register ``ds`` as SQL table(s), mirroring XarrayContext naming."""
         import xarray_sql as xql
 
-        for flat, dotted, sub in _group_tables(name, ds, table_names):
-            if dotted != flat:
-                self._renames[dotted] = flat
-            sub_chunks = (
-                {d: c for d, c in chunks.items() if d in sub.dims}
-                if isinstance(chunks, dict)
-                else chunks
-            ) or None
-            self._register(flat, xql.arrow_dataset(sub, sub_chunks))
+        tables = xql.arrow_datasets(
+            ds, name, chunks=chunks, table_names=table_names
+        )
+        for flat, dataset in tables.items():
+            if flat != name:
+                # `era5_surface` here is `era5.surface` in the case's SQL,
+                # which was written against XarrayContext's schema split.
+                group = flat[len(name) + 1 :]
+                self._renames[f"{name}.{group}"] = flat
+            self._register(flat, dataset)
 
     # -- querying ----------------------------------------------------------
 

docs/engines.md

@@ -90,6 +90,27 @@ exact expression via pyarrow — pruning is only an optimization on top.
 `XarrayArrowStream`, the dependency-light re-scannable C-stream wrapper
 without pushdown, remains available as a fallback.
 
+As is standard in for all Xarray-SQL engines, users may provide a mapping of
+groups of dimensions to their preferred table names, like so:
+
+```python
+xql.register(con, "era5", ds, table_names={
+  ("time", "latitude", "longitude"): "surface",
+  ("time", "level", "latitude", "longitude"): "atmosphere",
+})
+
+con.sql("SELECT AVG(temperature) FROM era5.atmosphere WHERE level = 500")
+```
+
+Mixed-dimension Datasets split as they do everywhere else, with one
+DuckDB-specific wrinkle: `con.register` can only place an object in
+DuckDB's temporary namespace, so each group is registered flat as
+`era5_surface` and mirrored as a view `era5.surface` in a schema of its
+own. Both spellings hit the same scan — pushdown and projection travel
+through the view — and the dotted one is what keeps the SQL portable.
+A read-only connection cannot create the schema; registration then
+warns and leaves the flat tables.
+
 Details that matter in production:
 
 - **Finely partitioned axes** (e.g. hourly-chunked reanalysis time with
@@ -146,6 +167,19 @@ out = (
 xql.to_dataset(out, template=ds)   # polars frames speak Arrow PyCapsule
 ```
 
+`arrow_dataset` wants a Dataset whose variables share one set of
+dimensions; `xql.arrow_datasets(ds, "era5", table_names=...)` splits a
+mixed-dimension one and hands back the tables named, reading the shared
+dimension coordinates once for all of them.
+
+```python
+tables = xql.arrow_datasets(ds, "era5", table_names={...})
+
+ctx = pl.SQLContext()
+for table, dataset in tables.items():     # 'era5_surface', ...
+  ctx.register(table, pl.scan_pyarrow_dataset(dataset))
+```
+
 Polars pushes its predicate and column selection into the dataset scan
 (verified: a filtered group-by read 1 of 20 chunks and 3 of 5 columns),
 and its results round-trip through `xql.to_dataset` unchanged. The
@@ -166,7 +200,8 @@ What each integration provides. Known issues and constraints live on
 | Eager round-trip (`xql.to_dataset`) | yes | yes | yes |
 | Chunked round-trip (`chunks=`) | re-execution | `spill=True` [^spill-only] | re-execution (streaming engine) |
 | `geometry` column ([geospatial](geospatial.md#geoarrow-point-geometry-columns)) | annotated WKB passes through | native `GEOMETRY` (`"wkb"` encoding) | plain binary/struct |
-| Mixed-dimension datasets | one schema, `name.group` tables | `<name>_<dims>` tables | filter `data_vars` before `arrow_dataset` |
+| Mixed-dimension datasets | one schema, `name.group` tables | `name.group` views over `name_group` tables | `xql.arrow_datasets(ds, name)`, one per group |
+| Naming those tables (`table_names=`) | yes | yes | yes |
 | Version floor | bundled (core dependency) | `duckdb >= 1.4` (tested on 1.5) | tested on `polars 1.42` |
 
 [^spill-only]: Why DuckDB relations do not re-execute — and two other

docs/examples.md

@@ -107,7 +107,11 @@ ctx.sql('''
 
 If you omit `table_names`, each table is named by joining its dimension names
 with underscores, e.g. `era5.time_latitude_longitude` and
-`era5.time_level_latitude_longitude`.
+`era5.time_level_latitude_longitude`. Keys naming a dimension group the Dataset
+does not have are ignored, so one naming map can be reused across Datasets
+holding different subsets of the same variables. The keyword is not
+DataFusion-only — `xql.register` takes it on every engine (see
+[the same tables on DuckDB and Polars](#the-same-tables-on-duckdb-and-polars)).
 
 ## GOES satellite imagery (scalar variables)
 
@@ -154,5 +158,44 @@ not DataFusion-specific: `xql.register(con, name, ds)` attaches the same lazy,
 pushdown-scanned table to a DuckDB connection, and
 `pl.scan_pyarrow_dataset(xql.arrow_dataset(ds))` serves Polars — same
 splitting rules for mixed-dimension Datasets, same round-trip through
-`xql.to_dataset(result, template=ds)`. See [Engines](engines.md) for the
-support matrix and per-engine details.
+`xql.to_dataset(result, template=ds)`.
+
+`table_names` travels with them, so the ERA5 example above changes engine
+without changing a word of its SQL:
+
+```python
+import duckdb
+
+con = duckdb.connect()
+xql.register(con, 'era5', ds, table_names={
+    ('time', 'latitude', 'longitude'): 'surface',
+    ('time', 'level', 'latitude', 'longitude'): 'atmosphere',
+})
+
+rel = con.sql('''
+  SELECT level, AVG(temperature) - 273.15 AS avg_c
+  FROM era5.atmosphere
+  WHERE time BETWEEN TIMESTAMP '2020-01-01'
+                 AND TIMESTAMP '2020-01-01 05:00:00'
+  GROUP BY level
+  ORDER BY level DESC
+''')
+xql.to_dataset(rel, template=ds, dims=['level'])
+```
+
+Polars registers table by table, so it takes the split datasets by name:
+
+```python
+import polars as pl
+
+tables = xql.arrow_datasets(ds, 'era5', table_names={
+    ('time', 'latitude', 'longitude'): 'surface',
+    ('time', 'level', 'latitude', 'longitude'): 'atmosphere',
+})
+
+ctx = pl.SQLContext()
+for table, dataset in tables.items():   # 'era5_surface', 'era5_atmosphere'
+    ctx.register(table, pl.scan_pyarrow_dataset(dataset))
+```
+
+See [Engines](engines.md) for the support matrix and per-engine details.

docs/limitations.md

@@ -144,12 +144,14 @@ A materialized table or bare C-stream has no query behind it, so the
 re-execution form of `chunks=` cannot serve it — `spill=True` (one
 pass to a temporary Parquet file) is the chunked path for these.
 
-### Mixed-dimension datasets split into one DuckDB table per dim group
-
-DuckDB registration has no schema namespace, so variables with
-different dims land in suffixed tables (`<name>_<dims>`), sharing one
-set of coordinate reads. DataFusion registers the same layout as
-`name.group` tables inside one schema.
+### A read-only DuckDB connection loses the dotted table names
+
+Mixed-dimension Datasets split into one table per dimension group on
+every engine, addressed as `name.group`. On DuckDB that dotted spelling
+is a view in a schema, because `con.register` reaches only the
+temporary namespace — and creating a schema needs a writable catalog.
+Registering on a read-only connection therefore warns and leaves the
+flat `name_group` tables, which are always registered and always work.
 
 ### Pointwise indexers on lazy round-trip arrays are slower