Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) by MisterRaindrop · Pull Request #1842 · apache/cloudberry

MisterRaindrop · 2026-07-06T07:52:47Z

Summary

Kernel-side DDL scaffolding for Iceberg lake tables, per the design proposal in #1683 (this PR is the "core syntax" milestone of the roadmap posted there). It adds the system catalogs, parse nodes, grammar, commands, and client-tool awareness that a datalake provider extension builds on. No table AM implementation or provider is included — those are later milestones (agent / FDW phases in the roadmap).

CREATE FOREIGN CATALOG hive_cat SERVER hive_cat_srv OPTIONS (type 'hive', uri 'thrift://...');
CREATE FOREIGN VOLUME s3_vol SERVER s3_srv OPTIONS (path 's3://bucket/prefix');
CREATE ICEBERG TABLE t (a int, b text) CATALOG hive_cat VOLUME s3_vol OPTIONS (fileformat 'parquet');
DROP CATALOG hive_cat;  DROP VOLUME s3_vol;

What's included (structured for commit-by-commit review)

catalog: pg_foreign_catalog, pg_foreign_volume (metadata service / storage location handles, hanging off pg_foreign_server), pg_lake_table (per-relation lake metadata keyed by relid). All three have TOAST tables; name indexes are single-column unique (see "Design decisions").
nodes: CreateLakeTableStmt (embeds CreateStmt), CreateForeignCatalogStmt, CreateForeignVolumeStmt + hand-maintained node-support functions.
parser: CREATE ICEBERG TABLE ... CATALOG ... VOLUME ... OPTIONS (...), CREATE FOREIGN CATALOG|VOLUME ... SERVER ..., DROP CATALOG|VOLUME via drop_type_name.
commands (foreign catalog/volume): create/drop with IF [NOT] EXISTS, dependencies, ownership, object addressing / COMMENT / event-trigger integration, syscaches, QD→QE dispatch with synchronized OIDs.
commands (CREATE ICEBERG TABLE): resolves catalog/volume (explicit clause or iceberg_default_catalog/iceberg_default_volume GUCs), validates the statement uses the iceberg AM, creates the relation + TOAST + pg_lake_table row + dependencies; guards ALTER TABLE SET ACCESS METHOD / SET DISTRIBUTED BY both directions; lake tables are forced DISTRIBUTED RANDOMLY.
bin: pg_dump skips iceberg tables (by AM name, with a warning — same pattern as PAX); psql tab completion.
tests: lake_table regression in greenplum_schedule (DDL lifecycle, duplicate/USING rejection, TOAST wide rows, ownership, dependency/CASCADE, QD/QE dispatch checks) + catalog-expected refresh across regress, singlenode_regress, and pax copies.

Design decisions to review

Provider decoupling: the kernel resolves the AM strictly by name (get_table_am_oid("iceberg")); no hardcoded AM OID and no extension-name checks. Everything degrades gracefully (with a hint) when no provider is installed.
Provider integration contract: CreateLakeTable() runs after DefineRelation() and ends with CommandCounterIncrement() + InvokeObjectPostCreateHook(LakeTableRelationId, ...) — a provider performs remote Iceberg creation from that object-access hook (on QD and QEs), where catalog/volume/options metadata is already visible. relation_set_new_filelocator remains local-storage-only.
Catalog/volume names are database-global (single-column unique index), like pg_foreign_server — every reference syntax (DROP CATALOG x, the CATALOG x clause, the GUCs) identifies them by bare name, so the uniqueness scope matches what the syntax can express. (The alternative — per-server names like user mappings — would require server-qualified reference syntax everywhere.)
CREATE ICEBERG TABLE rejects a USING clause naming any other AM; without it the statement implies USING iceberg.

Known limitations / open questions (deliberately out of scope here)

pg_dump / pg_upgrade: FOREIGN CATALOG/VOLUME objects and pg_lake_table metadata are not dumped yet; iceberg tables are skipped with a warning (PAX precedent). Restore/upgrade support is planned as a follow-up.
Permission model: creating a catalog/volume requires USAGE on its server. Referencing one from CREATE ICEBERG TABLE currently checks existence only — whether that should require USAGE on the underlying server, or dedicated ACLs (GRANT USAGE ON CATALOG), is an open question we'd like reviewer input on.
The GUC defaults are synced to QEs and behave like default_tablespace w.r.t. objects dropped after SET.

Discussion: #1683

Introduce the three system catalogs backing Iceberg lake-table DDL: - pg_foreign_catalog: named foreign catalog bound to a foreign server - pg_foreign_volume: named foreign volume bound to a foreign server - pg_lake_table: per-relation lake-table metadata (type, catalog, volume, options) Register the headers in the catalog Makefile and bump CATALOG_VERSION_NO. No code references the catalogs yet; DDL/commands land in later commits. OIDs 8549-8558 / 9901-9902 verified free via unused_oids; duplicate_oids clean.

Add three statement parse nodes and their hand-maintained node-support plumbing (copy/equal/out/read + fast serialization), mirroring the existing CreateDirectoryTableStmt and CreateForeignServerStmt patterns: - CreateLakeTableStmt (CREATE ICEBERG TABLE ...; embeds CreateStmt) - CreateForeignCatalogStmt (CREATE FOREIGN CATALOG ...) - CreateForeignVolumeStmt (CREATE FOREIGN VOLUME ...) Nodes are not yet produced by the grammar or dispatched; grammar and command handling land in later commits. ObjectType additions are deferred to the command commit to keep exhaustive switches complete.

Add the ICEBERG and VOLUME unreserved keywords and the CREATE-side grammar productions that build the parse nodes from the previous commit: - CREATE ICEBERG TABLE name (cols) [FOREIGN CATALOG c] [FOREIGN VOLUME v] OPTIONS (...) -> CreateLakeTableStmt (forced DISTRIBUTED RANDOMLY) - CREATE FOREIGN CATALOG name SERVER s OPTIONS (...) -> CreateForeignCatalogStmt - CREATE FOREIGN VOLUME name SERVER s OPTIONS (...) -> CreateForeignVolumeStmt Statements parse but are not yet dispatched; command handling, ObjectType entries and DROP support land in the next commit. Verified: bison reports no grammar conflicts; the statements parse and reach ProcessUtility.

Make the foreign catalog and foreign volume DDL commands functional on top of the previously added grammar, parse nodes and system catalogs: * CreateForeignCatalog()/CreateForeignVolume() insert into pg_foreign_catalog/pg_foreign_volume, check server existence and USAGE privilege, record dependencies on the server and owner, and dispatch to segments with preassigned OIDs. * Lookup helpers get_foreign_catalog_oid(), get_foreign_volume_oid() and GetForeignVolumeByName(); both objects are unique on (name, server). * New object infrastructure: OBJECT_FOREIGN_CATALOG/OBJECT_FOREIGN_VOLUME and OCLASS_FOREIGN_CATALOG/OCLASS_FOREIGN_VOLUME with handlers in all exhaustive switches (objectaddress, dependency, aclchk, event trigger, seclabel, dropcmds, alter). Ownership checks go through the generic object_ownercheck() via the new ObjectProperty entries. * Four new syscaches: FOREIGNCATALOGNAME/FOREIGNCATALOGOID and FOREIGNVOLUMENAMESERVER/FOREIGNVOLUMEOID. * DROP CATALOG / DROP VOLUME grammar via drop_type_name, going through the regular RemoveObjects() path with dependency handling, so DROP SERVER ... CASCADE also removes dependent catalogs and volumes. * utility.c dispatch and CREATE/DROP FOREIGN CATALOG|VOLUME command tags.

Make CREATE ICEBERG TABLE functional on top of the existing grammar, parse nodes and pg_lake_table catalog: * laketablecmds.c: CreateLakeTable() inserts the pg_lake_table entry after DefineRelation and records dependencies on the table's foreign catalog and volume; ValidateLakeTableOptions() runs the same resolution on the QD before DefineRelation so validation failures don't surface as QE-annotated errors; RemoveLakeTableEntry() cleans up on drop (hooked into heap_drop_with_catalog). * iceberg_default_catalog / iceberg_default_volume GUCs (synchronized to QEs) provide defaults when CREATE ICEBERG TABLE has no CATALOG or VOLUME clause; their check hooks verify the object exists. * The iceberg table access method is resolved strictly by name (get_table_am_oid) and is expected to be provided by a datalake extension; without it, CREATE ICEBERG TABLE fails up front with a hint. The kernel does not hardcode any extension name or AM OID. * Guard rails: reject the iceberg AM for every creation path other than CreateLakeTableStmt (CREATE TABLE ... USING iceberg, CTAS, matview, default_table_access_method, partition children), reject ALTER TABLE ... SET ACCESS METHOD to or from iceberg, and reject SET DISTRIBUTED BY on lake tables, which must stay DISTRIBUTED RANDOMLY. A relation created with the iceberg AM but without its pg_lake_table metadata would be unusable and undroppable. * utility.c dispatch (transformCreateStmt works on the embedded CreateStmt) and the CREATE LAKE TABLE command tag.

pg_dump cannot reproduce lake tables (their data lives in external object storage managed through the iceberg access method's foreign catalog and volume), so skip them with a warning, matching how other unsupported access methods are handled. The access method is matched by name, not by a hardcoded OID. psql tab completion learns CREATE FOREIGN CATALOG/VOLUME ... SERVER ... OPTIONS, CREATE ICEBERG TABLE, DROP CATALOG/VOLUME with CASCADE/ RESTRICT, and completes foreign catalog and volume names after the CATALOG and VOLUME keywords.

Add a lake_table test to the greenplum_schedule covering the new DDL end to end: CREATE/DROP FOREIGN CATALOG and FOREIGN VOLUME (duplicates, IF NOT EXISTS/IF EXISTS, missing server), segment dispatch, object descriptions, CREATE ICEBERG TABLE with explicit CATALOG/VOLUME clauses and via the iceberg_default_catalog/volume GUCs, the forced random distribution policy, every iceberg-AM misuse guard, ownership checks, and dependency behavior (RESTRICT errors, CASCADE, pg_lake_table cleanup on drop). The iceberg access method is simulated with a heap-backed CREATE ACCESS METHOD, since the kernel resolves it by name and the real AM comes from a datalake extension. Refresh expected output of tests that enumerate system catalogs for the three new ones: misc_sanity (pg_lake_table's toast-less varlena columns), sanity_check (catalog list), and oidjoins (BKI_LOOKUP references, including pg_lake_table.ltforeign_catalog pointing at pg_foreign_catalog).

Address review: new community-authored files should carry the standard Apache-2.0 header instead of the PostgreSQL boilerplate.

Match the community convention (see contrib/pax_storage pax_gbench.cc): the Apache-2.0 license block comes first, followed by the file name and IDENTIFICATION.

Address review: invert the if_not_exists check and error out early so the skip path is no longer nested in an else branch, in both CreateForeignCatalog and CreateForeignVolume.

MisterRaindrop · 2026-07-08T07:01:04Z

Sign in to view

+GetIcebergTableAmOid(bool missing_ok)
+{
+	return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}


Does this AM refer to the minimal Iceberg AM implementation that does not support scans or inserts? Is that correct?

One more question: CREATE ICEBERG TABLE USING
Is this syntax designed to support extensions via other frameworks? From my understanding, there should be no need to specify USING am if we're already using CREATE ICEBERG TABLE, right?

At this stage, I believe the regression testing coverage is sufficient, test stub is enough. Introducing the AM layer at this point would only add unnecessary complexity with no practical benefit, as we still cannot run any real data workloads with it.

yjhjstz · 2026-07-09T03:33:11Z


+	/* If this is a lake table, remove its pg_lake_table entry */
+	if (RelationIsIcebergTable(rel))
+		RemoveLakeTableEntry(relid);


can RemoveLakeTableEntry put into extension via object_access_hook ?

I can migrate this logic to object_access_hook, but the issue is that the AM layer hasn’t been implemented yet on my end, which blocks the migration.
So do I need to implement a default plugin for the AM layer first? Then replace this default AM plugin later once the official datalake plugin is fully rolled out?

Mirror the CREATE side so DROP matches: - DROP CATALOG / DROP VOLUME -> DROP FOREIGN CATALOG / DROP FOREIGN VOLUME (via drop_type_name, same FOREIGN prefix as FOREIGN DATA WRAPPER) - add DROP ICEBERG TABLE [IF EXISTS] as the pair for CREATE ICEBERG TABLE DROP ICEBERG TABLE carries a new DropStmt.isiceberg flag and validates in RangeVarCallbackForDropRelation that the target actually uses the iceberg access method, erroring '"%s" is not an iceberg table' otherwise (mirrors DROP FOREIGN TABLE). Plain DROP TABLE still removes an iceberg table. Adds CMDTAG_DROP_LAKE_TABLE, psql tab completion for the new forms, and refreshes the lake_table regression with new and negative cases. Addresses review feedback on PR apache#1842.

Promote the catalog type from a free-form OPTION to a required TYPE clause backed by a new pg_foreign_catalog.fctype column, per review on PR apache#1842. Every foreign catalog has a type (hive, hdfs, polaris, ...), so it is a property rather than an option; the value is stored verbatim as an open string and validated by the datalake provider, keeping the kernel provider-agnostic. Bumps CATALOG_VERSION_NO for the new column.

MisterRaindrop · 2026-07-10T08:01:25Z

Sign in to view

+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%' ORDER BY 1;
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (path 's3://bucket/prefix');


During design, this path functions as a prefix and is optional for users to specify. If a bucket is available, you may fully specify the path within the Iceberg table configuration. like name is base_path in Discussion: #1683

andr-sokolov · 2026-07-10T07:25:02Z

+	else if (Matches("DROP", "ICEBERG"))
+		COMPLETE_WITH("TABLE");
+	else if (Matches("DROP", "ICEBERG", "TABLE"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables);


Is it possible to write a query to get here iceberg tables only?

Co-authored-by: Andrey Sokolov <sokolov.andrey.yurevich@gmail.com>

MisterRaindrop added 7 commits July 6, 2026 11:31

tuhaihe reviewed Jul 6, 2026

View reviewed changes

Comment thread src/backend/commands/laketablecmds.c

tuhaihe requested review from leborchuk and my-ship-it July 6, 2026 08:42

Use ASF license header for new lake-table source files

5be5f9f

Address review: new community-authored files should carry the standard Apache-2.0 header instead of the PostgreSQL boilerplate.

Vlasdislav reviewed Jul 6, 2026

View reviewed changes

Comment thread contrib/pax_storage/src/test/regress/expected/oidjoins.out

Move ASF license header to top of file header block

272d910

Match the community convention (see contrib/pax_storage pax_gbench.cc): the Apache-2.0 license block comes first, followed by the file name and IDENTIFICATION.

andr-sokolov reviewed Jul 6, 2026

View reviewed changes

Comment thread src/backend/commands/foreigncmds.c Outdated

andr-sokolov reviewed Jul 6, 2026

View reviewed changes

Comment thread src/test/regress/sql/lake_table.sql Outdated

Comment thread src/test/regress/sql/lake_table.sql Outdated

foreigncmds: flatten IF NOT EXISTS duplicate handling

c70fbf9

Address review: invert the if_not_exists check and error out early so the skip path is no longer nested in an else branch, in both CreateForeignCatalog and CreateForeignVolume.

andr-sokolov reviewed Jul 7, 2026

View reviewed changes

Comment thread src/backend/commands/laketablecmds.c

yjhjstz reviewed Jul 9, 2026

View reviewed changes

MisterRaindrop added 2 commits July 10, 2026 10:12

MisterRaindrop requested a review from andr-sokolov July 10, 2026 06:24

andr-sokolov reviewed Jul 10, 2026

View reviewed changes

MisterRaindrop and others added 2 commits July 10, 2026 15:51

Update src/backend/foreign/foreign.c

7e6e7ca

Co-authored-by: Andrey Sokolov <sokolov.andrey.yurevich@gmail.com>

Update src/include/commands/laketablecmds.h

70b451a

Co-authored-by: Andrey Sokolov <sokolov.andrey.yurevich@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding)#1842

Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding)#1842
MisterRaindrop wants to merge 14 commits into
apache:mainfrom
MisterRaindrop:feature/iceberg-ddl-kernel

MisterRaindrop commented Jul 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

MisterRaindrop Jul 8, 2026

Uh oh!

MisterRaindrop Jul 10, 2026

Uh oh!

Uh oh!

yjhjstz Jul 9, 2026

Uh oh!

MisterRaindrop Jul 10, 2026

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

MisterRaindrop Jul 10, 2026

Uh oh!

andr-sokolov Jul 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

MisterRaindrop commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's included (structured for commit-by-commit review)

Design decisions to review

Known limitations / open questions (deliberately out of scope here)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

MisterRaindrop Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

MisterRaindrop Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yjhjstz Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

MisterRaindrop Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

MisterRaindrop Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

andr-sokolov Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MisterRaindrop commented Jul 6, 2026 •

edited

Loading