You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "MisterRaindrop (via GitHub)" <gi...@apache.org> on 2026/07/06 07:52:49 UTC
[PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
MisterRaindrop opened a new pull request, #1842:
URL: https://github.com/apache/cloudberry/pull/1842
## Summary
Kernel-side DDL scaffolding for Iceberg lake tables, per the design proposal in #1683 (this PR is the "core syntax" milestone of the roadmap posted there). It adds the system catalogs, parse nodes, grammar, commands, and client-tool awareness that a datalake provider extension builds on. **No table AM implementation or provider is included** — those are later milestones (agent / FDW phases in the roadmap).
```sql
CREATE FOREIGN CATALOG hive_cat SERVER hive_cat_srv OPTIONS (type 'hive', uri 'thrift://...');
CREATE FOREIGN VOLUME s3_vol SERVER s3_srv OPTIONS (path 's3://bucket/prefix');
CREATE ICEBERG TABLE t (a int, b text) CATALOG hive_cat VOLUME s3_vol OPTIONS (fileformat 'parquet');
DROP CATALOG hive_cat; DROP VOLUME s3_vol;
```
## What's included (structured for commit-by-commit review)
1. **catalog**: `pg_foreign_catalog`, `pg_foreign_volume` (metadata service / storage location handles, hanging off `pg_foreign_server`), `pg_lake_table` (per-relation lake metadata keyed by relid). All three have TOAST tables; name indexes are single-column unique (see "Design decisions").
2. **nodes**: `CreateLakeTableStmt` (embeds `CreateStmt`), `CreateForeignCatalogStmt`, `CreateForeignVolumeStmt` + hand-maintained node-support functions.
3. **parser**: `CREATE ICEBERG TABLE ... CATALOG ... VOLUME ... OPTIONS (...)`, `CREATE FOREIGN CATALOG|VOLUME ... SERVER ...`, `DROP CATALOG|VOLUME` via `drop_type_name`.
4. **commands (foreign catalog/volume)**: create/drop with IF [NOT] EXISTS, dependencies, ownership, object addressing / COMMENT / event-trigger integration, syscaches, QD→QE dispatch with synchronized OIDs.
5. **commands (CREATE ICEBERG TABLE)**: resolves catalog/volume (explicit clause or `iceberg_default_catalog`/`iceberg_default_volume` GUCs), validates the statement uses the `iceberg` AM, creates the relation + TOAST + `pg_lake_table` row + dependencies; guards `ALTER TABLE SET ACCESS METHOD` / `SET DISTRIBUTED BY` both directions; lake tables are forced `DISTRIBUTED RANDOMLY`.
6. **bin**: `pg_dump` skips iceberg tables (by AM name, with a warning — same pattern as PAX); psql tab completion.
7. **tests**: `lake_table` regression in `greenplum_schedule` (DDL lifecycle, duplicate/USING rejection, TOAST wide rows, ownership, dependency/CASCADE, QD/QE dispatch checks) + catalog-expected refresh across `regress`, `singlenode_regress`, and pax copies.
## Design decisions to review
- **Provider decoupling**: the kernel resolves the AM strictly by name (`get_table_am_oid("iceberg")`); no hardcoded AM OID and no extension-name checks. Everything degrades gracefully (with a hint) when no provider is installed.
- **Provider integration contract**: `CreateLakeTable()` runs after `DefineRelation()` and ends with `CommandCounterIncrement()` + `InvokeObjectPostCreateHook(LakeTableRelationId, ...)` — a provider performs remote Iceberg creation from that object-access hook (on QD and QEs), where catalog/volume/options metadata is already visible. `relation_set_new_filelocator` remains local-storage-only.
- **Catalog/volume names are database-global** (single-column unique index), like `pg_foreign_server` — every reference syntax (`DROP CATALOG x`, the `CATALOG x` clause, the GUCs) identifies them by bare name, so the uniqueness scope matches what the syntax can express. (The alternative — per-server names like user mappings — would require server-qualified reference syntax everywhere.)
- `CREATE ICEBERG TABLE` **rejects a `USING` clause naming any other AM**; without it the statement implies `USING iceberg`.
## Known limitations / open questions (deliberately out of scope here)
- **pg_dump / pg_upgrade**: FOREIGN CATALOG/VOLUME objects and pg_lake_table metadata are not dumped yet; iceberg tables are skipped with a warning (PAX precedent). Restore/upgrade support is planned as a follow-up.
- **Permission model**: creating a catalog/volume requires `USAGE` on its server. Referencing one from `CREATE ICEBERG TABLE` currently checks existence only — whether that should require `USAGE` on the underlying server, or dedicated ACLs (`GRANT USAGE ON CATALOG`), is an open question we'd like reviewer input on.
- The GUC defaults are synced to QEs and behave like `default_tablespace` w.r.t. objects dropped after `SET`.
## Testing
- Full CI matrix green on the fork (25 installcheck jobs incl. `ic-good-opt-off/on`, pax, singlenode, isolation2): https://github.com/MisterRaindrop/cloudberry/actions/runs/28768436319
- Validated live on a 3-segment demo cluster with a heap-backed stand-in AM (`CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler`): DDL lifecycle, duplicate-name rejection, USING rejection, wide-row TOAST inserts, QD/QE toast-OID consistency (`gp_dist_random('pg_class')`), drop cleanup, CASCADE.
Discussion: #1683
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3529567064
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fcname, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%' ORDER BY 1;
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%' ORDER BY 1;
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE ICEBERG TABLE lake_test_t0 (a int) CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
+
+-- The default catalog/volume GUCs verify that the object exists
+SET iceberg_default_catalog = 'no_such_catalog'; -- fail
+SET iceberg_default_volume = 'no_such_volume'; -- fail
+
+-- Simulate a datalake provider with a heap-backed iceberg AM
+CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler;
+
+-- CREATE ICEBERG TABLE with explicit catalog and volume
+CREATE ICEBERG TABLE lake_test_t1 (a int, b text) CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fileformat 'parquet');
+SELECT c.relname, lt.lttable_type, lt.ltoptions, fc.fcname, fv.fvname
+ FROM pg_lake_table lt
+ JOIN pg_class c ON c.oid = lt.ltrelid
+ JOIN pg_foreign_catalog fc ON fc.oid = lt.ltforeign_catalog
+ JOIN pg_foreign_volume fv ON fv.oid = lt.ltforeign_volume
+ ORDER BY 1;
+SELECT a.amname FROM pg_am a JOIN pg_class c ON c.relam = a.oid WHERE c.relname = 'lake_test_t1';
+-- Lake tables are always DISTRIBUTED RANDOMLY (policytype 'p', no distkey)
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t1'::regclass;
+
+-- The (heap-backed) table is usable
+INSERT INTO lake_test_t1 VALUES (1, 'x'), (2, 'y');
+SELECT count(*) FROM lake_test_t1;
+
+-- Lake tables get a TOAST table like plain tables, so wide values work
+SELECT reltoastrelid <> 0 AS has_toast FROM pg_class WHERE relname = 'lake_test_t1';
+INSERT INTO lake_test_t1 VALUES (3, repeat('x', 500000));
+SELECT a, length(b) FROM lake_test_t1 WHERE a = 3;
+
+-- Catalog and volume are both required
+CREATE ICEBERG TABLE lake_test_t2 (a int) VOLUME lake_test_vol; -- fail, no catalog
+CREATE ICEBERG TABLE lake_test_t2 (a int) CATALOG lake_test_cat; -- fail, no volume
+
+-- ... unless the GUCs provide defaults
+SET iceberg_default_catalog = 'lake_test_cat';
+SET iceberg_default_volume = 'lake_test_vol';
+CREATE ICEBERG TABLE lake_test_t2 (a int);
+RESET iceberg_default_catalog;
+RESET iceberg_default_volume;
+
+-- A DISTRIBUTED clause is ignored with a warning
+CREATE ICEBERG TABLE lake_test_t3 (a int) CATALOG lake_test_cat VOLUME lake_test_vol DISTRIBUTED BY (a);
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t3'::regclass;
+
+-- CREATE ICEBERG TABLE only accepts the iceberg access method
+CREATE ICEBERG TABLE lake_test_bad0 (a int) CATALOG lake_test_cat VOLUME lake_test_vol USING heap; -- fail
+CREATE ICEBERG TABLE lake_test_t4 (a int) CATALOG lake_test_cat VOLUME lake_test_vol USING iceberg; -- explicit iceberg is fine
+
+-- The iceberg AM is rejected for every path other than CREATE ICEBERG TABLE
+CREATE TABLE lake_test_bad1 (a int) USING iceberg DISTRIBUTED RANDOMLY; -- fail
+CREATE TABLE lake_test_bad2 USING iceberg AS SELECT 1 AS a DISTRIBUTED RANDOMLY; -- fail
+SET default_table_access_method = iceberg;
+CREATE TABLE lake_test_bad3 (a int) DISTRIBUTED RANDOMLY; -- fail
+RESET default_table_access_method;
+CREATE TABLE lake_test_heap (a int) DISTRIBUTED RANDOMLY;
+ALTER TABLE lake_test_heap SET ACCESS METHOD iceberg; -- fail
+ALTER TABLE lake_test_t1 SET ACCESS METHOD heap; -- fail
+ALTER TABLE lake_test_t1 SET DISTRIBUTED BY (a); -- fail
+
+-- Only the owner can drop a catalog or volume
+CREATE ROLE regress_lake_user;
+SET ROLE regress_lake_user;
+DROP CATALOG lake_test_cat; -- fail, not owner
+DROP VOLUME lake_test_vol; -- fail, not owner
+RESET ROLE;
+
+-- Dependencies: the server holds the catalog/volume, which hold the tables
+DROP SERVER lake_test_srv; -- fail, catalog and volume depend on it
+DROP CATALOG lake_test_cat; -- fail, tables depend on it
+
+-- Dropping a lake table removes its pg_lake_table entry
+DROP TABLE lake_test_t1;
+SELECT count(*) FROM pg_lake_table lt JOIN pg_class c ON c.oid = lt.ltrelid
+ WHERE c.relname = 'lake_test_t1';
+
+-- DROP CATALOG ... CASCADE takes the remaining tables with it
+DROP CATALOG lake_test_cat CASCADE;
+SELECT count(*) FROM pg_lake_table lt JOIN pg_class c ON c.oid = lt.ltrelid
+ WHERE c.relname LIKE 'lake\_test%';
+
+-- DROP variants
+DROP CATALOG lake_test_cat; -- fail, already gone
+DROP CATALOG IF EXISTS lake_test_cat; -- skip with notice
+DROP VOLUME lake_test_vol;
+DROP VOLUME lake_test_vol; -- fail, already gone
Review Comment:
When I have created foreign table, I cannot drop it using `drop table`, I should use `DROP FOREIGN TABLE`. But in the case of foreign catalog I should drop it using `drop catalog` without the `foreign` keyword. So I suggest to replace `DROP CATALOG` with `DROP FOREIGN CATALOG` and `DROP VOLUME` with `DROP FOREIGN VOLUME` for unification with existing syntax
```
postgres=# create foreign table ttt(i int, j int) server lake_test_srv;
CREATE FOREIGN TABLE
postgres=# drop table ttt;
ERROR: "ttt" is not a table
HINT: Use DROP FOREIGN TABLE to remove a foreign table.
postgres=# DROP FOREIGN CATALOG lake_test_cat;
ERROR: syntax error at or near "CATALOG"
LINE 1: DROP FOREIGN CATALOG lake_test_cat;
^
postgres=#
```
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
What about making `type` as a required parameter to make more strict checks? For example
```
CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' URI 'thrift://localhost:9083';
```
When I pass options I can pass nonexistent type and invalid option name, and the query is executed without errors
```
postgres=# CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'nonexistent_type', uriii 'thrift://localhost:9083');
CREATE FOREIGN CATALOG
postgres=# table pg_foreign_catalog;
oid | fcname | fcowner | fcserver | fcoptions
-------+---------------+---------+----------+-------------------------------------------------------
17022 | lake_test_cat | 10 | 17019 | {type=nonexistent_type,uriii=thrift://localhost:9083}
(1 row)
postgres=#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3527475759
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,469 @@
+/*-------------------------------------------------------------------------
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
Review Comment:
For the new files created by the Cloudberry community, we can use the standard ASF license header:
https://github.com/apache/cloudberry/blob/a4ece882be877e23ae3e1ea3981b39e43788d40e/contrib/pax_storage/src/cpp/pax_gbench.cc#L1-L26
If this PR includes the third-party files, feel free to let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3541968248
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
I don't recommend this approach, as it would result in too many dedicated catalog creation syntaxes. We may also support JDBC and other user-defined catalog types down the line. Creating a separate syntax for each type would clearly be redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "leborchuk (via GitHub)" <gi...@apache.org>.
leborchuk commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3639386498
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,158 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
Review Comment:
> Building a Postgres Data Warehouse using Iceberg.pdf
Yes, looks good. But how we setup catalog and lakehouse format? There should exist some definition ... It's good for pre-configured database instances, but your own database needs a more advanced syntax.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3663349339
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,158 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
Review Comment:
@MisterRaindrop If possible, let's not change the kernel to reduce conflicts when updating to new versions of PostgreSQL. I think it's better to use an extension.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3585478935
##########
src/include/tcop/cmdtaglist.h:
##########
@@ -176,10 +179,13 @@ PG_CMDTAG(CMDTAG_DROP_DOMAIN, "DROP DOMAIN", true, false, false)
PG_CMDTAG(CMDTAG_DROP_DYNAMIC_TABLE, "DROP DYNAMIC TABLE", true, false, false)
PG_CMDTAG(CMDTAG_DROP_EVENT_TRIGGER, "DROP EVENT TRIGGER", false, false, false)
PG_CMDTAG(CMDTAG_DROP_EXTENSION, "DROP EXTENSION", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_FOREIGN_CATALOG, "DROP FOREIGN CATALOG", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FOREIGN_DATA_WRAPPER, "DROP FOREIGN DATA WRAPPER", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FOREIGN_TABLE, "DROP FOREIGN TABLE", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_FOREIGN_VOLUME, "DROP FOREIGN VOLUME", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FUNCTION, "DROP FUNCTION", true, false, false)
PG_CMDTAG(CMDTAG_DROP_INDEX, "DROP INDEX", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_LAKE_TABLE, "DROP LAKE TABLE", true, false, false)
Review Comment:
When I run a `CREATE ICEBERG TABLE` query, I get output `CREATE LAKE TABLE`. The same replacement `ICEBERG` with `LAKE` happens in the case of `DROP ICEBERG TABLE`.
```
postgres=# CREATE ICEBERG TABLE lake_test_t1 (a int, b text) CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fileformat 'parquet');
CREATE LAKE TABLE
postgres=# DROP ICEBERG TABLE IF EXISTS lake_test_t1;
DROP LAKE TABLE
postgres=#
```
Usually the output is the beginning of CREATE or DROP query. May be it means that we should replace `CREATE ICEBERG TABLE` with `CREATE LAKE TABLE ... TYPE 'ICEBERG'` or `CREATE LAKE TABLE ... USING ICEBERG`? We have the laketablecmds.c and laketablecmds.h files, the `T_CreateLakeTableStmt` node with the `table_type` field and the `pg_lake_table` catalog table with `lttable_type` column. I think it will be cool if the syntax will correspond to data structures and names in the source code.
##########
doc/src/sgml/ref/create_iceberg_table.sgml:
##########
@@ -0,0 +1,193 @@
+<!--
+doc/src/sgml/ref/create_iceberg_table.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="sql-createicebergtable">
+ <indexterm zone="sql-createicebergtable">
+ <primary>CREATE ICEBERG TABLE</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>CREATE ICEBERG TABLE</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>CREATE ICEBERG TABLE</refname>
+ <refpurpose>define a new Iceberg table</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+CREATE ICEBERG TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [
+ <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [, ... ]
+ ] )
+ [ CATALOG <replaceable class="parameter">catalog_name</replaceable> ]
+ [ VOLUME <replaceable class="parameter">volume_name</replaceable> ]
+ [ OPTIONS ( <replaceable class="parameter">option</replaceable> '<replaceable class="parameter">value</replaceable>' [, ...] ) ]
+ [ DISTRIBUTED BY ( <replaceable class="parameter">column</replaceable> [, ... ] ) | DISTRIBUTED RANDOMLY | DISTRIBUTED REPLICATED ]
Review Comment:
It is written in the test comment: "Lake tables are always DISTRIBUTED RANDOMLY". Can we remove this line from syntax? This line seems meaningless.
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,155 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%' ORDER BY 1;
Review Comment:
Why do we need `ORDER BY 1` here and below? Tuples order doesn't matter for regression tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3628201792
##########
src/backend/parser/gram.y:
##########
@@ -9093,6 +9098,185 @@ CreateDirectoryTableStmt:
}
;
+/*****************************************************************************
+ *
+ * QUERY:
+ * CREATE FOREIGN CATALOG name SERVER server_name OPTIONS (...)
+ *
+ *****************************************************************************/
+
+CreateForeignCatalogStmt:
+ CREATE FOREIGN CATALOG_P name SERVER name TYPE_P Sconst create_generic_options
+ {
+ CreateForeignCatalogStmt *n = makeNode(CreateForeignCatalogStmt);
+ n->catalogname = $4;
+ n->servername = $6;
+ n->catalogtype = $8;
+ n->options = $9;
+ n->if_not_exists = false;
+ $$ = (Node *) n;
+ }
+ | CREATE FOREIGN CATALOG_P IF_P NOT EXISTS name SERVER name TYPE_P Sconst create_generic_options
+ {
+ CreateForeignCatalogStmt *n = makeNode(CreateForeignCatalogStmt);
+ n->catalogname = $7;
+ n->servername = $9;
+ n->catalogtype = $11;
+ n->options = $12;
+ n->if_not_exists = true;
+ $$ = (Node *) n;
+ }
+ ;
+
+/*****************************************************************************
+ *
+ * QUERY:
+ * CREATE FOREIGN VOLUME name SERVER server_name OPTIONS (...)
+ *
+ *****************************************************************************/
+
+CreateForeignVolumeStmt:
+ CREATE FOREIGN VOLUME name SERVER name create_generic_options
+ {
+ CreateForeignVolumeStmt *n = makeNode(CreateForeignVolumeStmt);
+ n->volumename = $4;
+ n->servername = $6;
+ n->options = $7;
+ n->if_not_exists = false;
+ $$ = (Node *) n;
+ }
+ | CREATE FOREIGN VOLUME IF_P NOT EXISTS name SERVER name create_generic_options
+ {
+ CreateForeignVolumeStmt *n = makeNode(CreateForeignVolumeStmt);
+ n->volumename = $7;
+ n->servername = $9;
+ n->options = $10;
+ n->if_not_exists = true;
+ $$ = (Node *) n;
+ }
+ ;
+
+OptForeignCatalog:
+ CATALOG_P name { $$ = $2; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+OptForeignVolume:
+ VOLUME name { $$ = $2; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+/*****************************************************************************
+ *
+ * QUERY:
+ * CREATE LAKE TABLE relname (columns) USING format
+ * [CATALOG cat] [VOLUME vol] OPTIONS (...)
+ *
+ * A lake table stores its data on external object storage; fragments are
+ * not hash-distributed across segments, so the distribution policy is
+ * forced to RANDOM to keep UPDATE/DELETE correct. The USING clause names
+ * the table format (currently only ICEBERG); the format also determines the
+ * table access method the relation is created with.
+ *
+ *****************************************************************************/
+
+CreateLakeTableStmt:
+ CREATE LAKE TABLE qualified_name '(' OptTableElementList ')'
+ USING name
+ OptForeignCatalog OptForeignVolume create_generic_options
+ OptDistributedBy
Review Comment:
I suggest to remove this line because DISTRIBUTED clause is not supported for lake tables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3661164809
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,158 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
Review Comment:
if we use `CREATE TABLE ... USING iceberg WITH (catalog 'c', volume 'v')`, we need not change kernel code, iceberg will be totally an extension .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3541885910
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
Yes, it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3541910054
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
What about type name without quotes around?
```
CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE builtin;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3636020805
##########
src/backend/commands/tablecmds.c:
##########
@@ -17059,13 +17110,32 @@ static void
ATPrepSetAccessMethod(AlteredTableInfo *tab, Relation rel, const char *amname)
{
Oid amoid;
+ Oid iceberg_amoid;
/* Check that the table access method exists */
amoid = get_table_am_oid(amname, false);
if (rel->rd_rel->relam == amoid)
return;
+ /*
+ * The iceberg AM relies on catalog entries that only the CREATE/DROP
+ * LAKE TABLE paths manage, so a table cannot be converted to or from
+ * it with SET ACCESS METHOD.
+ */
+ iceberg_amoid = GetIcebergTableAmOid(true);
+ if (OidIsValid(iceberg_amoid) && amoid == iceberg_amoid)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot change access method of table \"%s\" to \"%s\"",
+ RelationGetRelationName(rel), ICEBERG_TABLE_AM_NAME),
+ errhint("Use CREATE LAKE TABLE ... USING ICEBERG to create a lake table.")));
+ if (OidIsValid(iceberg_amoid) && rel->rd_rel->relam == iceberg_amoid)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot change access method of lake table \"%s\"",
+ RelationGetRelationName(rel))));
Review Comment:
Let's check `OidIsValid(iceberg_amoid)` one time.
ICEBERG_TABLE_AM_NAME is a constant, but not variable, we can concatenate it at compile time
```suggestion
if (OidIsValid(iceberg_amoid) {
if (amoid == iceberg_amoid)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("cannot change access method of table \"%s\" to \"" ICEBERG_TABLE_AM_NAME "\"",
RelationGetRelationName(rel)),
errhint("Use CREATE LAKE TABLE ... USING ICEBERG to create a lake table.")));
if (rel->rd_rel->relam == iceberg_amoid)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("cannot change access method of lake table \"%s\"",
RelationGetRelationName(rel))));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3556149077
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,483 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
Review Comment:
At this stage, I believe the regression testing coverage is sufficient, test stub is enough. Introducing the AM layer at this point would only add unnecessary complexity with no practical benefit, as we still cannot run any real data workloads with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3535094250
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
I fully agree. However, I haven’t implemented validation at this stage because the syntax is still incomplete and subject to frequent changes. Given this, would you recommend we add validation now, or hold off until the entire workflow is fully functional before enforcing validation rules?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#issuecomment-5165043390
@andr-sokolov @yjhjstz
I open this draft PR about lake table ext (create table xxx using iceberg xxx), please suggest and review it.
https://github.com/apache/cloudberry/pull/1881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3901251321
##########
doc/src/sgml/ref/create_iceberg_table.sgml:
##########
@@ -0,0 +1,193 @@
+<!--
+doc/src/sgml/ref/create_iceberg_table.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="sql-createicebergtable">
+ <indexterm zone="sql-createicebergtable">
+ <primary>CREATE ICEBERG TABLE</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>CREATE ICEBERG TABLE</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>CREATE ICEBERG TABLE</refname>
+ <refpurpose>define a new Iceberg table</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+CREATE ICEBERG TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [
+ <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [, ... ]
+ ] )
+ [ CATALOG <replaceable class="parameter">catalog_name</replaceable> ]
+ [ VOLUME <replaceable class="parameter">volume_name</replaceable> ]
+ [ OPTIONS ( <replaceable class="parameter">option</replaceable> '<replaceable class="parameter">value</replaceable>' [, ...] ) ]
+ [ DISTRIBUTED BY ( <replaceable class="parameter">column</replaceable> [, ... ] ) | DISTRIBUTED RANDOMLY | DISTRIBUTED REPLICATED ]
Review Comment:
You were right, and the point outlived the design it was made against.
That file is gone — the kernel DDL it documented was reverted when this PR became an extension, so there is no `CREATE ICEBERG TABLE` grammar and no `DISTRIBUTED` clause in the syntax to remove. `doc/` no longer appears in this PR's diff at all.
The underlying judgement was adopted rather than dropped. A lake table's rows live outside PostgreSQL, so no local key can describe where they are; the module injects `DISTRIBUTED RANDOMLY` itself and does not ask the user for it. `DISTRIBUTED BY` and `DISTRIBUTED REPLICATED` are both refused.
There is one deliberate exception, and it is the only reason any clause is accepted: `pg_dump` writes `DISTRIBUTED RANDOMLY` into the `CREATE TABLE` it emits, so refusing every clause would mean refusing to restore a dump this module produced. An explicit clause is therefore accepted only when it asks for exactly what would have been injected anyway (`pg_iceberg_extensible.c`, in `CREATE TABLE` handling).
Resolving this one, since the file it points at no longer exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3575931280
##########
src/backend/catalog/heap.c:
##########
@@ -2322,6 +2323,10 @@ heap_drop_with_catalog(Oid relid)
*/
CheckTableForSerializableConflictIn(rel);
+ /* If this is a lake table, remove its pg_lake_table entry */
+ if (RelationIsIcebergTable(rel))
+ RemoveLakeTableEntry(relid);
Review Comment:
I’m still unsure how best to provide this. Could you share some existing examples for reference?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3642067994
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,158 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
Review Comment:
Em, currently how about change `CREATE LAKE TABLE lake_test_t0 ` -> `CREATE TABLE lake_test_t0`, no need `LAKE` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3642062606
##########
src/backend/catalog/heap.c:
##########
@@ -2322,6 +2323,10 @@ heap_drop_with_catalog(Oid relid)
*/
CheckTableForSerializableConflictIn(rel);
+ /* If this is a lake table, remove its pg_lake_table entry */
+ if (RelationIsIcebergTable(rel))
+ RemoveLakeTableEntry(relid);
Review Comment:
just add a fixme .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3662263724
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,158 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
Review Comment:
@yjhjstz Agreed — this looks technically feasible, so I sketched what the fully-extension shape could look like.
Since Cloudberry's table AM API already supports per-AM reloptions validation (`TableAmRoutine->amoptions`, unlike vanilla PG16), `WITH (catalog ..., volume ...)` can be validated by the iceberg AM itself with no kernel changes. The catalog/volume objects could reuse the existing FDW infrastructure (`SERVER` + `USER MAPPING`), which gives ACLs, pg_dump support, QD/QE dispatch and dependency tracking for free:
```sql
CREATE EXTENSION pg_iceberg; -- provides the table AM, two FDWs, and hooks
CREATE SERVER hive_cat FOREIGN DATA WRAPPER iceberg_catalog
OPTIONS (type 'hive', uri 'thrift://metastore:9083');
CREATE USER MAPPING FOR alice SERVER hive_cat
OPTIONS (user '...', password '...');
CREATE SERVER s3_vol FOREIGN DATA WRAPPER iceberg_volume
OPTIONS (base_path 's3://bucket/prefix');
CREATE TABLE t (a int, b text) USING iceberg
WITH (catalog 'hive_cat', volume 's3_vol');
DROP TABLE t; -- plain DROP TABLE applies
```
A few things the extension would need to own, instead of kernel guards:
- registering table metadata and `pg_depend` entries (table → server) via `object_access_hook`, so `DROP SERVER` stays protected and CASCADE works;
- intercepting paths that don't make sense for iceberg (CTAS, partition children, `ALTER TABLE ... SET ACCESS METHOD`) via `ProcessUtility_hook`, and forcing random distribution;
- the library would need to be in `shared_preload_libraries` so the hooks are always installed.
This is essentially the pg_lake architecture, and `USER MAPPING` would also give a cleaner credentials story than embedding keys in options. If we go this way, the kernel scaffolding in this PR (the grammar, the three catalogs, the guards) would no longer be needed — the work would move into the datalake extension instead. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3528533057
##########
src/backend/commands/foreigncmds.c:
##########
@@ -1026,6 +1028,266 @@ CreateForeignServer(CreateForeignServerStmt *stmt)
}
+/*
+ * Create a foreign catalog
+ */
+ObjectAddress
+CreateForeignCatalog(CreateForeignCatalogStmt *stmt)
+{
+ Relation rel;
+ Datum catalogoptions;
+ Datum values[Natts_pg_foreign_catalog];
+ bool nulls[Natts_pg_foreign_catalog];
+ HeapTuple tuple;
+ Oid catalogId;
+ Oid ownerId;
+ AclResult aclresult;
+ ObjectAddress myself;
+ ObjectAddress referenced;
+ ForeignServer *server;
+
+ rel = table_open(ForeignCatalogRelationId, RowExclusiveLock);
+
+ /* For now the owner cannot be specified on create. Use effective user ID. */
+ ownerId = GetUserId();
+
+ /*
+ * Check that there is no other foreign catalog by this name. Catalog
+ * names are global (like server names): every reference syntax (DROP
+ * CATALOG, the CATALOG clause of CREATE ICEBERG TABLE, GUCs) identifies
+ * a catalog by bare name, so the name alone must be unique. If there is
+ * one, do nothing if IF NOT EXISTS was specified.
+ */
+ catalogId = get_foreign_catalog_oid(stmt->catalogname, true);
+ if (OidIsValid(catalogId))
+ {
+ if (stmt->if_not_exists)
+ {
+ /*
+ * If we are in an extension script, insist that the pre-existing
+ * object be a member of the extension, to avoid security risks.
+ */
+ ObjectAddressSet(myself, ForeignCatalogRelationId, catalogId);
+ checkMembershipInCurrentExtension(&myself);
+
+ /* OK to skip */
+ ereport(NOTICE,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("foreign catalog \"%s\" already exists, skipping",
+ stmt->catalogname)));
+ table_close(rel, RowExclusiveLock);
+ return InvalidObjectAddress;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("foreign catalog \"%s\" already exists",
+ stmt->catalogname)));
Review Comment:
I suggest to invert the condition to reduce indentation and get rid of `else`. Here and in `CreateForeignVolume` too.
```suggestion
if (!stmt->if_not_exists)
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("foreign catalog \"%s\" already exists",
stmt->catalogname)));
/*
* If we are in an extension script, insist that the pre-existing
* object be a member of the extension, to avoid security risks.
*/
ObjectAddressSet(myself, ForeignCatalogRelationId, catalogId);
checkMembershipInCurrentExtension(&myself);
/* OK to skip */
ereport(NOTICE,
(errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("foreign catalog \"%s\" already exists, skipping",
stmt->catalogname)));
table_close(rel, RowExclusiveLock);
return InvalidObjectAddress;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3900092804
##########
.github/workflows/build-cloudberry.yml:
##########
@@ -326,6 +326,10 @@ jobs:
"contrib/pg_buffercache:installcheck",
"contrib/sslinfo:installcheck"]
},
+ {"test":"ic-datalake-fdw",
+ "make_configs":["contrib/datalake_fdw:installcheck"],
+ "shared_preload_libraries":"datalake_fdw"
Review Comment:
Done in a58aec79f14 — datalake_fdw is now built and tested the way anyone outside the project would consume it, rather than compiled into the server we ship.
- `configure-cloudberry.sh` no longer passes `--enable-datalake-fdw`, so the RPM every test job installs is unchanged by this module. `configure-cloudberry.sh` has therefore dropped out of this PR's diff entirely.
- The test job gained a step that, for a matrix entry naming a `pgxs_extension`, builds and installs it with PGXS against the server that was just installed. It has to run before the demo cluster is created: a module named in `shared_preload_libraries` must exist by the time the cluster starts.
- `--enable-datalake-fdw` itself stays, for people building from source who do want it in-tree.
I validated it on a real run before bringing it here (fork, `workflow_dispatch`, `ic-datalake-fdw` only):
- the build job's log shows `checking whether to build with datalake_fdw support ... no`, and zero compilation of the module — the RPM genuinely does not contain it;
- the new step compiled and installed it in 2.5s (`datalake_fdw.so`, `.control`, `--1.0.sql`);
- the cluster started with it preloaded and the regression passed 3/3.
One thing to weigh, since it is a real cost rather than a detail: **the in-tree path now has no CI coverage at all.** The `contrib/Makefile` conditional and the configure option feeding it are no longer built by anything automated — building from source by hand is the only thing that exercises them. That was previously verified precisely because `ic-datalake-fdw` was green on the RPM build.
@tuhaihe you had said the previous shape looked good and matched PAX, which it did — every optional module here (`pax`, `gpcloud`, `orafce`, `mapreduce`, `ic-proxy`) is `no` by default in `configure.ac` and switched on in `configure-cloudberry.sh`. This change deliberately steps away from that pattern in favour of exercising the extension path instead. Happy to go back to the PAX shape if you'd rather keep the in-tree wiring covered — the previous commit is still in this branch's history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#issuecomment-5368194748
> What about moving the extension to gpcontrib to minimize conflicts with PostgreSQL?
I think it's OK to place it in the `contrib` folder, just like `pax` which was created by the Cloudberry community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3527480837
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,469 @@
+/*-------------------------------------------------------------------------
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
Review Comment:
Ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3536652621
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,483 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
Review Comment:
In the b4704be776f814b46a5a59eb543df80fad32a864 commit description I read " The iceberg table access method is resolved strictly by name (get_table_am_oid) and is expected to be provided by a datalake extension; without it, CREATE ICEBERG TABLE fails up front with a hint. The kernel does not hardcode any extension name or AM OID."
In my opinion, it is strange to have queries (like `CREATE ICEBERG TABLE`) in the core which are useless without extensions. What about default iceberg AM which can be replaced (using `USING`) with AM from some extension? Without default AM we cannot test the code properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3541968248
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
I don't recommend this approach, as it would result in too many dedicated catalog creation syntaxes. We may also support JDBC catalog and other user-defined catalog types down the line. Creating a separate syntax for each type would clearly be redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3542291425
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,483 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
+
+/*
+ * RelationIsIcebergTable
+ *
+ * True iff the relation uses the iceberg table access method. Resolved by
+ * access method name so the kernel does not depend on any particular
+ * extension's OID assignments.
+ */
+bool
+RelationIsIcebergTable(Relation rel)
Review Comment:
If we can add AM using an extension, then we can have more than one such extensions. User may want to install several extensions. I'm not sure if several iceberg extensions installation is really useful. But your idea about possible support of other formats (`CREATE LAKEHOUSE xxx USING xxx`) seems to be very interesting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3900277374
##########
.github/workflows/build-cloudberry.yml:
##########
@@ -326,6 +326,10 @@ jobs:
"contrib/pg_buffercache:installcheck",
"contrib/sslinfo:installcheck"]
},
+ {"test":"ic-datalake-fdw",
+ "make_configs":["contrib/datalake_fdw:installcheck"],
+ "shared_preload_libraries":"datalake_fdw"
Review Comment:
Makes sense. Since the extension is still in the skeleton stage, proceeding with the separated CI / PGXS extension path works well for now. Once the implementation matures, we can revisit and switch it back to the `configure.ac` option if needed to align with other modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3840218872
##########
contrib/datalake_fdw/src/iceberg_volume_fdw/iceberg_volume_fdw.c:
##########
@@ -0,0 +1,157 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * iceberg_volume_fdw.c
+ * Option validator for Iceberg volume foreign servers.
+ *
+ * IDENTIFICATION
+ * contrib/datalake_fdw/src/iceberg_volume_fdw/iceberg_volume_fdw.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/reloptions.h"
+#include "am_iceberg/pg_iceberg_options.h"
+#include "catalog/pg_foreign_data_wrapper.h"
+#include "catalog/pg_foreign_server.h"
+#include "catalog/pg_user_mapping.h"
+#include "commands/defrem.h"
+#include "common/dl_option_util.h"
+#include "fmgr.h"
+#include "iceberg_volume_fdw/iceberg_volume_option.h"
+
+PG_FUNCTION_INFO_V1(iceberg_volume_fdw_validator);
+
+static bool is_volume_server_option(const char *name);
+static bool is_volume_user_mapping_option(const char *name);
+
+static bool
+is_volume_server_option(const char *name)
+{
+ return strcmp(name, DATALAKE_ICEBERG_VOLUME_BASE_PATH) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_ENDPOINT) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_REGION) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_PATH_STYLE_ACCESS) == 0;
+}
+
+/*
+ * Every credential here is optional, so that ambient storage credentials -- an
+ * instance profile, a ticket cache -- remain a valid deployment choice.
+ */
+static bool
+is_volume_user_mapping_option(const char *name)
+{
+ return strcmp(name, DATALAKE_ICEBERG_VOLUME_USERNAME) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_AWS_ACCESS_KEY_ID) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_AWS_SECRET_ACCESS_KEY) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_AWS_SESSION_TOKEN) == 0;
+}
+
+Datum
+iceberg_volume_fdw_validator(PG_FUNCTION_ARGS)
+{
+ List *options = untransformRelOptions(PG_GETARG_DATUM(0));
+ Oid catalog = PG_GETARG_OID(1);
+ ListCell *lc;
+ IcebergVolumeServerOptions server_options;
+ IcebergForeignVolumeOptions volume_options;
Review Comment:
Applied in 2adeb73da50, and your reasoning is the better part of it. The declarations sit about sixty lines above where the memsets were, with the option-name validation loop and two early returns in between, so anything added to that stretch that read the structs would have read them uninitialized. Zeroing at the declaration removes that possibility.
Being precise about what it buys: **no existing test can tell the two forms apart.** Both parse functions assign every field they own — `get_string_option()` returns NULL when an option is absent, and `get_bool_option_ex()` writes both the value and its "was set" flag — so the memsets were belt-and-braces rather than load bearing, and nothing between the declarations and the parse calls touched the structs. So this is future safety, not a fix, and I would rather say that than imply it closed a live bug.
`= {0}` is also the form the rest of the tree uses (over a hundred occurrences in `src/backend/commands` and `src/backend/catalog` alone), so it improves consistency too.
The module has one other `memset`, in `pg_iceberg_options.c`, which clears an out parameter reached through a pointer — there is no declaration there to attach an initializer to, so I left it alone rather than have it look overlooked.
Verified: zero compiler warnings (the module builds with `-Werror=uninitialized` and without `-Wextra`, so the initializer neither trips nor silences anything) and `installcheck` 3/3 on a three-segment cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3862737789
##########
.github/workflows/build-cloudberry.yml:
##########
@@ -326,6 +326,10 @@ jobs:
"contrib/pg_buffercache:installcheck",
"contrib/sslinfo:installcheck"]
},
+ {"test":"ic-datalake-fdw",
+ "make_configs":["contrib/datalake_fdw:installcheck"],
+ "shared_preload_libraries":"datalake_fdw"
Review Comment:
test datalake_fdw as a true extension on a separated CI ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3556789791
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fcname, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%' ORDER BY 1;
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%' ORDER BY 1;
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE ICEBERG TABLE lake_test_t0 (a int) CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
+
+-- The default catalog/volume GUCs verify that the object exists
+SET iceberg_default_catalog = 'no_such_catalog'; -- fail
+SET iceberg_default_volume = 'no_such_volume'; -- fail
+
+-- Simulate a datalake provider with a heap-backed iceberg AM
+CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler;
+
+-- CREATE ICEBERG TABLE with explicit catalog and volume
+CREATE ICEBERG TABLE lake_test_t1 (a int, b text) CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fileformat 'parquet');
+SELECT c.relname, lt.lttable_type, lt.ltoptions, fc.fcname, fv.fvname
+ FROM pg_lake_table lt
+ JOIN pg_class c ON c.oid = lt.ltrelid
+ JOIN pg_foreign_catalog fc ON fc.oid = lt.ltforeign_catalog
+ JOIN pg_foreign_volume fv ON fv.oid = lt.ltforeign_volume
+ ORDER BY 1;
+SELECT a.amname FROM pg_am a JOIN pg_class c ON c.relam = a.oid WHERE c.relname = 'lake_test_t1';
+-- Lake tables are always DISTRIBUTED RANDOMLY (policytype 'p', no distkey)
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t1'::regclass;
+
+-- The (heap-backed) table is usable
+INSERT INTO lake_test_t1 VALUES (1, 'x'), (2, 'y');
+SELECT count(*) FROM lake_test_t1;
+
+-- Lake tables get a TOAST table like plain tables, so wide values work
+SELECT reltoastrelid <> 0 AS has_toast FROM pg_class WHERE relname = 'lake_test_t1';
+INSERT INTO lake_test_t1 VALUES (3, repeat('x', 500000));
+SELECT a, length(b) FROM lake_test_t1 WHERE a = 3;
+
+-- Catalog and volume are both required
+CREATE ICEBERG TABLE lake_test_t2 (a int) VOLUME lake_test_vol; -- fail, no catalog
+CREATE ICEBERG TABLE lake_test_t2 (a int) CATALOG lake_test_cat; -- fail, no volume
+
+-- ... unless the GUCs provide defaults
+SET iceberg_default_catalog = 'lake_test_cat';
+SET iceberg_default_volume = 'lake_test_vol';
+CREATE ICEBERG TABLE lake_test_t2 (a int);
+RESET iceberg_default_catalog;
+RESET iceberg_default_volume;
+
+-- A DISTRIBUTED clause is ignored with a warning
+CREATE ICEBERG TABLE lake_test_t3 (a int) CATALOG lake_test_cat VOLUME lake_test_vol DISTRIBUTED BY (a);
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t3'::regclass;
+
+-- CREATE ICEBERG TABLE only accepts the iceberg access method
+CREATE ICEBERG TABLE lake_test_bad0 (a int) CATALOG lake_test_cat VOLUME lake_test_vol USING heap; -- fail
+CREATE ICEBERG TABLE lake_test_t4 (a int) CATALOG lake_test_cat VOLUME lake_test_vol USING iceberg; -- explicit iceberg is fine
+
+-- The iceberg AM is rejected for every path other than CREATE ICEBERG TABLE
+CREATE TABLE lake_test_bad1 (a int) USING iceberg DISTRIBUTED RANDOMLY; -- fail
+CREATE TABLE lake_test_bad2 USING iceberg AS SELECT 1 AS a DISTRIBUTED RANDOMLY; -- fail
+SET default_table_access_method = iceberg;
+CREATE TABLE lake_test_bad3 (a int) DISTRIBUTED RANDOMLY; -- fail
+RESET default_table_access_method;
+CREATE TABLE lake_test_heap (a int) DISTRIBUTED RANDOMLY;
+ALTER TABLE lake_test_heap SET ACCESS METHOD iceberg; -- fail
+ALTER TABLE lake_test_t1 SET ACCESS METHOD heap; -- fail
+ALTER TABLE lake_test_t1 SET DISTRIBUTED BY (a); -- fail
+
+-- Only the owner can drop a catalog or volume
+CREATE ROLE regress_lake_user;
+SET ROLE regress_lake_user;
+DROP CATALOG lake_test_cat; -- fail, not owner
+DROP VOLUME lake_test_vol; -- fail, not owner
+RESET ROLE;
+
+-- Dependencies: the server holds the catalog/volume, which hold the tables
+DROP SERVER lake_test_srv; -- fail, catalog and volume depend on it
+DROP CATALOG lake_test_cat; -- fail, tables depend on it
+
+-- Dropping a lake table removes its pg_lake_table entry
+DROP TABLE lake_test_t1;
+SELECT count(*) FROM pg_lake_table lt JOIN pg_class c ON c.oid = lt.ltrelid
+ WHERE c.relname = 'lake_test_t1';
+
+-- DROP CATALOG ... CASCADE takes the remaining tables with it
+DROP CATALOG lake_test_cat CASCADE;
+SELECT count(*) FROM pg_lake_table lt JOIN pg_class c ON c.oid = lt.ltrelid
+ WHERE c.relname LIKE 'lake\_test%';
+
+-- DROP variants
+DROP CATALOG lake_test_cat; -- fail, already gone
+DROP CATALOG IF EXISTS lake_test_cat; -- skip with notice
+DROP VOLUME lake_test_vol;
+DROP VOLUME lake_test_vol; -- fail, already gone
Review Comment:
Create and delete now share the same syntax.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#issuecomment-5023511438
@andr-sokolov It is ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#issuecomment-5274958026
> > @andr-sokolov @yjhjstz I open this draft PR about lake table ext (create table xxx using iceberg xxx), please give some suggest and review it. #1881
>
> Hi, is there any way to introduce your new commits to this place, where we have a context for review? FYI. @MisterRaindrop
I don’t recommend moving it here, as this would entail extensive changes. I suggest submitting these adjustments via a new PR for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3628301540
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,448 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
+
+/*
+ * RelationIsIcebergTable
+ *
+ * True iff the relation uses the iceberg table access method. Resolved by
+ * access method name so the kernel does not depend on any particular
+ * extension's OID assignments.
+ */
+bool
+RelationIsIcebergTable(Relation rel)
+{
+ Oid iceberg_amoid;
+
+ if (!OidIsValid(rel->rd_rel->relam))
+ return false;
+
+ iceberg_amoid = GetIcebergTableAmOid(true);
+
+ return OidIsValid(iceberg_amoid) && rel->rd_rel->relam == iceberg_amoid;
+}
+
+/*
+ * Validate table type
+ */
+static void
+validate_table_type(const char *table_type)
+{
+ if (!table_type)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("lake table format cannot be NULL")));
+
+ if (strcmp(table_type, "ICEBERG") != 0)
Review Comment:
What about using ICEBERG_TABLE_AM_NAME here?
```suggestion
if (strcmp(table_type, ICEBERG_TABLE_AM_NAME) != 0)
```
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,448 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
+
+/*
+ * RelationIsIcebergTable
+ *
+ * True iff the relation uses the iceberg table access method. Resolved by
+ * access method name so the kernel does not depend on any particular
+ * extension's OID assignments.
+ */
+bool
+RelationIsIcebergTable(Relation rel)
+{
+ Oid iceberg_amoid;
+
+ if (!OidIsValid(rel->rd_rel->relam))
+ return false;
+
+ iceberg_amoid = GetIcebergTableAmOid(true);
+
+ return OidIsValid(iceberg_amoid) && rel->rd_rel->relam == iceberg_amoid;
+}
+
+/*
+ * Validate table type
+ */
+static void
+validate_table_type(const char *table_type)
+{
+ if (!table_type)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("lake table format cannot be NULL")));
+
+ if (strcmp(table_type, "ICEBERG") != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("unsupported lake table format \"%s\"", table_type),
+ errhint("The only supported format is ICEBERG (USING ICEBERG).")));
+}
+
+/*
+ * Validate foreign catalog exists
+ */
+static Oid
+validate_foreign_catalog(const char *catalog_name)
+{
+ if (!catalog_name || catalog_name[0] == '\0')
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("no foreign catalog specified"),
+ errhint("Specify CATALOG in CREATE LAKE TABLE or set iceberg_default_catalog.")));
+
+ return get_foreign_catalog_oid(catalog_name, false);
+}
+
+/*
+ * Validate foreign volume exists
+ */
+static Oid
+validate_foreign_volume(const char *volume_name)
+{
+ if (!volume_name || volume_name[0] == '\0')
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("no foreign volume specified"),
+ errhint("Specify VOLUME in CREATE LAKE TABLE or set iceberg_default_volume.")));
+
+ return get_foreign_volume_oid(volume_name, false);
+}
+
+/*
+ * ResolveLakeTableOptions
+ *
+ * Resolve and validate the table type, catalog and volume of a
+ * CreateLakeTableStmt, returning the catalog/volume OIDs.
+ *
+ * Also exposed (via ValidateLakeTableStmt) so ProcessUtilitySlow can run
+ * the validation on the QD before DefineRelation: DefineRelation dispatches
+ * the statement to the QEs, so a validation failure raised only inside
+ * CreateLakeTable() would surface as a confusing QE-annotated error.
+ */
+static void
+ResolveLakeTableOptions(CreateLakeTableStmt *stmt,
+ Oid *catalog_oid_out, Oid *volume_oid_out)
+{
+ const char *catalog_name;
+ const char *volume_name;
+
+ /*
+ * Lake tables are unusable without an extension providing the iceberg
+ * table access method; check it first so the install hint takes
+ * precedence over catalog/volume resolution errors.
+ */
+ if (!OidIsValid(GetIcebergTableAmOid(true)))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("table access method \"%s\" does not exist",
+ ICEBERG_TABLE_AM_NAME),
+ errhint("CREATE LAKE TABLE ... USING ICEBERG requires an extension that provides the \"%s\" table access method.",
+ ICEBERG_TABLE_AM_NAME)));
+
+ /* Validate the table format named in the USING clause */
+ validate_table_type(stmt->table_type);
Review Comment:
Let's validate table type before checking for AM existence to make it possible to pass any table type in the future
```suggestion
/* Validate the table format named in the USING clause */
validate_table_type(stmt->table_type);
/*
* Lake tables are unusable without an extension providing
* a table access method; check it first so the install hint takes
* precedence over catalog/volume resolution errors.
*/
if (!OidIsValid(get_table_am_oid(stmt->table_type, true)))
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("table access method \"%s\" does not exist",
stmt->table_type),
errhint("CREATE LAKE TABLE ... USING %s requires an extension that provides the \"%s\" table access method.",
stmt->table_type, stmt->table_type)));
```
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,448 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
+
+/*
+ * RelationIsIcebergTable
Review Comment:
I suggest to rename this function to `RelationIsLakeTable` and check for the table oid existence in `pg_lake_table`. It seems that we call this function to check whether relation is lake table, but not iceberg table only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3635759593
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,157 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
+
+-- The default catalog/volume GUCs verify that the object exists
+SET iceberg_default_catalog = 'no_such_catalog'; -- fail
+SET iceberg_default_volume = 'no_such_volume'; -- fail
+
+-- Simulate a datalake provider with a heap-backed iceberg AM
+CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler;
+
+-- CREATE LAKE TABLE with explicit catalog and volume. The format is the
+-- table's access method (pg_class.relam) and its options are the relation's
+-- reloptions; pg_lake_table records only the catalog/volume binding.
+CREATE LAKE TABLE lake_test_t1 (a int, b text) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol;
+SELECT c.relname, am.amname, c.reloptions, fc.fcname, fv.fvname
+ FROM pg_lake_table lt
+ JOIN pg_class c ON c.oid = lt.ltrelid
+ JOIN pg_am am ON am.oid = c.relam
+ JOIN pg_foreign_catalog fc ON fc.oid = lt.ltforeign_catalog
+ JOIN pg_foreign_volume fv ON fv.oid = lt.ltforeign_volume;
+-- Lake tables are always DISTRIBUTED RANDOMLY (policytype 'p', no distkey)
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t1'::regclass;
+
+-- The (heap-backed) table is usable
+INSERT INTO lake_test_t1 VALUES (1, 'x'), (2, 'y');
+SELECT count(*) FROM lake_test_t1;
+
+-- OPTIONS become the relation's reloptions and are validated by the access
+-- method: a value the AM accepts is stored, an unknown one is rejected.
+CREATE LAKE TABLE lake_test_opt (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fillfactor '70');
+SELECT reloptions FROM pg_class WHERE relname = 'lake_test_opt';
+CREATE LAKE TABLE lake_test_optbad (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (bogus_opt 'x'); -- fail, AM rejects unknown option
+DROP LAKE TABLE lake_test_opt;
+
+-- Catalog and volume are both required
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG VOLUME lake_test_vol; -- fail, no catalog
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG CATALOG lake_test_cat; -- fail, no volume
+
+-- ... unless the GUCs provide defaults
+SET iceberg_default_catalog = 'lake_test_cat';
+SET iceberg_default_volume = 'lake_test_vol';
Review Comment:
@andr-sokolov
1. about renaming GUCs
The current design supports multiple formats. If we rename iceberg_default_volume to a generic GUC parameter like lake_default_volume, we will not be able to determine whether a lake table uses Iceberg or another format. In my opinion, there should be one default configuration per format, such as iceberg_default_catalog and hudi_default_catalog. This eliminates ambiguity, at the cost of introducing additional GUC variables.
2. about CREATE FOREIGN CATALOG ... FORMAT 'xxx'
```
CREATE FOREIGN CATALOG ... FORMAT 'iceberg' TYPE 'hive';
CREATE FOREIGN VOLUME ... FORMAT 'iceberg' SERVER ...;
```
This binds catalogs and volumes to the Iceberg format. I am unsure whether catalogs and volumes will be reused across multiple formats in the future. Nevertheless, such binding lacks flexibility. Besides, we already have corresponding GUC parameters, making the explicit FORMAT clause redundant. When creating lake tables, users will clearly specify the catalog and volume to use. Therefore, I suggest not binding catalogs or volumes to specific formats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3655222183
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,158 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
Review Comment:
@andr-sokolov @leborchuk @yjhjstz I looked into how Crunchy Data Warehouse (now open-sourced by Snowflake as [pg_lake](https://github.com/Snowflake-Labs/pg_lake)) handles the catalog — hope this helps answer the question above.
**TL;DR: Crunchy doesn't specify a catalog at CREATE time, because Postgres itself acts as the catalog.**
- Managed Iceberg tables are created with plain `CREATE TABLE ... USING iceberg`. The Iceberg metadata lives inside Postgres and is exposed through an `iceberg_tables` view compatible with the Iceberg JDBC/SQL catalog protocol, so Spark / pyiceberg / iceberg-rust connect **to Postgres as the catalog** (JDBC with Postgres credentials; `catalog_name` = the database name). ([Crunchy docs](https://docs.crunchybridge.com/warehouse/iceberg))
- I couldn't find any syntax for attaching an external catalog (Glue / HMS / REST) — the product doesn't seem to have that concept, which would explain why it doesn't appear in the presentation.
- Storage location is a GUC default plus an optional per-table reloption: `SET crunchy_iceberg.default_location_prefix TO 's3://...'`, or `CREATE TABLE ... USING iceberg WITH (location = 's3://...')`. With neither set, the cluster's bundled managed storage is used — so in the demo nothing needs to be specified at all. @leborchuk's reading ("good for pre-configured database instances") matches what I found.
One more data point that may be relevant: pg_lake is currently adding external Iceberg REST catalog support as a **client** ([pg_lake#94](https://github.com/Snowflake-Labs/pg_lake/issues/94)), and that roadmap mentions handling multiple endpoints/credentials "likely using a Postgres `SERVER`-like concept" — quite close to what `CREATE FOREIGN CATALOG ... SERVER` does in this PR. So the two designs may be converging from different starting points.
For reference, the two syntax shapes side by side:
- Plain `CREATE TABLE ... USING iceberg`: the catalog/volume bindings move into reloptions, e.g. `WITH (catalog 'c', volume 'v')`, with GUC defaults; no new grammar, and plain `DROP TABLE` applies (the Spark / pg_lake convention).
- `CREATE LAKE TABLE ... USING <format> CATALOG c VOLUME v`: catalog/volume are first-class clauses with dedicated `CREATE/DROP LAKE TABLE` commands (the current PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3541911470
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,483 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
+
+/*
+ * RelationIsIcebergTable
+ *
+ * True iff the relation uses the iceberg table access method. Resolved by
+ * access method name so the kernel does not depend on any particular
+ * extension's OID assignments.
+ */
+bool
+RelationIsIcebergTable(Relation rel)
Review Comment:
Adding RELKIND_ICEBERG requires modifications across all reference points, which will be a substantial change. Do we plan to support multiple AMs in the future? From my understanding, CREATE ICEBERG statements natively only create Iceberg-related tables. you mean like hudi, Paimon ?,Could you share your thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3541919169
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,483 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
+
+/*
+ * RelationIsIcebergTable
+ *
+ * True iff the relation uses the iceberg table access method. Resolved by
+ * access method name so the kernel does not depend on any particular
+ * extension's OID assignments.
+ */
+bool
+RelationIsIcebergTable(Relation rel)
Review Comment:
If that’s the case, a syntax like CREATE LAKEHOUSE xxx USING xxx would be more appropriate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3541875937
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,483 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
Review Comment:
1. Does this AM refer to the minimal Iceberg AM implementation that does not support scans or inserts? Is that correct?
2. One more question: CREATE ICEBERG TABLE USING <am>
Is this syntax designed to support extensions via other <am> frameworks? From my understanding, there should be no need to specify USING am if we're already using CREATE ICEBERG TABLE, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3535306719
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
May be. But I think every catalog has its type. So type is not an option, but a required property, which should be in `pg_foreign_catalog` as a column and in grammar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3541859592
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
The currently supported types are hive, hdfs, polaris, s3 and builtin.
```
CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'builtin';
CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 's3';
CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive';
CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'polaris';
CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hdfs';
```
Is it correct like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3636494724
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,158 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
Review Comment:
We can align syntax with crunchydata.com
[Building a Postgres Data Warehouse using Iceberg.pdf](https://github.com/user-attachments/files/30297764/Building.a.Postgres.Data.Warehouse.using.Iceberg.pdf)
```sql
> Iceberg as a Postgres table format Capture queries, writes, & schema changes to provide a transactional table experience for Iceberg in S3.
create table chats (
message_id bigserial not null,
thread_id bigint not null,
…
) using iceberg;
```
@leborchuk @andr-sokolov
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3556049890
##########
src/backend/catalog/heap.c:
##########
@@ -2322,6 +2323,10 @@ heap_drop_with_catalog(Oid relid)
*/
CheckTableForSerializableConflictIn(rel);
+ /* If this is a lake table, remove its pg_lake_table entry */
+ if (RelationIsIcebergTable(rel))
+ RemoveLakeTableEntry(relid);
Review Comment:
I can migrate this logic to object_access_hook, but the issue is that the AM layer hasn’t been implemented yet on my end, which blocks the migration.
So do I need to implement a default plugin for the AM layer first? Then replace this default AM plugin later once the official datalake plugin is fully rolled out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3774391929
##########
contrib/Makefile:
##########
@@ -25,6 +25,7 @@ SUBDIRS = \
btree_gin \
btree_gist \
citext \
+ datalake_fdw \
Review Comment:
Good call -- done in 6f40dd0f7db, following the PAX mechanism: `PGAC_ARG_BOOL` defaulting to `no`, substituted into `Makefile.global`, and a conditional in `contrib/Makefile` that moves the directory to `ALWAYS_SUBDIRS` when disabled so the clean targets still reach it.
Three things I decided while doing it, all cheap to reverse if you disagree:
- **The option is spelled `--enable-datalake-fdw`, with dashes.** Every option in this tree uses dashes (`--enable-tap-tests`, `--enable-ic-udp2`, `--enable-catalog-ext`), and autoconf maps the dashes to `enable_datalake_fdw` for the Makefile variable, so the directory name still matches. Happy to use the underscore form if you would rather it read exactly like the directory.
- **No `AC_DEFINE`.** PAX and ic-udp2 define a macro because C code tests it; nothing here does, and adding an unread macro would mean touching `src/include/pg_config.h.in` for no reader. `--enable-pxf` and `--enable-orafce` already use this shorter four-argument form.
- **`--enable-datalake-fdw` is also added to `devops/build/automation/cloudberry/scripts/configure-cloudberry.sh`**, next to `--enable-pax`. Without it the `ic-datalake-fdw` CI job would install nothing and silently test nothing. I deliberately did *not* add it to `coverity.yml` and `sonarqube.yml`, which also pass `--enable-pax` -- say the word if you want the module in the weekly scans too.
**One thing you should know about `configure`.** I regenerated it by hand rather than running autoconf over the whole file, because the committed `configure` and `configure.ac` are currently out of sync in *both* directions:
- `configure.ac` has a restructured PAX liburing check (the `case $host_os in linux*)` block with the macOS fallback comment) that was never regenerated into `configure`;
- `configure` has a Darwin python shared-library lookup (trying `.dylib` in addition to `DLSUFFIX`) that does not exist in `configure.ac`.
A full `autoconf` run therefore produced twelve hunks, only four of which were mine; taking all of them would have swept those two unrelated changes into this PR and, in the python case, reverted working behaviour. So I applied only the four hunks for this option and left the drift alone. It is worth someone reconciling those two separately -- I did not want to do it inside this PR.
Verified by configuring a real tree both ways, not by inspection:
| | configure says | `Makefile.global` | `SUBDIRS` | `ALWAYS_SUBDIRS` |
|---|---|---|---|---|
| `--enable-datalake-fdw` | `... yes` | `enable_datalake_fdw = yes` | has `datalake_fdw` | -- |
| default | `... no` | `enable_datalake_fdw = no` | -- | has `datalake_fdw` |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3620406567
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,154 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
+
+-- The default catalog/volume GUCs verify that the object exists
+SET iceberg_default_catalog = 'no_such_catalog'; -- fail
+SET iceberg_default_volume = 'no_such_volume'; -- fail
+
+-- Simulate a datalake provider with a heap-backed iceberg AM
+CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler;
+
+-- CREATE LAKE TABLE with explicit catalog and volume
+CREATE LAKE TABLE lake_test_t1 (a int, b text) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fileformat 'parquet');
+SELECT c.relname, lt.lttable_type, lt.ltoptions, fc.fcname, fv.fvname
+ FROM pg_lake_table lt
+ JOIN pg_class c ON c.oid = lt.ltrelid
+ JOIN pg_foreign_catalog fc ON fc.oid = lt.ltforeign_catalog
+ JOIN pg_foreign_volume fv ON fv.oid = lt.ltforeign_volume;
+SELECT a.amname FROM pg_am a JOIN pg_class c ON c.relam = a.oid WHERE c.relname = 'lake_test_t1';
+-- Lake tables are always DISTRIBUTED RANDOMLY (policytype 'p', no distkey)
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t1'::regclass;
+
+-- The (heap-backed) table is usable
+INSERT INTO lake_test_t1 VALUES (1, 'x'), (2, 'y');
+SELECT count(*) FROM lake_test_t1;
+
+-- Lake tables get a TOAST table like plain tables, so wide values work
+SELECT reltoastrelid <> 0 AS has_toast FROM pg_class WHERE relname = 'lake_test_t1';
+INSERT INTO lake_test_t1 VALUES (3, repeat('x', 500000));
+SELECT a, length(b) FROM lake_test_t1 WHERE a = 3;
Review Comment:
TOAST table is needed to save data in it, but lake table doesn't save its data in database files. Do we really need this part of the test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3556785663
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
I added a type category to the grammar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3548712356
##########
src/backend/catalog/heap.c:
##########
@@ -2322,6 +2323,10 @@ heap_drop_with_catalog(Oid relid)
*/
CheckTableForSerializableConflictIn(rel);
+ /* If this is a lake table, remove its pg_lake_table entry */
+ if (RelationIsIcebergTable(rel))
+ RemoveLakeTableEntry(relid);
Review Comment:
can RemoveLakeTableEntry put into extension via object_access_hook ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3542028479
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
Ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3557092247
##########
src/bin/psql/tab-complete.c:
##########
@@ -3819,7 +3854,15 @@ psql_completion(const char *text, int start, int end)
else if (Matches("DROP", "AGGREGATE|FUNCTION|PROCEDURE|ROUTINE", MatchAny, "("))
COMPLETE_WITH_FUNCTION_ARG(prev2_wd);
else if (Matches("DROP", "FOREIGN"))
- COMPLETE_WITH("DATA WRAPPER", "TABLE");
+ COMPLETE_WITH("CATALOG", "DATA WRAPPER", "TABLE", "VOLUME");
+ else if (Matches("DROP", "FOREIGN", "CATALOG"))
+ COMPLETE_WITH_QUERY(Query_for_list_of_foreign_catalogs);
+ else if (Matches("DROP", "FOREIGN", "VOLUME"))
+ COMPLETE_WITH_QUERY(Query_for_list_of_foreign_volumes);
+ else if (Matches("DROP", "ICEBERG"))
+ COMPLETE_WITH("TABLE");
+ else if (Matches("DROP", "ICEBERG", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables);
Review Comment:
Is it possible to write a query to get here iceberg tables only?
##########
src/include/commands/laketablecmds.h:
##########
@@ -0,0 +1,61 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.h
+ * prototypes for laketablecmds.c.
+ *
+ * src/include/commands/laketablecmds.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef LAKETABLECMDS_H
+#define LAKETABLECMDS_H
+
+#include "catalog/pg_lake_table.h"
+#include "nodes/parsenodes.h"
+#include "utils/guc.h"
+#include "utils/rel.h"
+
+/*
+ * Name of the table access method lake tables are created with. The
+ * kernel only provides the DDL scaffolding; the access method itself is
+ * provided by a datalake extension.
+ */
+#define ICEBERG_TABLE_AM_NAME "iceberg"
+
+/* GUC variables */
+extern char *iceberg_default_catalog;
+extern char *iceberg_default_volume;
+
+/* GUC check hooks */
+extern bool check_iceberg_default_catalog(char **newval, void **extra, GucSource source);
+extern bool check_iceberg_default_volume(char **newval, void **extra, GucSource source);
+
+/* Functions to get default values */
+extern const char *GetDefaultIcebergCatalog(void);
+extern const char *GetDefaultIcebergVolume(void);
+
+/* Lake table management */
+extern Oid GetIcebergTableAmOid(bool missing_ok);
+extern bool RelationIsIcebergTable(Relation rel);
+extern void ValidateLakeTableOptions(CreateLakeTableStmt *stmt);
Review Comment:
I think the `Options` word in the function name is not correct because `CreateLakeTableStmt` contains the `options` field and reading the function name I get a false expectation that the function validates the `options` field only. I suggest to rename the function. For example, `ValidateLakeTableStmt`, because it validates all fields of the structure
```suggestion
extern void ValidateLakeTableStmt(CreateLakeTableStmt *stmt);
```
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,150 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%' ORDER BY 1;
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (path 's3://bucket/prefix');
Review Comment:
The proposal says that volume is "Where data files physically live". Is it possible that a volume can be without path? What about moving path from options to required first-class properties?
##########
src/backend/foreign/foreign.c:
##########
@@ -982,6 +984,98 @@ get_foreign_server_oid(const char *servername, bool missing_ok)
return oid;
}
+/*
+ * get_foreign_catalog_oid - given a foreign catalog name, look up the OID
+ *
+ * If missing_ok is false, throw an error if name not found. If true, just
+ * return InvalidOid.
+ */
+Oid
+get_foreign_catalog_oid(const char *catalogname, bool missing_ok)
+{
+ Oid oid;
+
+ oid = GetSysCacheOid1(FOREIGNCATALOGNAME,
+ Anum_pg_foreign_catalog_oid,
+ CStringGetDatum(catalogname));
+ if (!OidIsValid(oid) && !missing_ok)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ catalogname)));
+
+ return oid;
+}
+
+/*
+ * get_foreign_volume_oid - given a foreign volume name, look up the OID
+ *
+ * If missing_ok is false, throw an error if name not found. If true, just
+ * return InvalidOid.
+ */
+Oid
+get_foreign_volume_oid(const char *volumename, bool missing_ok)
+{
+ Oid oid;
+
+ oid = GetSysCacheOid1(FOREIGNVOLUMENAME,
+ Anum_pg_foreign_volume_oid,
+ CStringGetDatum(volumename));
+ if (!OidIsValid(oid) && !missing_ok)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ volumename)));
+
+ return oid;
+}
+
+/*
+ * GetForeignVolumeByName - look up a foreign volume by name
+ */
+ForeignVolume *
+GetForeignVolumeByName(const char *volumename, bool missing_ok)
+{
+ HeapTuple tp;
+ Form_pg_foreign_volume fvform;
+ ForeignVolume *volume;
+ Datum datum;
+ bool isnull;
+
+ tp = SearchSysCache1(FOREIGNVOLUMENAME,
+ PointerGetDatum(volumename));
+ if (!HeapTupleIsValid(tp))
+ {
+ if (!missing_ok)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ volumename)));
+ return NULL;
+ }
+
+ fvform = (Form_pg_foreign_volume) GETSTRUCT(tp);
+
+ volume = (ForeignVolume *) palloc(sizeof(ForeignVolume));
+ volume->volumeid = fvform->oid;
+ volume->serverid = fvform->fvserver;
+ volume->volumename = pstrdup(NameStr(fvform->fvname));
+
+ /* Extract the volume options */
+ datum = SysCacheGetAttr(FOREIGNVOLUMENAME,
+ tp,
+ Anum_pg_foreign_volume_fvoptions,
+ &isnull);
+ if (isnull)
+ volume->options = NIL;
+ else
+ volume->options = untransformRelOptions(datum);
Review Comment:
I suggest using ternary operator to make the code more compact and to explicitly write that for any value of the condition, we change one variable
```suggestion
volume->options = (isnull) ? NIL : untransformRelOptions(datum);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3557307183
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,150 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%' ORDER BY 1;
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (path 's3://bucket/prefix');
Review Comment:
During design, this path functions as a prefix and is optional for users to specify. If a bucket is available, you may fully specify the path within the Iceberg table configuration. like name is base_path in Discussion: https://github.com/apache/cloudberry/discussions/1683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#issuecomment-4965255918
> When a foreign table has been created and I press tab after `drop table`, there is no foreign table names in the suggestion. I cannot use `drop table` to remove a foreign table.
>
> ```
> postgres=# CREATE FOREIGN TABLE ft (i int) SERVER lake_test_srv;
> CREATE FOREIGN TABLE
> postgres=# drop table
> gp_toolkit. information_schema. public. t t2
> postgres=# drop table ft;
> ERROR: "ft" is not a table
> HINT: Use DROP FOREIGN TABLE to remove a foreign table.
> postgres=#
> ```
>
> Iceberg table name is displayed in the suggestion after `drop table`. I can use `drop table` to remove an iceberg table.
>
> ```
> postgres=# CREATE ICEBERG TABLE lake_test_t1 (a int, b text) CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fileformat 'parquet');
> CREATE LAKE TABLE
> postgres=# drop table
> gp_toolkit. information_schema. lake_test_t1 public. t t2
> postgres=# drop table lake_test_t1;
> DROP TABLE
> postgres=#
> ```
>
> Is it correct? I expect that iceberg table is something like foreign table and should be processed using a similar syntax
It's been updated. Tables are now dropped using the DROP ICEBERG TABLE syntax.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3635929892
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,157 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
+
+-- The default catalog/volume GUCs verify that the object exists
+SET iceberg_default_catalog = 'no_such_catalog'; -- fail
+SET iceberg_default_volume = 'no_such_volume'; -- fail
+
+-- Simulate a datalake provider with a heap-backed iceberg AM
+CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler;
+
+-- CREATE LAKE TABLE with explicit catalog and volume. The format is the
+-- table's access method (pg_class.relam) and its options are the relation's
+-- reloptions; pg_lake_table records only the catalog/volume binding.
+CREATE LAKE TABLE lake_test_t1 (a int, b text) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol;
+SELECT c.relname, am.amname, c.reloptions, fc.fcname, fv.fvname
+ FROM pg_lake_table lt
+ JOIN pg_class c ON c.oid = lt.ltrelid
+ JOIN pg_am am ON am.oid = c.relam
+ JOIN pg_foreign_catalog fc ON fc.oid = lt.ltforeign_catalog
+ JOIN pg_foreign_volume fv ON fv.oid = lt.ltforeign_volume;
+-- Lake tables are always DISTRIBUTED RANDOMLY (policytype 'p', no distkey)
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t1'::regclass;
+
+-- The (heap-backed) table is usable
+INSERT INTO lake_test_t1 VALUES (1, 'x'), (2, 'y');
+SELECT count(*) FROM lake_test_t1;
+
+-- OPTIONS become the relation's reloptions and are validated by the access
+-- method: a value the AM accepts is stored, an unknown one is rejected.
+CREATE LAKE TABLE lake_test_opt (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fillfactor '70');
+SELECT reloptions FROM pg_class WHERE relname = 'lake_test_opt';
+CREATE LAKE TABLE lake_test_optbad (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (bogus_opt 'x'); -- fail, AM rejects unknown option
+DROP LAKE TABLE lake_test_opt;
+
+-- Catalog and volume are both required
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG VOLUME lake_test_vol; -- fail, no catalog
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG CATALOG lake_test_cat; -- fail, no volume
+
+-- ... unless the GUCs provide defaults
+SET iceberg_default_catalog = 'lake_test_cat';
+SET iceberg_default_volume = 'lake_test_vol';
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG;
+RESET iceberg_default_catalog;
+RESET iceberg_default_volume;
+
+-- A DISTRIBUTED clause is rejected (lake tables are always distributed randomly)
+CREATE LAKE TABLE lake_test_t3 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol DISTRIBUTED BY (a); -- fail
+
+-- The USING clause names the table format; only ICEBERG is supported (any case)
+CREATE LAKE TABLE lake_test_bad0 (a int) USING heap CATALOG lake_test_cat VOLUME lake_test_vol; -- fail, unsupported format
+CREATE LAKE TABLE lake_test_t4 (a int) USING "IceBerg" CATALOG lake_test_cat VOLUME lake_test_vol; -- quoted mixed-case format resolves the iceberg AM
+
+-- The iceberg AM is rejected for every path other than CREATE LAKE TABLE
+CREATE TABLE lake_test_bad1 (a int) USING iceberg DISTRIBUTED RANDOMLY; -- fail
+CREATE TABLE lake_test_bad2 USING iceberg AS SELECT 1 AS a DISTRIBUTED RANDOMLY; -- fail
+SET default_table_access_method = iceberg;
+CREATE TABLE lake_test_bad3 (a int) DISTRIBUTED RANDOMLY; -- fail
+RESET default_table_access_method;
+CREATE TABLE lake_test_heap (a int) DISTRIBUTED RANDOMLY;
+ALTER TABLE lake_test_heap SET ACCESS METHOD iceberg; -- fail
+ALTER TABLE lake_test_t1 SET ACCESS METHOD heap; -- fail
+ALTER TABLE lake_test_t1 SET DISTRIBUTED BY (a); -- fail
+
+-- Only the owner can drop a catalog or volume
+CREATE ROLE regress_lake_user;
+SET ROLE regress_lake_user;
+DROP FOREIGN CATALOG lake_test_cat; -- fail, not owner
+DROP FOREIGN VOLUME lake_test_vol; -- fail, not owner
+RESET ROLE;
+
+-- Dependencies: the server holds the catalog/volume, which hold the tables
+DROP SERVER lake_test_srv; -- fail, catalog and volume depend on it
+DROP FOREIGN CATALOG lake_test_cat; -- fail, tables depend on it
+
+-- Dropping a lake table removes its pg_lake_table entry
+DROP LAKE TABLE lake_test_t1;
+SELECT count(*) FROM pg_lake_table lt JOIN pg_class c ON c.oid = lt.ltrelid
+ WHERE c.relname = 'lake_test_t1';
Review Comment:
This query returns 0 when a lake table entry is deleted from pg_class only. It doesn't check that `DROP LAKE TABLE` deletes entry from pg_lake_table. We should save the table oid before `DROP LAKE TABLE` and find the oid in pg_lake_table after the query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3592261522
##########
src/include/tcop/cmdtaglist.h:
##########
@@ -176,10 +179,13 @@ PG_CMDTAG(CMDTAG_DROP_DOMAIN, "DROP DOMAIN", true, false, false)
PG_CMDTAG(CMDTAG_DROP_DYNAMIC_TABLE, "DROP DYNAMIC TABLE", true, false, false)
PG_CMDTAG(CMDTAG_DROP_EVENT_TRIGGER, "DROP EVENT TRIGGER", false, false, false)
PG_CMDTAG(CMDTAG_DROP_EXTENSION, "DROP EXTENSION", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_FOREIGN_CATALOG, "DROP FOREIGN CATALOG", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FOREIGN_DATA_WRAPPER, "DROP FOREIGN DATA WRAPPER", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FOREIGN_TABLE, "DROP FOREIGN TABLE", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_FOREIGN_VOLUME, "DROP FOREIGN VOLUME", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FUNCTION, "DROP FUNCTION", true, false, false)
PG_CMDTAG(CMDTAG_DROP_INDEX, "DROP INDEX", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_LAKE_TABLE, "DROP LAKE TABLE", true, false, false)
Review Comment:
This is a good question. I suggest other reviewers join the discussion: which approach is better, `CREATE ICEBERG`, or `CREATE LAKE TABLE ... USING ICEBERG`?
`CREATE LAKE TABLE ... USING ICEBERG` This offers the benefit of decoupling the data lake from the kernel. The AM plugin can be controlled via registration.
@andr-sokolov @yjhjstz @leborchuk @my-ship-it What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3600754028
##########
src/include/tcop/cmdtaglist.h:
##########
@@ -176,10 +179,13 @@ PG_CMDTAG(CMDTAG_DROP_DOMAIN, "DROP DOMAIN", true, false, false)
PG_CMDTAG(CMDTAG_DROP_DYNAMIC_TABLE, "DROP DYNAMIC TABLE", true, false, false)
PG_CMDTAG(CMDTAG_DROP_EVENT_TRIGGER, "DROP EVENT TRIGGER", false, false, false)
PG_CMDTAG(CMDTAG_DROP_EXTENSION, "DROP EXTENSION", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_FOREIGN_CATALOG, "DROP FOREIGN CATALOG", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FOREIGN_DATA_WRAPPER, "DROP FOREIGN DATA WRAPPER", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FOREIGN_TABLE, "DROP FOREIGN TABLE", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_FOREIGN_VOLUME, "DROP FOREIGN VOLUME", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FUNCTION, "DROP FUNCTION", true, false, false)
PG_CMDTAG(CMDTAG_DROP_INDEX, "DROP INDEX", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_LAKE_TABLE, "DROP LAKE TABLE", true, false, false)
Review Comment:
@MisterRaindrop I vote for `CREATE LAKE TABLE ... USING ICEBERG` too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3629067354
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,157 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
+
+-- The default catalog/volume GUCs verify that the object exists
+SET iceberg_default_catalog = 'no_such_catalog'; -- fail
+SET iceberg_default_volume = 'no_such_volume'; -- fail
+
+-- Simulate a datalake provider with a heap-backed iceberg AM
+CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler;
+
+-- CREATE LAKE TABLE with explicit catalog and volume. The format is the
+-- table's access method (pg_class.relam) and its options are the relation's
+-- reloptions; pg_lake_table records only the catalog/volume binding.
+CREATE LAKE TABLE lake_test_t1 (a int, b text) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol;
+SELECT c.relname, am.amname, c.reloptions, fc.fcname, fv.fvname
+ FROM pg_lake_table lt
+ JOIN pg_class c ON c.oid = lt.ltrelid
+ JOIN pg_am am ON am.oid = c.relam
+ JOIN pg_foreign_catalog fc ON fc.oid = lt.ltforeign_catalog
+ JOIN pg_foreign_volume fv ON fv.oid = lt.ltforeign_volume;
+-- Lake tables are always DISTRIBUTED RANDOMLY (policytype 'p', no distkey)
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t1'::regclass;
+
+-- The (heap-backed) table is usable
+INSERT INTO lake_test_t1 VALUES (1, 'x'), (2, 'y');
+SELECT count(*) FROM lake_test_t1;
+
+-- OPTIONS become the relation's reloptions and are validated by the access
+-- method: a value the AM accepts is stored, an unknown one is rejected.
+CREATE LAKE TABLE lake_test_opt (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fillfactor '70');
+SELECT reloptions FROM pg_class WHERE relname = 'lake_test_opt';
+CREATE LAKE TABLE lake_test_optbad (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (bogus_opt 'x'); -- fail, AM rejects unknown option
+DROP LAKE TABLE lake_test_opt;
+
+-- Catalog and volume are both required
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG VOLUME lake_test_vol; -- fail, no catalog
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG CATALOG lake_test_cat; -- fail, no volume
+
+-- ... unless the GUCs provide defaults
+SET iceberg_default_catalog = 'lake_test_cat';
+SET iceberg_default_volume = 'lake_test_vol';
Review Comment:
What about renaming GUCs
```suggestion
SET lake_default_catalog = 'lake_test_cat';
SET lake_default_volume = 'lake_test_vol';
```
and adding a new GUC with default lake table AM? The new GUC with default value "iceberg" will be useful to don't write `USING ICEBERG` in every `CREATE LAKE TABLE` query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3828851558
##########
contrib/datalake_fdw/src/iceberg_volume_fdw/iceberg_volume_fdw.c:
##########
@@ -0,0 +1,157 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * iceberg_volume_fdw.c
+ * Option validator for Iceberg volume foreign servers.
+ *
+ * IDENTIFICATION
+ * contrib/datalake_fdw/src/iceberg_volume_fdw/iceberg_volume_fdw.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/reloptions.h"
+#include "am_iceberg/pg_iceberg_options.h"
+#include "catalog/pg_foreign_data_wrapper.h"
+#include "catalog/pg_foreign_server.h"
+#include "catalog/pg_user_mapping.h"
+#include "commands/defrem.h"
+#include "common/dl_option_util.h"
+#include "fmgr.h"
+#include "iceberg_volume_fdw/iceberg_volume_option.h"
+
+PG_FUNCTION_INFO_V1(iceberg_volume_fdw_validator);
+
+static bool is_volume_server_option(const char *name);
+static bool is_volume_user_mapping_option(const char *name);
+
+static bool
+is_volume_server_option(const char *name)
+{
+ return strcmp(name, DATALAKE_ICEBERG_VOLUME_BASE_PATH) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_ENDPOINT) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_REGION) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_PATH_STYLE_ACCESS) == 0;
+}
+
+/*
+ * Every credential here is optional, so that ambient storage credentials -- an
+ * instance profile, a ticket cache -- remain a valid deployment choice.
+ */
+static bool
+is_volume_user_mapping_option(const char *name)
+{
+ return strcmp(name, DATALAKE_ICEBERG_VOLUME_USERNAME) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_AWS_ACCESS_KEY_ID) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_AWS_SECRET_ACCESS_KEY) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_AWS_SESSION_TOKEN) == 0;
+}
+
+Datum
+iceberg_volume_fdw_validator(PG_FUNCTION_ARGS)
+{
+ List *options = untransformRelOptions(PG_GETARG_DATUM(0));
+ Oid catalog = PG_GETARG_OID(1);
+ ListCell *lc;
+ IcebergVolumeServerOptions server_options;
+ IcebergForeignVolumeOptions volume_options;
Review Comment:
What about using `{0}` instead of `memset`? The code will be simpler and some potential errors, which compiler cannot detect, will be avoided
```suggestion
IcebergVolumeServerOptions server_options = {0};
IcebergForeignVolumeOptions volume_options = {0};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3774249922
##########
contrib/Makefile:
##########
@@ -25,6 +25,7 @@ SUBDIRS = \
btree_gin \
btree_gist \
citext \
+ datalake_fdw \
Review Comment:
Is `datalake_fdw` enabled by default? We can disable it by default and enable it using the `--enable-datalake_fdw` configure option, the way like PAX.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3556804953
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,483 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
+
+/*
+ * RelationIsIcebergTable
+ *
+ * True iff the relation uses the iceberg table access method. Resolved by
+ * access method name so the kernel does not depend on any particular
+ * extension's OID assignments.
+ */
+bool
+RelationIsIcebergTable(Relation rel)
Review Comment:
At present, I do not expect multiple Iceberg AM plugins to exist, hence adding RELKIND_ICEBERG is unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "Vlasdislav (via GitHub)" <gi...@apache.org>.
Vlasdislav commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3527932261
##########
contrib/pax_storage/src/test/regress/expected/oidjoins.out:
##########
@@ -235,6 +235,10 @@ NOTICE: checking pg_foreign_data_wrapper {fdwhandler} => pg_proc {oid}
NOTICE: checking pg_foreign_data_wrapper {fdwvalidator} => pg_proc {oid}
NOTICE: checking pg_foreign_server {srvowner} => pg_authid {oid}
NOTICE: checking pg_foreign_server {srvfdw} => pg_foreign_data_wrapper {oid}
+NOTICE: checking pg_foreign_catalog {fcowner} => pg_authid {oid}
+NOTICE: checking pg_foreign_catalog {fcserver} => pg_foreign_server {oid}
+NOTICE: checking pg_foreign_volume {fvowner} => pg_authid {oid}
+NOTICE: checking pg_foreign_volume {fvserver} => pg_foreign_server {oid}
Review Comment:
Maybe you will use `SET client_min_messages TO error;` to ignore `NOTICE` in whole file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#issuecomment-4954736392
When a foreign table has been created and I press tab after `drop table`, there is no foreign table names in the suggestion. I cannot use `drop table` to remove a foreign table.
```
postgres=# CREATE FOREIGN TABLE ft (i int) SERVER lake_test_srv;
CREATE FOREIGN TABLE
postgres=# drop table
gp_toolkit. information_schema. public. t t2
postgres=# drop table ft;
ERROR: "ft" is not a table
HINT: Use DROP FOREIGN TABLE to remove a foreign table.
postgres=#
```
Iceberg table name is displayed in the suggestion after `drop table`. I can use `drop table` to remove an iceberg table.
```
postgres=# CREATE ICEBERG TABLE lake_test_t1 (a int, b text) CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fileformat 'parquet');
CREATE LAKE TABLE
postgres=# drop table
gp_toolkit. information_schema. lake_test_t1 public. t t2
postgres=# drop table lake_test_t1;
DROP TABLE
postgres=#
```
Is it correct? I expect that iceberg table is something like foreign table and should be processed using a similar syntax
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#issuecomment-4955052185
What about adding new syntax in psql help (`\h`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3628813258
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,458 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
+
+/*
+ * RelationIsLakeTable
+ *
+ * True iff the relation has a pg_lake_table entry, i.e. it was created by
+ * CREATE LAKE TABLE. Lake tables are told apart from ordinary relations by
+ * this catalog membership rather than by their access method.
+ */
+bool
+RelationIsLakeTable(Relation rel)
+{
+ Relation ltRel;
+ ScanKeyData skey;
+ SysScanDesc scan;
+ bool found;
+
+ ltRel = table_open(LakeTableRelationId, AccessShareLock);
+ ScanKeyInit(&skey,
+ Anum_pg_lake_table_ltrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationGetRelid(rel)));
+ scan = systable_beginscan(ltRel, LakeTableRelidIndexId, true, NULL, 1, &skey);
+ found = HeapTupleIsValid(systable_getnext(scan));
+ systable_endscan(scan);
+ table_close(ltRel, AccessShareLock);
+
+ return found;
+}
+
+/*
+ * Validate table type
+ */
+static void
+validate_table_type(const char *table_type)
+{
+ if (!table_type)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("lake table format cannot be NULL")));
+
+ if (strcmp(table_type, ICEBERG_TABLE_AM_NAME) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("unsupported lake table format \"%s\"", table_type),
+ errhint("The only supported format is ICEBERG (USING ICEBERG).")));
+}
+
+/*
+ * Validate foreign catalog exists
+ */
+static Oid
+validate_foreign_catalog(const char *catalog_name)
+{
+ if (!catalog_name || catalog_name[0] == '\0')
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("no foreign catalog specified"),
+ errhint("Specify CATALOG in CREATE LAKE TABLE or set iceberg_default_catalog.")));
+
+ return get_foreign_catalog_oid(catalog_name, false);
+}
+
+/*
+ * Validate foreign volume exists
+ */
+static Oid
+validate_foreign_volume(const char *volume_name)
+{
+ if (!volume_name || volume_name[0] == '\0')
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("no foreign volume specified"),
+ errhint("Specify VOLUME in CREATE LAKE TABLE or set iceberg_default_volume.")));
+
+ return get_foreign_volume_oid(volume_name, false);
+}
+
+/*
+ * ResolveLakeTableOptions
+ *
+ * Resolve and validate the table type, catalog and volume of a
+ * CreateLakeTableStmt, returning the catalog/volume OIDs.
+ *
+ * Also exposed (via ValidateLakeTableStmt) so ProcessUtilitySlow can run
+ * the validation on the QD before DefineRelation: DefineRelation dispatches
+ * the statement to the QEs, so a validation failure raised only inside
+ * CreateLakeTable() would surface as a confusing QE-annotated error.
+ */
+static void
+ResolveLakeTableOptions(CreateLakeTableStmt *stmt,
+ Oid *catalog_oid_out, Oid *volume_oid_out)
+{
+ const char *catalog_name;
+ const char *volume_name;
+
+ /* Validate the table format named in the USING clause first, so an
+ * unsupported format is reported before anything else. */
+ validate_table_type(stmt->table_type);
+
+ /*
+ * The format is implemented by a like-named table access method that a
+ * datalake extension provides; a lake table is unusable without it, so
+ * check it here (after the format) so the install hint takes precedence
+ * over catalog/volume resolution errors.
+ */
+ if (!OidIsValid(get_table_am_oid(stmt->table_type, true)))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("table access method \"%s\" does not exist",
+ stmt->table_type),
+ errhint("CREATE LAKE TABLE ... USING ICEBERG requires an extension that provides the \"%s\" table access method.",
+ stmt->table_type)));
Review Comment:
There is ICEBERG word in the hint. I suggest to replace it with stmt->table_type because table_type may be not iceberg only in the future
```suggestion
errhint("CREATE LAKE TABLE ... USING \"%s\" requires an extension that provides the \"%s\" table access method.",
stmt->table_type, stmt->table_type)));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3620230198
##########
src/test/regress/expected/lake_table.out:
##########
@@ -0,0 +1,318 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+ Table "pg_catalog.pg_foreign_catalog"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+-----------+--------+-----------+----------+---------+----------+--------------+-------------
+ oid | oid | | not null | | plain | |
+ fcname | name | | not null | | plain | |
+ fcowner | oid | | not null | | plain | |
+ fcserver | oid | | not null | | plain | |
+ fctype | text | C | not null | | extended | |
+ fcoptions | text[] | C | | | extended | |
+Indexes:
+ "pg_foreign_catalog_oid_index" PRIMARY KEY, btree (oid)
+ "pg_foreign_catalog_name_index" UNIQUE CONSTRAINT, btree (fcname)
+
+\d+ pg_foreign_volume
+ Table "pg_catalog.pg_foreign_volume"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+-----------+--------+-----------+----------+---------+----------+--------------+-------------
+ oid | oid | | not null | | plain | |
+ fvname | name | | not null | | plain | |
+ fvowner | oid | | not null | | plain | |
+ fvserver | oid | | not null | | plain | |
+ fvoptions | text[] | C | | | extended | |
+Indexes:
+ "pg_foreign_volume_oid_index" PRIMARY KEY, btree (oid)
+ "pg_foreign_volume_name_index" UNIQUE CONSTRAINT, btree (fvname)
+
+\d+ pg_lake_table
+ Table "pg_catalog.pg_lake_table"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+-------------------+--------+-----------+----------+---------+----------+--------------+-------------
+ ltrelid | oid | | not null | | plain | |
+ ltforeign_catalog | oid | | not null | | plain | |
+ ltforeign_volume | oid | | not null | | plain | |
+ lttable_type | text | C | | | extended | |
+ ltoptions | text[] | C | | | extended | |
Review Comment:
Can we use `pg_class.relam` instead of `lttable_type` and `pg_class.reloptions` instead of `ltoptions`?
When a lake table is created, `lttable_type` contains the same string as `amname` does but in upper case, and the `reloptions` column is unused.
```
postgres=# CREATE LAKE TABLE lake_test_t1 (a int, b text) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fileformat 'parquet');
CREATE LAKE TABLE
postgres=# select lttable_type, ltoptions from pg_lake_table;
lttable_type | ltoptions
--------------+----------------------
ICEBERG | {fileformat=parquet}
(1 row)
postgres=# select amname, reloptions from pg_class, pg_am where relname='lake_test_t1' and relam=pg_am.oid;
amname | reloptions
---------+------------
iceberg |
(1 row)
postgres=#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#issuecomment-5278710109
@tuhaihe as you asked, the extension work now lives in this PR instead of a separate one, so the review context stays in one place. @andr-sokolov @yjhjstz please note what changed below before looking at the diff again.
**How it was moved, and why this way.** The branch was *not* force-pushed. It now has, in order:
1. the original 26 kernel commits, untouched -- so your 64 inline comments stay anchored to the code they were written about;
2. `Revert the kernel-side lake-table DDL scaffolding` -- one commit that undoes all of them. The tree after it is identical to the merge base, verified by comparing tree object ids rather than by reading a diff;
3. the two extension commits, moved over from #1881. The resulting diff is identical to what #1881 showed -- I compared the two patches line by line; the only differences are blob hashes and one hunk offset, because the two branches had different merge bases.
Rewriting the branch would have been shorter, but it would have orphaned the review history, and it would have made going back expensive. This way the kernel scaffolding is one `git revert` away if the extension approach turns out to be wrong. The pre-revert tip is also kept as `archive/iceberg-ddl-kernel-pr1842` on my fork.
**Please re-review from scratch.** The diff is now 100% different from what was reviewed and approved earlier in this thread, so the earlier approval does not apply to what is here now. I would rather say that explicitly than have it merged on the strength of it.
**One thing that will look out of place:** the second commit changes two lines in `contrib/interconnect/sql/interconnect.sql`. That test restarts the cluster with `gpstop -raiq`; an immediate shutdown skips the shutdown checkpoint, so the next startup runs crash recovery, and while the postmaster is in `PM_RECOVERY` it rejects the connection `gpstart` makes to read the segment configuration. `gpstop -r` then exits CRITICAL, psql gives up at the `\c` that follows, the whole file is skipped, and the cluster is left with only the coordinator running -- which is why `pg_trgm` and `indexscan` failed after it in the same job. A fast shutdown checkpoints on the way out, so that state is never entered. Measured with recovery deliberately slowed: `-raiq` failed 2/2, `-rafq` passed 3/3. Fast is what the rest of the tree already uses; this file was the only `-raiq`. Happy to split it into its own PR if you would rather keep this one to the extension.
The original description is kept at the bottom of the PR body, struck through. #1881 is superseded by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop merged PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3534278770
##########
contrib/pax_storage/src/test/regress/expected/oidjoins.out:
##########
@@ -235,6 +235,10 @@ NOTICE: checking pg_foreign_data_wrapper {fdwhandler} => pg_proc {oid}
NOTICE: checking pg_foreign_data_wrapper {fdwvalidator} => pg_proc {oid}
NOTICE: checking pg_foreign_server {srvowner} => pg_authid {oid}
NOTICE: checking pg_foreign_server {srvfdw} => pg_foreign_data_wrapper {oid}
+NOTICE: checking pg_foreign_catalog {fcowner} => pg_authid {oid}
+NOTICE: checking pg_foreign_catalog {fcserver} => pg_foreign_server {oid}
+NOTICE: checking pg_foreign_volume {fvowner} => pg_authid {oid}
+NOTICE: checking pg_foreign_volume {fvserver} => pg_foreign_server {oid}
Review Comment:
This is not related to Iceberg DDL. We shouldn't change client_min_messages in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3536328190
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fcname, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%' ORDER BY 1;
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%' ORDER BY 1;
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE ICEBERG TABLE lake_test_t0 (a int) CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
+
+-- The default catalog/volume GUCs verify that the object exists
+SET iceberg_default_catalog = 'no_such_catalog'; -- fail
+SET iceberg_default_volume = 'no_such_volume'; -- fail
+
+-- Simulate a datalake provider with a heap-backed iceberg AM
+CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler;
+
+-- CREATE ICEBERG TABLE with explicit catalog and volume
+CREATE ICEBERG TABLE lake_test_t1 (a int, b text) CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fileformat 'parquet');
+SELECT c.relname, lt.lttable_type, lt.ltoptions, fc.fcname, fv.fvname
+ FROM pg_lake_table lt
+ JOIN pg_class c ON c.oid = lt.ltrelid
+ JOIN pg_foreign_catalog fc ON fc.oid = lt.ltforeign_catalog
+ JOIN pg_foreign_volume fv ON fv.oid = lt.ltforeign_volume
+ ORDER BY 1;
+SELECT a.amname FROM pg_am a JOIN pg_class c ON c.relam = a.oid WHERE c.relname = 'lake_test_t1';
+-- Lake tables are always DISTRIBUTED RANDOMLY (policytype 'p', no distkey)
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t1'::regclass;
+
+-- The (heap-backed) table is usable
+INSERT INTO lake_test_t1 VALUES (1, 'x'), (2, 'y');
+SELECT count(*) FROM lake_test_t1;
+
+-- Lake tables get a TOAST table like plain tables, so wide values work
+SELECT reltoastrelid <> 0 AS has_toast FROM pg_class WHERE relname = 'lake_test_t1';
+INSERT INTO lake_test_t1 VALUES (3, repeat('x', 500000));
+SELECT a, length(b) FROM lake_test_t1 WHERE a = 3;
+
+-- Catalog and volume are both required
+CREATE ICEBERG TABLE lake_test_t2 (a int) VOLUME lake_test_vol; -- fail, no catalog
+CREATE ICEBERG TABLE lake_test_t2 (a int) CATALOG lake_test_cat; -- fail, no volume
+
+-- ... unless the GUCs provide defaults
+SET iceberg_default_catalog = 'lake_test_cat';
+SET iceberg_default_volume = 'lake_test_vol';
+CREATE ICEBERG TABLE lake_test_t2 (a int);
+RESET iceberg_default_catalog;
+RESET iceberg_default_volume;
+
+-- A DISTRIBUTED clause is ignored with a warning
+CREATE ICEBERG TABLE lake_test_t3 (a int) CATALOG lake_test_cat VOLUME lake_test_vol DISTRIBUTED BY (a);
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t3'::regclass;
+
+-- CREATE ICEBERG TABLE only accepts the iceberg access method
+CREATE ICEBERG TABLE lake_test_bad0 (a int) CATALOG lake_test_cat VOLUME lake_test_vol USING heap; -- fail
+CREATE ICEBERG TABLE lake_test_t4 (a int) CATALOG lake_test_cat VOLUME lake_test_vol USING iceberg; -- explicit iceberg is fine
+
+-- The iceberg AM is rejected for every path other than CREATE ICEBERG TABLE
+CREATE TABLE lake_test_bad1 (a int) USING iceberg DISTRIBUTED RANDOMLY; -- fail
+CREATE TABLE lake_test_bad2 USING iceberg AS SELECT 1 AS a DISTRIBUTED RANDOMLY; -- fail
+SET default_table_access_method = iceberg;
+CREATE TABLE lake_test_bad3 (a int) DISTRIBUTED RANDOMLY; -- fail
+RESET default_table_access_method;
+CREATE TABLE lake_test_heap (a int) DISTRIBUTED RANDOMLY;
+ALTER TABLE lake_test_heap SET ACCESS METHOD iceberg; -- fail
+ALTER TABLE lake_test_t1 SET ACCESS METHOD heap; -- fail
+ALTER TABLE lake_test_t1 SET DISTRIBUTED BY (a); -- fail
+
+-- Only the owner can drop a catalog or volume
+CREATE ROLE regress_lake_user;
+SET ROLE regress_lake_user;
+DROP CATALOG lake_test_cat; -- fail, not owner
+DROP VOLUME lake_test_vol; -- fail, not owner
+RESET ROLE;
+
+-- Dependencies: the server holds the catalog/volume, which hold the tables
+DROP SERVER lake_test_srv; -- fail, catalog and volume depend on it
+DROP CATALOG lake_test_cat; -- fail, tables depend on it
+
+-- Dropping a lake table removes its pg_lake_table entry
+DROP TABLE lake_test_t1;
+SELECT count(*) FROM pg_lake_table lt JOIN pg_class c ON c.oid = lt.ltrelid
+ WHERE c.relname = 'lake_test_t1';
+
+-- DROP CATALOG ... CASCADE takes the remaining tables with it
+DROP CATALOG lake_test_cat CASCADE;
+SELECT count(*) FROM pg_lake_table lt JOIN pg_class c ON c.oid = lt.ltrelid
+ WHERE c.relname LIKE 'lake\_test%';
+
+-- DROP variants
+DROP CATALOG lake_test_cat; -- fail, already gone
+DROP CATALOG IF EXISTS lake_test_cat; -- skip with notice
+DROP VOLUME lake_test_vol;
+DROP VOLUME lake_test_vol; -- fail, already gone
Review Comment:
I suggest to add `DROP ICEBERG TABLE` as a pair for `CREATE ICEBERG TABLE` too.
```
postgres=# CREATE ICEBERG TABLE lake_test_t0 (a int) CATALOG lake_test_cat VOLUME lake_test_vol;
CREATE LAKE TABLE
postgres=# DROP ICEBERG TABLE lake_test_t0;
ERROR: syntax error at or near "ICEBERG"
LINE 1: DROP ICEBERG TABLE lake_test_t0;
^
postgres=#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3536772395
##########
src/backend/commands/laketablecmds.c:
##########
@@ -0,0 +1,483 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * laketablecmds.c
+ * lake table creation/manipulation commands
+ *
+ * IDENTIFICATION
+ * src/backend/commands/laketablecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_foreign_catalog.h"
+#include "catalog/pg_foreign_volume.h"
+#include "catalog/pg_lake_table.h"
+#include "commands/defrem.h"
+#include "commands/laketablecmds.h"
+#include "foreign/foreign.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/* GUC variables for default Iceberg catalog and volume */
+char *iceberg_default_catalog = NULL;
+char *iceberg_default_volume = NULL;
+
+/*
+ * check_iceberg_default_catalog: validate new iceberg_default_catalog GUC value
+ */
+bool
+check_iceberg_default_catalog(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid catalog_oid = get_foreign_catalog_oid(*newval, true);
+
+ if (!OidIsValid(catalog_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent catalog, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign catalog \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign catalog \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * check_iceberg_default_volume: validate new iceberg_default_volume GUC value
+ */
+bool
+check_iceberg_default_volume(char **newval, void **extra, GucSource source)
+{
+ /*
+ * If we aren't inside a transaction, or connected to a database, we
+ * cannot do the catalog accesses necessary to verify the name. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (**newval != '\0')
+ {
+ Oid volume_oid = get_foreign_volume_oid(*newval, true);
+
+ if (!OidIsValid(volume_oid))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent volume, only a NOTICE. See comments in guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("foreign volume \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Foreign volume \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * GetDefaultIcebergCatalog -- get the name of the current default Iceberg catalog
+ *
+ * Returns NULL if no default catalog is set.
+ * This function hides the iceberg_default_catalog GUC variable.
+ */
+const char *
+GetDefaultIcebergCatalog(void)
+{
+ if (iceberg_default_catalog == NULL || iceberg_default_catalog[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the catalog still exists. We don't cache this because
+ * the catalog could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_catalog_oid(iceberg_default_catalog, true)))
+ return NULL;
+
+ return iceberg_default_catalog;
+}
+
+/*
+ * GetDefaultIcebergVolume -- get the name of the current default Iceberg volume
+ *
+ * Returns NULL if no default volume is set.
+ * This function hides the iceberg_default_volume GUC variable.
+ */
+const char *
+GetDefaultIcebergVolume(void)
+{
+ if (iceberg_default_volume == NULL || iceberg_default_volume[0] == '\0')
+ return NULL;
+
+ /*
+ * Verify that the volume still exists. We don't cache this because
+ * the volume could be dropped after the GUC was set.
+ */
+ if (!OidIsValid(get_foreign_volume_oid(iceberg_default_volume, true)))
+ return NULL;
+
+ return iceberg_default_volume;
+}
+
+/*
+ * GetIcebergTableAmOid
+ *
+ * Look up the OID of the iceberg table access method, which is provided by
+ * a datalake extension rather than the kernel. Returns InvalidOid if the
+ * access method is not installed and missing_ok is true.
+ */
+Oid
+GetIcebergTableAmOid(bool missing_ok)
+{
+ return get_table_am_oid(ICEBERG_TABLE_AM_NAME, missing_ok);
+}
+
+/*
+ * RelationIsIcebergTable
+ *
+ * True iff the relation uses the iceberg table access method. Resolved by
+ * access method name so the kernel does not depend on any particular
+ * extension's OID assignments.
+ */
+bool
+RelationIsIcebergTable(Relation rel)
Review Comment:
What about adding `RELKIND_ICEBERG` constant like `RELKIND_FOREIGN_TABLE` and checking `rel->rd_rel->relkind` here to make it possible to use more that one extension with AM?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3620431169
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,154 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
+
+-- The default catalog/volume GUCs verify that the object exists
+SET iceberg_default_catalog = 'no_such_catalog'; -- fail
+SET iceberg_default_volume = 'no_such_volume'; -- fail
+
+-- Simulate a datalake provider with a heap-backed iceberg AM
+CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler;
+
+-- CREATE LAKE TABLE with explicit catalog and volume
+CREATE LAKE TABLE lake_test_t1 (a int, b text) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fileformat 'parquet');
+SELECT c.relname, lt.lttable_type, lt.ltoptions, fc.fcname, fv.fvname
+ FROM pg_lake_table lt
+ JOIN pg_class c ON c.oid = lt.ltrelid
+ JOIN pg_foreign_catalog fc ON fc.oid = lt.ltforeign_catalog
+ JOIN pg_foreign_volume fv ON fv.oid = lt.ltforeign_volume;
+SELECT a.amname FROM pg_am a JOIN pg_class c ON c.relam = a.oid WHERE c.relname = 'lake_test_t1';
+-- Lake tables are always DISTRIBUTED RANDOMLY (policytype 'p', no distkey)
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t1'::regclass;
+
+-- The (heap-backed) table is usable
+INSERT INTO lake_test_t1 VALUES (1, 'x'), (2, 'y');
+SELECT count(*) FROM lake_test_t1;
+
+-- Lake tables get a TOAST table like plain tables, so wide values work
+SELECT reltoastrelid <> 0 AS has_toast FROM pg_class WHERE relname = 'lake_test_t1';
+INSERT INTO lake_test_t1 VALUES (3, repeat('x', 500000));
+SELECT a, length(b) FROM lake_test_t1 WHERE a = 3;
+
+-- Catalog and volume are both required
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG VOLUME lake_test_vol; -- fail, no catalog
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG CATALOG lake_test_cat; -- fail, no volume
+
+-- ... unless the GUCs provide defaults
+SET iceberg_default_catalog = 'lake_test_cat';
+SET iceberg_default_volume = 'lake_test_vol';
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG;
+RESET iceberg_default_catalog;
+RESET iceberg_default_volume;
+
+-- A DISTRIBUTED clause is ignored with a warning
+CREATE LAKE TABLE lake_test_t3 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol DISTRIBUTED BY (a);
Review Comment:
I suggest to show an error message instead of warning, because `DISTRIBUTED BY` is not supported for the `CREATE LAKE TABLE` query
```
postgres=# \h CREATE LAKE TABLE
Command: CREATE LAKE TABLE
Description: define a new lake table
Syntax:
CREATE LAKE TABLE [ IF NOT EXISTS ] table_name ( [
column_name data_type [, ... ]
] )
USING format
[ CATALOG catalog_name ]
[ VOLUME volume_name ]
[ OPTIONS ( option 'value' [, ...] ) ]
URL: https://www.postgresql.org/docs/16/sql-createlaketable.html
postgres=#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3630047930
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,157 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
+
+-- The default catalog/volume GUCs verify that the object exists
+SET iceberg_default_catalog = 'no_such_catalog'; -- fail
+SET iceberg_default_volume = 'no_such_volume'; -- fail
+
+-- Simulate a datalake provider with a heap-backed iceberg AM
+CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER heap_tableam_handler;
+
+-- CREATE LAKE TABLE with explicit catalog and volume. The format is the
+-- table's access method (pg_class.relam) and its options are the relation's
+-- reloptions; pg_lake_table records only the catalog/volume binding.
+CREATE LAKE TABLE lake_test_t1 (a int, b text) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol;
+SELECT c.relname, am.amname, c.reloptions, fc.fcname, fv.fvname
+ FROM pg_lake_table lt
+ JOIN pg_class c ON c.oid = lt.ltrelid
+ JOIN pg_am am ON am.oid = c.relam
+ JOIN pg_foreign_catalog fc ON fc.oid = lt.ltforeign_catalog
+ JOIN pg_foreign_volume fv ON fv.oid = lt.ltforeign_volume;
+-- Lake tables are always DISTRIBUTED RANDOMLY (policytype 'p', no distkey)
+SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'lake_test_t1'::regclass;
+
+-- The (heap-backed) table is usable
+INSERT INTO lake_test_t1 VALUES (1, 'x'), (2, 'y');
+SELECT count(*) FROM lake_test_t1;
+
+-- OPTIONS become the relation's reloptions and are validated by the access
+-- method: a value the AM accepts is stored, an unknown one is rejected.
+CREATE LAKE TABLE lake_test_opt (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (fillfactor '70');
+SELECT reloptions FROM pg_class WHERE relname = 'lake_test_opt';
+CREATE LAKE TABLE lake_test_optbad (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol OPTIONS (bogus_opt 'x'); -- fail, AM rejects unknown option
+DROP LAKE TABLE lake_test_opt;
+
+-- Catalog and volume are both required
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG VOLUME lake_test_vol; -- fail, no catalog
+CREATE LAKE TABLE lake_test_t2 (a int) USING ICEBERG CATALOG lake_test_cat; -- fail, no volume
+
+-- ... unless the GUCs provide defaults
+SET iceberg_default_catalog = 'lake_test_cat';
+SET iceberg_default_volume = 'lake_test_vol';
Review Comment:
Catalogs and volumes can potentially be specified for any format. But the low-level implementations for various formats are distinct, so we should specify format when catalogs and volumes are created.
For example
```
CREATE FOREIGN CATALOG ... FORMAT 'iceberg' TYPE 'hive';
CREATE FOREIGN VOLUME ... FORMAT 'iceberg' SERVER ...;
```
When FORMAT is not specified we can get it from the new GUC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3637458536
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,158 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
Review Comment:
@yjhjstz How does crunchydata specify catalog? I didn't find this syntax in their presentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3535254998
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
That might be difficult. We need to support various external catalogs with numerous distinct parameter configurations. Therefore, using SQL options would be the optimal approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3535201349
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,142 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, ICEBERG TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv OPTIONS (type 'hive', uri 'thrift://localhost:9083');
Review Comment:
I recommend to make syntax without the OPTIONS keyword, if possible, because in this case validation will be implemented in grammar. So the validation will be an integral part of the syntax implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3562556374
##########
src/backend/catalog/heap.c:
##########
@@ -2322,6 +2323,10 @@ heap_drop_with_catalog(Oid relid)
*/
CheckTableForSerializableConflictIn(rel);
+ /* If this is a lake table, remove its pg_lake_table entry */
+ if (RelationIsIcebergTable(rel))
+ RemoveLakeTableEntry(relid);
Review Comment:
if you can provide a draft version, that's good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "leborchuk (via GitHub)" <gi...@apache.org>.
leborchuk commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3594550857
##########
src/include/tcop/cmdtaglist.h:
##########
@@ -176,10 +179,13 @@ PG_CMDTAG(CMDTAG_DROP_DOMAIN, "DROP DOMAIN", true, false, false)
PG_CMDTAG(CMDTAG_DROP_DYNAMIC_TABLE, "DROP DYNAMIC TABLE", true, false, false)
PG_CMDTAG(CMDTAG_DROP_EVENT_TRIGGER, "DROP EVENT TRIGGER", false, false, false)
PG_CMDTAG(CMDTAG_DROP_EXTENSION, "DROP EXTENSION", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_FOREIGN_CATALOG, "DROP FOREIGN CATALOG", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FOREIGN_DATA_WRAPPER, "DROP FOREIGN DATA WRAPPER", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FOREIGN_TABLE, "DROP FOREIGN TABLE", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_FOREIGN_VOLUME, "DROP FOREIGN VOLUME", true, false, false)
PG_CMDTAG(CMDTAG_DROP_FUNCTION, "DROP FUNCTION", true, false, false)
PG_CMDTAG(CMDTAG_DROP_INDEX, "DROP INDEX", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_LAKE_TABLE, "DROP LAKE TABLE", true, false, false)
Review Comment:
Hi! I vote for `CREATE LAKE TABLE ... USING ICEBERG` - add space for the other formats )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3662263724
##########
src/test/regress/sql/lake_table.sql:
##########
@@ -0,0 +1,158 @@
+--
+-- Test lake table DDL: FOREIGN CATALOG, FOREIGN VOLUME, LAKE TABLE
+--
+
+-- Display the lake table catalogs
+\d+ pg_foreign_catalog
+\d+ pg_foreign_volume
+\d+ pg_lake_table
+
+-- Setup: foreign servers for the catalogs and volumes to hang off
+CREATE FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv FOREIGN DATA WRAPPER lake_test_fdw;
+CREATE SERVER lake_test_srv2 FOREIGN DATA WRAPPER lake_test_fdw;
+
+-- CREATE FOREIGN CATALOG: TYPE is a required first-class property
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive' OPTIONS (uri 'thrift://localhost:9083');
+CREATE FOREIGN CATALOG lake_test_notype SERVER lake_test_srv; -- fail, TYPE is required
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv TYPE 'hive'; -- skip with notice
+-- catalog names are global: the same name on another server is still a duplicate
+CREATE FOREIGN CATALOG lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- fail, duplicate
+CREATE FOREIGN CATALOG IF NOT EXISTS lake_test_cat SERVER lake_test_srv2 TYPE 'hive'; -- skip with notice
+CREATE FOREIGN CATALOG lake_test_bad SERVER no_such_server TYPE 'hive'; -- fail, no server
+SELECT fcname, fctype, fcoptions FROM pg_foreign_catalog WHERE fcname LIKE 'lake\_test%';
+
+-- CREATE FOREIGN VOLUME
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv OPTIONS (base_path 's3://bucket/prefix');
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv; -- skip with notice
+-- volume names are global: the same name on another server is still a duplicate
+CREATE FOREIGN VOLUME lake_test_vol SERVER lake_test_srv2; -- fail, duplicate
+CREATE FOREIGN VOLUME IF NOT EXISTS lake_test_vol SERVER lake_test_srv2; -- skip with notice
+CREATE FOREIGN VOLUME lake_test_bad SERVER no_such_server; -- fail, no server
+SELECT fvname, fvoptions FROM pg_foreign_volume WHERE fvname LIKE 'lake\_test%';
+
+-- Object descriptions
+SELECT pg_catalog.pg_describe_object('pg_foreign_catalog'::regclass, oid, 0)
+ FROM pg_foreign_catalog WHERE fcname = 'lake_test_cat';
+SELECT pg_catalog.pg_describe_object('pg_foreign_volume'::regclass, oid, 0)
+ FROM pg_foreign_volume WHERE fvname = 'lake_test_vol';
+
+-- Catalog and volume rows are dispatched to all segments
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_catalog') WHERE fcname = 'lake_test_cat';
+SELECT count(DISTINCT gp_segment_id) > 1 AS on_all_segments
+ FROM gp_dist_random('pg_foreign_volume') WHERE fvname = 'lake_test_vol';
+
+-- Without a provider extension there is no iceberg table AM
+CREATE LAKE TABLE lake_test_t0 (a int) USING ICEBERG CATALOG lake_test_cat VOLUME lake_test_vol; -- fail with hint
Review Comment:
@yjhjstz Agreed — this looks technically feasible, so I sketched what the fully-extension shape could look like.
Since Cloudberry's table AM API already supports per-AM reloptions validation (`TableAmRoutine->amoptions`, unlike vanilla PG16), `WITH (catalog ..., volume ...)` can be validated by the iceberg AM itself with no kernel changes. The catalog/volume objects could reuse the existing FDW infrastructure (`SERVER` + `USER MAPPING`), which gives ACLs, pg_dump support, QD/QE dispatch and dependency tracking for free:
```sql
CREATE EXTENSION pg_iceberg; -- provides the table AM, two FDWs, and hooks
CREATE SERVER hive_cat FOREIGN DATA WRAPPER iceberg_catalog
OPTIONS (type 'hive', uri 'thrift://metastore:9083');
CREATE USER MAPPING FOR alice SERVER hive_cat
OPTIONS (user '...', password '...');
CREATE SERVER s3_vol FOREIGN DATA WRAPPER iceberg_volume
OPTIONS (base_path 's3://bucket/prefix');
CREATE TABLE t (a int, b text) USING iceberg
WITH (catalog 'hive_cat', volume 's3_vol');
DROP TABLE t; -- plain DROP TABLE applies
```
A few things the extension would need to own, instead of kernel guards:
- registering table metadata and `pg_depend` entries (table → server) via `object_access_hook`, so `DROP SERVER` stays protected and CASCADE works;
- intercepting paths that don't make sense for iceberg (CTAS, partition children, `ALTER TABLE ... SET ACCESS METHOD`) via `ProcessUtility_hook`, and forcing random distribution;
- the library would need to be in `shared_preload_libraries` so the hooks are always installed.
This is essentially the pg_lake architecture, and `USER MAPPING` would also give a cleaner credentials story than embedding keys in options. If we go this way, the kernel scaffolding in this PR (the grammar, the three catalogs, the guards) would no longer be needed — the work would move into the datalake extension instead. WDYT?
@leborchuk @andr-sokolov
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3840218562
##########
contrib/interconnect/sql/interconnect.sql:
##########
@@ -83,5 +89,5 @@ DROP EXTENSION interconnect;
-- start_ignore
\! gpconfig -r shared_preload_libraries
-\! gpstop -raiq
+\! gpstop -rafq
Review Comment:
Fair objection — split out into **#1919** and reverted here in a720cf69798, so this PR no longer touches `contrib/interconnect` at all.
It was here because that race is what made this PR's `ic-contrib` job fail, and the failure is destructive rather than local: `gpstop -raiq` skips the shutdown checkpoint, the next startup runs crash recovery, and while the postmaster is in `PM_RECOVERY` it rejects the connection `gpstart` makes to read the segment configuration. `gpstop -r` then exits CRITICAL, `psql` gives up at the following `\c`, the whole file is skipped, and the cluster is left with only the coordinator running — which is why `pg_trgm` and `indexscan` also failed after it in the same job. #1919 has the full chain and the before/after measurements.
Reverted rather than dropped by rewriting the branch, so this thread stays anchored to the change it is about.
**One consequence to expect:** `ic-contrib` here may go red again until #1919 lands. That failure is in `contrib/interconnect` and has nothing to do with this module — the library is not in `shared_preload_libraries` for that job, so it is never loaded there.
##########
contrib/datalake_fdw/src/iceberg_volume_fdw/iceberg_volume_fdw.c:
##########
@@ -0,0 +1,157 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * iceberg_volume_fdw.c
+ * Option validator for Iceberg volume foreign servers.
+ *
+ * IDENTIFICATION
+ * contrib/datalake_fdw/src/iceberg_volume_fdw/iceberg_volume_fdw.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/reloptions.h"
+#include "am_iceberg/pg_iceberg_options.h"
+#include "catalog/pg_foreign_data_wrapper.h"
+#include "catalog/pg_foreign_server.h"
+#include "catalog/pg_user_mapping.h"
+#include "commands/defrem.h"
+#include "common/dl_option_util.h"
+#include "fmgr.h"
+#include "iceberg_volume_fdw/iceberg_volume_option.h"
+
+PG_FUNCTION_INFO_V1(iceberg_volume_fdw_validator);
+
+static bool is_volume_server_option(const char *name);
+static bool is_volume_user_mapping_option(const char *name);
+
+static bool
+is_volume_server_option(const char *name)
+{
+ return strcmp(name, DATALAKE_ICEBERG_VOLUME_BASE_PATH) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_ENDPOINT) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_REGION) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_PATH_STYLE_ACCESS) == 0;
+}
+
+/*
+ * Every credential here is optional, so that ambient storage credentials -- an
+ * instance profile, a ticket cache -- remain a valid deployment choice.
+ */
+static bool
+is_volume_user_mapping_option(const char *name)
+{
+ return strcmp(name, DATALAKE_ICEBERG_VOLUME_USERNAME) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_AWS_ACCESS_KEY_ID) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_AWS_SECRET_ACCESS_KEY) == 0 ||
+ strcmp(name, DATALAKE_ICEBERG_VOLUME_AWS_SESSION_TOKEN) == 0;
+}
+
+Datum
+iceberg_volume_fdw_validator(PG_FUNCTION_ARGS)
+{
+ List *options = untransformRelOptions(PG_GETARG_DATUM(0));
+ Oid catalog = PG_GETARG_OID(1);
+ ListCell *lc;
+ IcebergVolumeServerOptions server_options;
+ IcebergForeignVolumeOptions volume_options;
Review Comment:
Applied in 2adeb73da50, and your reasoning is the better part of it. The declarations sit about sixty lines above where the memsets were, with the option-name validation loop and two early returns in between, so anything added to that stretch that read the structs would have read them uninitialized. Zeroing at the declaration removes that possibility.
Being precise about what it buys: **no existing test can tell the two forms apart.** Both parse functions assign every field they own — `get_string_option()` returns NULL when an option is absent, and `get_bool_option_ex()` writes both the value and its "was set" flag — so the memsets were belt-and-braces rather than load bearing, and nothing between the declarations and the parse calls touched the structs. So this is future safety, not a fix, and I would rather say that than imply it closed a live bug.
`= {0}` is also the form the rest of the tree uses (over a hundred occurrences in `src/backend/commands` and `src/backend/catalog` alone), so it improves consistency too.
The module has one other `memset`, in `pg_iceberg_options.c`, which clears an out parameter reached through a pointer — there is no declaration there to attach an initializer to, so I left it alone rather than have it look overlooked.
Verified: zero compiler warnings (the module builds with `-Werror=uninitialized` and without `-Wextra`, so the initializer neither trips nor silences anything) and `installcheck` 3/3 on a three-segment cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3891442962
##########
.github/workflows/build-cloudberry.yml:
##########
@@ -326,6 +326,10 @@ jobs:
"contrib/pg_buffercache:installcheck",
"contrib/sslinfo:installcheck"]
},
+ {"test":"ic-datalake-fdw",
+ "make_configs":["contrib/datalake_fdw:installcheck"],
+ "shared_preload_libraries":"datalake_fdw"
Review Comment:
I think it looks good now. Same with the PAX way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Add Iceberg lake-table DDL: CREATE ICEBERG TABLE, FOREIGN CATALOG, FOREIGN VOLUME (kernel scaffolding) [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#issuecomment-5248374467
> @andr-sokolov @yjhjstz I open this draft PR about lake table ext (create table xxx using iceberg xxx), please give some suggest and review it. #1881
Hi, is there any way to introduce your new commits to this place, where we have a context for review? FYI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on code in PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#discussion_r3828714300
##########
contrib/interconnect/sql/interconnect.sql:
##########
@@ -83,5 +89,5 @@ DROP EXTENSION interconnect;
-- start_ignore
\! gpconfig -r shared_preload_libraries
-\! gpstop -raiq
+\! gpstop -rafq
Review Comment:
Why are the changes in another extension tests relate to datalake_fdw?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) [cloudberry]
Posted by "andr-sokolov (via GitHub)" <gi...@apache.org>.
andr-sokolov commented on PR #1842:
URL: https://github.com/apache/cloudberry/pull/1842#issuecomment-5367480954
What about moving the extension to gpcontrib to minimize conflicts with PostgreSQL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org