You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "CTTY (via GitHub)" <gi...@apache.org> on 2026/01/20 19:55:48 UTC
[I] Support CREATE TABLE PARTITIONED BY syntax with DataFusion [iceberg-rust]
CTTY opened a new issue, #2050:
URL: https://github.com/apache/iceberg-rust/issues/2050
### Is your feature request related to a problem or challenge?
We want to support registering partitioned tables into datafusion using `CREATE TABLE PARTITIONED BY`
### Describe the solution you'd like
_No response_
### Willingness to contribute
I can contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support CREATE TABLE PARTITIONED BY syntax with DataFusion [iceberg-rust]
Posted by "charlesdong1991 (via GitHub)" <gi...@apache.org>.
charlesdong1991 commented on issue #2050:
URL: https://github.com/apache/iceberg-rust/issues/2050#issuecomment-3905041944
Hello, I understand this issue has a dependency on https://github.com/apache/iceberg-rust/pull/2022, once this PR is merged, can I give it shot at this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support CREATE EXTERNAL TABLE PARTITIONED BY syntax with DataFusion [iceberg-rust]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on issue #2050:
URL: https://github.com/apache/iceberg-rust/issues/2050#issuecomment-4609688377
Thank you @CTTY
I took a shot on this https://github.com/apache/iceberg-rust/pull/2575.
Two open questions:
1. **Non-identity transforms are blocked for now.** DataFusion's `PARTITIONED BY` only accepts plain column names — it can't express transforms like `bucket(16, id)` or `days(ts)` (Spark's native grammar can). So I return a error when a non-identity-partitioned table is given a `PARTITIONED BY` clause Proper support likely needs an upstream proposal — does that path sound right? (left a `TODO` in code).
2. **Escape hatch:** such tables can still be registered read-only by just omitting `PARTITIONED BY` — the spec always comes from the Iceberg metadata.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support CREATE TABLE PARTITIONED BY syntax with DataFusion [iceberg-rust]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on issue #2050:
URL: https://github.com/apache/iceberg-rust/issues/2050#issuecomment-4595802411
Question here:
Do we want to support `CREATE TABLE PARTITIONED BY` or `CREATE EXTERNAL TABLE PARTITIONED BY`？
These two readings of the title imply very different work:
- **`CREATE TABLE foo (...) PARTITIONED BY (...)`** — what Spark/Trino/Flink users expect. But today DataFusion's planner rejects `partition_by` on plain `CREATE TABLE`, and `CreateMemoryTable` + `SchemaProvider::register_table` have no slot for partition info. So this requires upstream apache/datafusion changes first.
- **`CREATE EXTERNAL TABLE foo STORED AS ICEBERG LOCATION '...' PARTITIONED BY (...)`** — reachable today via `IcebergTableProviderFactory` (the path PR #2022 took). But this only loads existing tables, can't create new ones, and `CREATE EXTERNAL TABLE` is essentially a Hive-legacy construct that no other modern Iceberg integration uses.
cc @CTTY @liurenjie1024 @charlesdong1991 for thoughts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support CREATE EXTERNAL TABLE PARTITIONED BY syntax with DataFusion [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on issue #2050:
URL: https://github.com/apache/iceberg-rust/issues/2050#issuecomment-4607830283
Hi @huan233usc , thanks for the question! we actually want to support `CREATE EXTERNAL TABLE PARTITIONED BY` due to the limitation of datafusion as of now. I've updated the title and information in this issue to help address the ambiguity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org