You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "JerAguilon (via GitHub)" <gi...@apache.org> on 2025/12/22 21:08:28 UTC
[I] Hidden partitioning clarification: do `identity` partition columns need to be in data files? [iceberg]
JerAguilon opened a new issue, #14914:
URL: https://github.com/apache/iceberg/issues/14914
### Query engine
Trino and ClickHouse, though this question is more about the spec itself
### Question
I've been reading [this document](https://apache.github.io/iceberg/docs/1.4.3/partitioning/#icebergs-hidden-partitioning) and I am unsure what to conclude for this particular use case, and I am seeing divergent behavior across some table engines.
Suppose I have a table with a schema of:
```
sale_id: BIGINT
price: BIGINT
date: DATE
```
And it is partitioned on `date`.
If we guarantee that each Parquet data file contains data for one and only one `date`, are we allowed to just have `sale_id, price` in the physical files, while marking `date` values in the manifest layer?
Given that there's a "separation between physical and logical," this seems to be analogous to a transformation like `bucket`, except with some constant value. But the spec doesn't explicitly discuss this.
I have a table with this format. On Trino's implementation, it seems to handle this gracefully, substituting partition values as you'd except. However, some other engines (https://github.com/ClickHouse/ClickHouse/issues/92732) instead return `NULL` values.
It'd be very helpful to explicitly clarify if `identity` partitions can also be hidden, in the same way that `bucket` transformations are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hidden partitioning clarification: do `identity` partition columns need to be in data files? [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #14914:
URL: https://github.com/apache/iceberg/issues/14914#issuecomment-3688000556
Column projection was added to pyiceberg in https://github.com/apache/iceberg-python/pull/1443
I couldn't find the exact place Trino implemented this. But from your experiment, it does seem like Trino projects the `identity` transform partition value correctly. @ebyhr would you know where the column projection rules are implemented in Trino? I see you added the `add_files` procedure in https://github.com/trinodb/trino/pull/22751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hidden partitioning clarification: do `identity` partition columns need to be in data files? [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #14914:
URL: https://github.com/apache/iceberg/issues/14914#issuecomment-3687913124
Actually @JerAguilon @kevinjqliu brought up the column projection rules to me https://iceberg.apache.org/spec/#column-projection and I did forget about the Hive table migration case, so perhaps Clickhouse needs to implement the column projection logic to substitute on read for parquet files written and that were just added to the iceberg table state. That's probably what Trino does as well.
But Iceberg writers have to be materializing the partition values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hidden partitioning clarification: do `identity` partition columns need to be in data files? [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #14914: Hidden partitioning clarification: do `identity` partition columns need to be in data files?
URL: https://github.com/apache/iceberg/issues/14914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hidden partitioning clarification: do `identity` partition columns need to be in data files? [iceberg]
Posted by "JerAguilon (via GitHub)" <gi...@apache.org>.
JerAguilon commented on issue #14914:
URL: https://github.com/apache/iceberg/issues/14914#issuecomment-3688199636
Thanks all for the clarifications. Seems like we need to fix our own parquet writers internally, but +1 on cleaning up the language in the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hidden partitioning clarification: do `identity` partition columns need to be in data files? [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #14914:
URL: https://github.com/apache/iceberg/issues/14914#issuecomment-3687858095
Iceberg does require explicit materialization of columns in the data file, even those that are used in partitioning schemes. Ultimately partitioning is a transformation or derivation on a column (or columns) but the materialization in data files is helpful in case metadata gets corrupted.
@JerAguilon Here's the current spec language https://iceberg.apache.org/spec/#writing-data-files
```
All columns must be written to data files even if they introduce redundancy with metadata stored in manifest files (e.g. columns with identity partition transforms). Writing all columns provides a backup in case of corruption or bugs in the metadata layer.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hidden partitioning clarification: do `identity` partition columns need to be in data files? [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #14914:
URL: https://github.com/apache/iceberg/issues/14914#issuecomment-3688217179
Created https://github.com/apache/iceberg/issues/14925 to track the change to clarify the spec. Thanks for bringing this up!
Closing this issue, feel free to reopen as necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org