You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2025/08/16 23:19:51 UTC
[I] Partition field and schema field name conflicts not validated on schema evolution [iceberg]
kevinjqliu opened a new issue, #13833:
URL: https://github.com/apache/iceberg/issues/13833
### Feature Request / Improvement
### Description
Iceberg validates conflicts between schema field names and partition field names when adding a partition field. For example, in [`checkAndAddPartitionName`](https://github.com/apache/iceberg/blob/4dbc7f578eee7ceb9def35ebfa1a4cc236fb598f/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L392-L416), conflicts are rejected unless itβs an identity/void transform sourced from the same field.
However, the same validation is missing during schema evolution. If a user later adds or renames a schema column to the same name as a partition field, the conflict is not detected. This creates ambiguity and may cause incorrect behavior at query or planning time.
We should add a similar check when adding a new column to the table schema ([`SchemaUpdate`](https://github.com/apache/iceberg/blob/4dbc7f578eee7ceb9def35ebfa1a4cc236fb598f/core/src/main/java/org/apache/iceberg/SchemaUpdate.java#L48))
Found this edge case while we were implementing the same check in pyiceberg, https://github.com/apache/iceberg-python/pull/2305
### Query engine
None
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [x] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Partition field and schema field name conflicts not validated on schema evolution [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #13833:
URL: https://github.com/apache/iceberg/issues/13833#issuecomment-3194000851
Open #13834 to add a new test for this scenario
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Partition field and schema field name conflicts not validated on schema evolution [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #13833:
URL: https://github.com/apache/iceberg/issues/13833#issuecomment-3195550196
This is not covered today:
<img width="1845" height="1533" alt="Image" src="https://github.com/user-attachments/assets/736909eb-a59f-4a3b-99b8-70ab9a7efd45" />
That test should be failing π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] bug: validate schema and partition field name conflicts during updates [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13833:
URL: https://github.com/apache/iceberg/issues/13833#issuecomment-3994397479
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Partition field and schema field name conflicts not validated on schema evolution [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #13833:
URL: https://github.com/apache/iceberg/issues/13833#issuecomment-3208186526
thats a good one. Here's the call stack for this bug:
`updateSpec()` [calls `PartitionSpec.Builder.add()`](https://github.com/apache/iceberg/blob/d5e3a56b3150b0e97adf52673526104d01e3aaa6/core/src/main/java/org/apache/iceberg/BaseUpdatePartitionSpec.java#L313), which [passes `name` and `sourceId` to `checkAndAddPartitionName`](https://github.com/apache/iceberg/blob/d5e3a56b3150b0e97adf52673526104d01e3aaa6/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L615-L620)
Because [`sourceColumnId` is provided, `checkAndAddPartitionName`](https://github.com/apache/iceberg/blob/d5e3a56b3150b0e97adf52673526104d01e3aaa6/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L404) goes to the if statement which handles identity transform.
This is a great illustration of why refactoring `checkAndAddPartitionName` to be more explicit in #13835 helps prevent bugs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] bug: validate schema and partition field name conflicts during updates [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13833:
URL: https://github.com/apache/iceberg/issues/13833#issuecomment-3911155615
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] fix: validate naming conflicts between schema field names and partition field names during schema update and partition spec update [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #13833:
URL: https://github.com/apache/iceberg/issues/13833#issuecomment-3208203420
the refactor in #13835 actually caught this bug π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Partition field and schema field name conflicts not validated on schema evolution [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #13833:
URL: https://github.com/apache/iceberg/issues/13833#issuecomment-3193997935
this actually isnt an issue because table update will rebuild partition spec which will trigger the name collision check
https://github.com/apache/iceberg/blob/0b599bc37e39b3893e443393136c7fddc8391d6d/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1098-L1100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] bug: validate schema and partition field name conflicts during updates [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #13833: bug: validate schema and partition field name conflicts during updates
URL: https://github.com/apache/iceberg/issues/13833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org