You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "syun64 (via GitHub)" <gi...@apache.org> on 2024/07/16 01:39:48 UTC
[I] NameMapping flattens the names and causes `a.b` field to collide with child `b` field of field `a` [iceberg-python]
syun64 opened a new issue, #935:
URL: https://github.com/apache/iceberg-python/issues/935
### Apache Iceberg version
None
### Please describe the bug 🐞
According to the Iceberg documentation on Column Projection:
```
A name may contain . but this refers to a literal name, not a nested field. For example, a.b refers to a field named a.b, not child field b of field a.
```
The current implementation of NameMapping flattens the name by joining the parent child relationships with a `.`. This causes name collisions issues with fields that should not collide with each other.
For example, this flat map causes `a.b` field to collide with child `b` field of field `a`.
We should update `_field_by_name()` and `find()` methods of NameMapping to use a tree structure instead of a flat dict, and traverse the tree in order to retrieve `MappedField` of the provided name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] NameMapping flattens the names and causes `a.b` field to collide with child `b` field of field `a` [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #935:
URL: https://github.com/apache/iceberg-python/issues/935#issuecomment-2231927898
Hi @Fokko - I wanted to make note of this Rest Catalog Open API Spec [PR](https://github.com/apache/iceberg/pull/9695/files#diff-02549ca620d020dc9ead80088cc14e311e12a69651fa8d394cd41a4308debb2eR3922-R3933), where the community may be weighing the pros and cons of flattening the nested field names in our APIs:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] NameMapping flattens the names and causes `a.b` field to collide with child `b` field of field `a` [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #935:
URL: https://github.com/apache/iceberg-python/issues/935#issuecomment-2231716869
Yes of course!
Just a note that's hopefully helpful: while working on covering more cases for #921 , I realized this may require a bit more work than I originally thought. We currently rely on a flat name mapping in many places throughout the repository, including when we aggregate stats from the parquet files:
https://github.com/apache/iceberg-python/blob/0f2e19e49eb8b859cfcd7f89cd182461a61f15a7/pyiceberg/io/pyarrow.py#L2027-L2031
So I think we will need to build a tree representation of the Name to ID mapping for a given pyarrow schema as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] NameMapping flattens the names and causes `a.b` field to collide with child `b` field of field `a` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #935:
URL: https://github.com/apache/iceberg-python/issues/935#issuecomment-2231655618
@syun64 can I pick this one up? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] NameMapping flattens the names and causes `a.b` field to collide with child `b` field of field `a` [iceberg-python]
Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy closed issue #935: NameMapping flattens the names and causes `a.b` field to collide with child `b` field of field `a`
URL: https://github.com/apache/iceberg-python/issues/935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org