You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "brancz (via GitHub)" <gi...@apache.org> on 2024/09/04 13:42:29 UTC
[I] Relax `dict_id` equality in field merging [arrow-rs]
brancz opened a new issue, #6356:
URL: https://github.com/apache/arrow-rs/issues/6356
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Trying to merge two schemas that are compatible, other than having fields whose `dict_id`s are different.
**Describe the solution you'd like**
Relax `dict_id` handling during merging of fields, such that if they are different the `dict_id` is unset, and only if they are equal are they kept.
**Describe alternatives you've considered**
Before opening this, I tried implementing my own merge function, but would have to change it quite significantly compared to the existing code, since I can't access internal fields. I could still make this work, but wanted to first check whether the project would be open to relaxing this, since this feels more aligned with how merging works otherwise (eg. making a field nullable).
**Additional context**
All of this is under the assumption that I understand `dict_id` correctly, which is that it's primarily used for IPC's ability to duplicate dicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Relax `dict_id` equality in field merging [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6356:
URL: https://github.com/apache/arrow-rs/issues/6356#issuecomment-2332132400
Unfortunately that isn't how the IPC implementation treats them, in particular IPC files use dictionary IDs as unique identifiers to immutable dictionaries. It is all a bit confused, and it is peculiar that they're part of the schema definition at all, but that is what #5981 concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Relax `dict_id` equality in field merging [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6356:
URL: https://github.com/apache/arrow-rs/issues/6356#issuecomment-2331741770
https://github.com/apache/arrow-rs/issues/5981 is possibly related.
As it stands I am not sure we can ignore the dict ID when merging, as unlike with nulls, two arrays with different dictionary IDs can't be "merged", as there is no covering relationship between two different dict IDs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Relax `dict_id` equality in field merging [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #6356: Relax `dict_id` equality in field merging
URL: https://github.com/apache/arrow-rs/issues/6356
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Relax `dict_id` equality in field merging [arrow-rs]
Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #6356:
URL: https://github.com/apache/arrow-rs/issues/6356#issuecomment-2332102012
Isn't that just an optimization though? I understand that if two dict IDs are identical one can just append the index arrays to each other safely, and if not then the dicts needs to be unified and indexes added transposed (using the terminology that the Go arrow package uses not sure it's universal it's just the library I have the most experience with).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Relax `dict_id` equality in field merging [arrow-rs]
Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #6356:
URL: https://github.com/apache/arrow-rs/issues/6356#issuecomment-2333487144
If I understand this right then, it should be possible to remove `dict_id` from `Field` if we solve IPC dictionary ID handling some other way. And consequently `dict_id` would no longer be part of equality of `Field`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org