You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rluvaton (via GitHub)" <gi...@apache.org> on 2024/12/15 11:36:55 UTC
[I] Allow changing globally the default list field name [arrow-rs]
rluvaton opened a new issue, #6881:
URL: https://github.com/apache/arrow-rs/issues/6881
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The arrow implementation of Java uses `$data$` rather than `item` like in here, which cause schema mismatch if not making sure the field name are matching.
This is problematic as sometimes other places are the one that create the `list` (like DataFusion)
**Describe the solution you'd like**
I want to be able to globally configure the name of the list field
**Describe alternatives you've considered**
passing the wanted field name everywhere, although it's not always possible when the list is created somewhere else (like DataFusion)
**Additional context**
I'm able to submit a PR to add this if wanted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow changing globally the default list field name [arrow-rs]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #6881:
URL: https://github.com/apache/arrow-rs/issues/6881#issuecomment-2576375333
FYI, I've opened a PR in Java Arrow to allow changing the default field name as it's not possible to create a list with specific field name
https://github.com/apache/arrow-java/pull/488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow changing globally the default list field name [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6881:
URL: https://github.com/apache/arrow-rs/issues/6881#issuecomment-2543863993
I wonder if the core problem is that the logic that compares schemas treats Lists with different field names as semantically different, even when it seems as if a different field name doesn't actually make the type semantically different.
Maybe we could look into changing the comparison logic to ignore the list field name 🤔 (same for LargeList, FixedSizedList, Union, etc)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow changing globally the default list field name [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6881:
URL: https://github.com/apache/arrow-rs/issues/6881#issuecomment-2543879212
> Maybe we could look into changing the comparison logic to ignore the list field name 🤔 (same for LargeList, FixedSizedList, Union, etc)?
I think there is a broader and somewhat separate question around logical schema equality, that concerns not only this but metadata and possibly even logical type equality (is StringView the same as StringArray).
IMO we shouldn't be opinionated here and instead leave this to systems integrating arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow changing globally the default list field name [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6881:
URL: https://github.com/apache/arrow-rs/issues/6881#issuecomment-2545940067
> Well, but then it seems like the library is making assertions about what Schema equality means.
I'll agree to disagree on whether providing equality on a datastructure is making a judgement on how engines should choose to enforce or not enforce logical schema equivalence.
> Making it easier (and clearly documented) to compare schemas with different, common ideas of equality
Perhaps we can roll this issue into https://github.com/apache/arrow-rs/issues/6735?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow changing globally the default list field name [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6881:
URL: https://github.com/apache/arrow-rs/issues/6881#issuecomment-2543844234
Did you have a concrete proposal on what this might look like?
> although it's not always possible when the list is created somewhere else
I can't help feeling this makes your proposal impractical, external systems are free to name this whatever they feel like.
> which cause schema mismatch if not making sure the field name are matching
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow changing globally the default list field name [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6881:
URL: https://github.com/apache/arrow-rs/issues/6881#issuecomment-2544131502
I think providing `Eq` that implements an exact match is not only consistent with the trait, but also important for things like tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow changing globally the default list field name [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6881:
URL: https://github.com/apache/arrow-rs/issues/6881#issuecomment-2544068224
> I think there is a broader and somewhat separate question around logical schema equality, that concerns not only this but metadata and possibly even logical type equality (is StringView the same as StringArray).
>
> IMO we shouldn't be opinionated here and instead leave this to systems integrating arrow.
That seems like a reasonable approach. However, since `DataType` implements `PartialEq` I think that by necessity enforces some idea of what schema equality means
If we had to do it again, maybe we would support different types of Eq for `DataType` somehow (or not support the Eq trait and force downstream users to decide how to compare different Fields)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow changing globally the default list field name [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6881:
URL: https://github.com/apache/arrow-rs/issues/6881#issuecomment-2545916929
> I think providing `Eq` that implements an exact match is not only consistent with the trait, but also important for things like tests.
Well, but then it seems like the library is making assertions about what `Schema` equality means.
But yes, I do see your point that practically speaking removing `PartialEq` / `Eq` implementations would likely cause more pain than it was worth
Making it easier (and clearly documented) to compare schemas with different, common ideas of equality (e.g. ignore list field names) I think would add significant value by avoiding seemingly unnecessary schema mismatch issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow changing globally the default list field name [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #6881: Allow changing globally the default list field name
URL: https://github.com/apache/arrow-rs/issues/6881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org