You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/02/10 22:28:33 UTC
[I] `COLLATION` Support [arrow-datafusion]
alamb opened a new issue, #9192:
URL: https://github.com/apache/arrow-datafusion/issues/9192
### Is your feature request related to a problem or challenge?
"Collation" generically means how to compare and sort string values.
Databases like Postgres allow you to change the default collation order to control this more carefully to match whatever the user wants rather than what the standard sort order means
Someone asked about this on discord: https://discord.com/channels/885562378132000778/1166447479609376850/1205554368292855868
Here are some details on how this works in Postgres: https://www.postgresql.org/docs/current/collation.html
### Describe the solution you'd like
Someone to design and implement `COLLATION`
This probably looks like a `SessionConfig` setting to control collation at the session level and possibly some way to define it as part of the table definition
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `COLLATION` Support [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9192:
URL: https://github.com/apache/datafusion/issues/9192#issuecomment-3438381222
I started a discussion on an API to support customizing behavior like this on
- https://github.com/apache/datafusion/issues/18223
I think COLLATION would be a natural additional property for types (as it controlls comparisons and sorting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `COLLATION` Support [datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #9192:
URL: https://github.com/apache/datafusion/issues/9192#issuecomment-3201004552
A major feature of Spark 4.0.0 is support for string collation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `COLLATION` Support [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9192:
URL: https://github.com/apache/arrow-datafusion/issues/9192#issuecomment-1938501790
I believe `collation` is an important feature for postgres compatibility, but it is not as widely used in other databases.
I agree having something that was optional would be ideal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `COLLATION` Support [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9192:
URL: https://github.com/apache/arrow-datafusion/issues/9192#issuecomment-1937294290
cc @gruuya as I believe you mentioned you might also be interested in this feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `COLLATION` Support [arrow-datafusion]
Posted by "gruuya (via GitHub)" <gi...@apache.org>.
gruuya commented on issue #9192:
URL: https://github.com/apache/arrow-datafusion/issues/9192#issuecomment-1938574306
Yeah, for reference DuckDB also provides optional general collations, using an extension for the ICU project: https://duckdb.org/docs/sql/expressions/collations.html#icu-collations
There's a corresponding Rust crate as well: https://docs.rs/icu/latest/icu/collator/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `COLLATION` Support [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9192:
URL: https://github.com/apache/datafusion/issues/9192#issuecomment-3201876755
I think it would be great to sketch out what supporting collation would look like.
As I recall, it is fairly invasive, as it potentially changes all string comparisons (both for ORDER BY as well as for `=`, `<`, etc type functions).
The first major question, is where the support would live (upstream in arrow-rs or in DataFusion) 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `COLLATION` Support [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #9192:
URL: https://github.com/apache/datafusion/issues/9192#issuecomment-3201051237
> There's a corresponding Rust crate as well: https://docs.rs/icu/latest/icu/collator/index.html
There's also Google's [rust_icu](https://github.com/google/rust_icu).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `COLLATION` Support [arrow-datafusion]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #9192:
URL: https://github.com/apache/arrow-datafusion/issues/9192#issuecomment-1937927060
I think we would need to identify a mature Rust library for supporting collations, as I suspect this is not something we wish to implement ourselves
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `COLLATION` Support [datafusion]
Posted by "mjgarton (via GitHub)" <gi...@apache.org>.
mjgarton commented on issue #9192:
URL: https://github.com/apache/datafusion/issues/9192#issuecomment-3438345439
It seems collation support is being discussed for the arrow c++ implementation. I don't know if this implies arrow-rs is a more suitable place than datafusion?
https://github.com/apache/arrow/issues/27875
Datafusion would need some degree of support in any case I suppose, even if it delegates the actual ordering decisions to arrow-rs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `COLLATION` Support [datafusion]
Posted by "mjgarton (via GitHub)" <gi...@apache.org>.
mjgarton commented on issue #9192:
URL: https://github.com/apache/datafusion/issues/9192#issuecomment-4192098653
I notice that support for extension types has been merged. I can't quite see how to get from that to collation support. Is more work needed in extension types first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org