You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/05 21:59:19 UTC

[GitHub] [arrow-datafusion] Dandandan opened a new issue, #3049: Simplify / speed up implementation of character_length to unicode points

Dandandan opened a new issue, #3049:
URL: https://github.com/apache/arrow-datafusion/issues/3049

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   It looks like postgresql and spark have a different, (probably much faster) implementation of calculating the string length, which doesn't depend on calculating the grapheme clusters, but on utf8 code points.
   
   E.g. see this example:
   
   `select length('ä')`
   PostgreSQL | Spark | DataFusion
   | Postgres | Spark | DataFusion |
   |----------|-------|------------|
   | 2        | 2     | 1          |
   
   **Describe the solution you'd like**
   Probably `.chars().count()` is a faster solution.
   
   **Describe alternatives you've considered**
   Accepting our own implementation being slower but "superior" to other solutions.
   
   **Additional context**
   This came up as being quite slow when profiling this benchmark: https://github.com/DataPsycho/data-pipelines-in-rust/tree/main/amazon_review_pipeline 
   We're using the `grapheme` option in other places as well, maybe here we could use the faster 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove closed issue #3049: Simplify / speed up implementation of character_length to unicode points

Posted by GitBox <gi...@apache.org>.

andygrove closed issue #3049: Simplify / speed up implementation of character_length to unicode points
URL: https://github.com/apache/arrow-datafusion/issues/3049


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove commented on issue #3049: Simplify / speed up implementation of character_length to unicode points

Posted by GitBox <gi...@apache.org>.

andygrove commented on issue #3049:
URL: https://github.com/apache/arrow-datafusion/issues/3049#issuecomment-1207105642

   This is something that we could potentially add a configuration option for as well, defaulting to the fast version but allowing the user to select a slower and more accurate version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] Dandandan commented on issue #3049: Simplify / speed up implementation of character_length to unicode points

Posted by GitBox <gi...@apache.org>.

Dandandan commented on issue #3049:
URL: https://github.com/apache/arrow-datafusion/issues/3049#issuecomment-1207205155

   > This is something that we could potentially add a configuration option for as well, defaulting to the fast version but allowing the user to select a slower and more accurate version.
   
   In general, I think it would be better to choose one version of the algorithm rather than have the result depend on the configuration.
   
   As we aim to be mostly compatible with PostgreSQL, it makes sense to me to choose counting the code points instead of grapheme clusters.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org