You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Omega359 (via GitHub)" <gi...@apache.org> on 2024/01/29 15:46:02 UTC
[I] strpos doesn't account for unicode grapheme characters [arrow-datafusion]
Omega359 opened a new issue, #9052:
URL: https://github.com/apache/arrow-datafusion/issues/9052
### Describe the bug
❯ docker run -it -v /tmp:/data datafusion-cli
DataFusion CLI v35.0.0
❯ SELECT strpos('since ̯ͅI can ̤̗do ̲̬̝th̟̟̱i̹s̹̱', 'can');
+-------------------------------------------------+
| strpos(Utf8("since ̯ͅI can ̤̗do ̲̬̝th̟̟̱i̹s̹̱"),Utf8("can")) |
+-------------------------------------------------+
| 0 |
+-------------------------------------------------+
1 row in set. Query took 0.067 seconds.
Should be 9.
### To Reproduce
_No response_
### Expected behavior
position should account for unicode grapheme characters, not just unicode scalar values.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] strpos doesn't account for unicode grapheme characters [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 closed issue #9052: strpos doesn't account for unicode grapheme characters
URL: https://github.com/apache/arrow-datafusion/issues/9052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] strpos doesn't account for unicode grapheme characters [arrow-datafusion]
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on issue #9052:
URL: https://github.com/apache/arrow-datafusion/issues/9052#issuecomment-1923328369
Is there any other database that implements `strpos` in terms of grapheme clusters? When I did research on string functions across databases, nearly all of them implemented it in term of scalar values. I even argued to use bytes instead, for performance, and since a position would most likely be used as input to another string function like `substring`. But I got overruled, compatibility is a pretty good argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] strpos doesn't account for unicode grapheme characters [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #9052:
URL: https://github.com/apache/arrow-datafusion/issues/9052#issuecomment-1914994922
Note that this 'bug' may not be something that can be easily fixed given the nature of unicode grapheme's. For example, I came across this insightful response for a similar postgres issue - https://stackoverflow.com/a/64889162/1695525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] strpos doesn't account for unicode grapheme characters [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 closed issue #9052: strpos doesn't account for unicode grapheme characters
URL: https://github.com/apache/arrow-datafusion/issues/9052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] strpos doesn't account for unicode grapheme characters [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #9052:
URL: https://github.com/apache/arrow-datafusion/issues/9052#issuecomment-1924251854
Probably not supported anywhere tbh. It was something I noticed when browsing the code however in looking into it for postgres and spark it seems that it's likely not something that can be easily supported. Thus I'm going to close this as essentially 'won't fix'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org