You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by "urlyy (via GitHub)" <gi...@apache.org> on 2024/07/27 18:42:02 UTC
[PR] feat(rust): support simd convert utf16 to utf8 [fury]
urlyy opened a new pull request, #1775:
URL: https://github.com/apache/fury/pull/1775
## What does this PR do?
For the conversion from UTF-16 to UTF-8, a SIMD method based on AVX/SSE/NEON instruction sets was added on the basis of #1730 , and benchmarks were written.
referencing
- https://github.com/simdutf/simdutf/blob/master/src/westmere/sse_convert_utf16_to_utf8.cpp
- https://github.com/simdutf/simdutf/blob/master/src/haswell/avx2_convert_utf16_to_utf8.cpp
- https://github.com/simdutf/simdutf/blob/5c1a86887010cd2b4d648049c4d73de81a026341/src/arm64/arm_convert_utf16_to_utf8.cpp
- https://github.com/simdutf/simdutf/blob/master/src/tables/utf16_to_utf8_tables.h
## Related issues
- #1547
- #1730
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary.
-->
- [x] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
In local machine benchmark , simd approach seems to be Three to Eight times faster than normal approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(rust): support simd convert utf16 to utf8 [fury]
Posted by "urlyy (via GitHub)" <gi...@apache.org>.
urlyy closed pull request #1775: feat(rust): support simd convert utf16 to utf8
URL: https://github.com/apache/fury/pull/1775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(rust): support simd convert utf16 to utf8 [fury]
Posted by "urlyy (via GitHub)" <gi...@apache.org>.
urlyy commented on PR #1775:
URL: https://github.com/apache/fury/pull/1775#issuecomment-2254241005
I learned the technique used in https://github.com/simdutf/simdutf , and implemented in fury.
Here are some questions:
1. Sorry for I can only test on x86 platform. I rely on GitHub CI to debug the logic of the AArch64 version.😢 I can't give the aarch64 benchmark data.
2. There may be a lot of comments. If they are not desired , I will remove them.
3. I have seen another SIMD code in rust , `string_util.rs`, should I merge it with my code like what has done in `cpp`?
4. As it is helpful to use the precomputing table, I have put it in my code , but takes approximately 1600 lines. Do you have some advice on this?
5. For better readability, I have used nested mod and divided the various steps into different functions. If this approach is not satisfactory, I am willing to make adjustments。
6. To be honest, I am uncertain whether introducing such an extensive amount of code is justified merely for the purpose of converting UTF to UTF-8. However, I have observed optimizations, particularly for the conversion of 1 UTF-16 to 1, 2, or 3 UTF-8 bytes. This has added complexity。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org