You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "bhanreddy1973 (via GitHub)" <gi...@apache.org> on 2025/12/31 16:41:46 UTC
[PR] Optimize SparkHex function using lookup table [datafusion]
bhanreddy1973 opened a new pull request, #19585:
URL: https://github.com/apache/datafusion/pull/19585
## Which issue does this PR close?
Closes #15986
## Rationale for this change
The current `hex` function implementation uses `write!(&mut s, "{b:02x}")` for each byte, which has significant overhead from format string parsing.
## What changes are included in this PR?
Replaced the slow format-based hex encoding with pre-computed lookup tables (HEX_UPPER/HEX_LOWER). Each byte value maps directly to its two-character hex representation via simple array indexing.
Benefits:
- Eliminates format string parsing overhead
- Uses a single pre-sized allocation
- Simple array indexing instead of function calls per byte
## How are these changes tested?
All existing tests pass:
- test_hex_int64
- test_spark_hex_int64
- test_dictionary_hex_utf8
- test_dictionary_hex_int64
- test_dictionary_hex_binary
## Are these changes safe?
Yes - the lookup tables contain only valid ASCII hex characters, so the conversion is safe. The behavior is identical to before, just faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Optimize SparkHex function using lookup table [datafusion]
Posted by "bhanreddy1973 (via GitHub)" <gi...@apache.org>.
bhanreddy1973 closed pull request #19585: Optimize SparkHex function using lookup table
URL: https://github.com/apache/datafusion/pull/19585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Optimize SparkHex function using lookup table [datafusion]
Posted by "bhanreddy1973 (via GitHub)" <gi...@apache.org>.
bhanreddy1973 commented on PR #19585:
URL: https://github.com/apache/datafusion/pull/19585#issuecomment-3704609036
Closing this PR - looks like this optimization was already implemented in a different way in the main branch. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org