You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "shinzoxD (via GitHub)" <gi...@apache.org> on 2026/08/19 17:20:58 UTC
[PR] perf: return Utf8View from Spark hex with inline short path [datafusion]
shinzoxD opened a new pull request, #24494:
URL: https://github.com/apache/datafusion/pull/24494
## Which issue does this PR close?
- Closes #15986
## Rationale for this change
Review of #15947 asked Spark `hex` to return `Utf8View` instead of `Utf8`, with a fast path for encodings that fit in the StringView inline prefix (12 bytes). The lookup table (#21836) and criterion benches (#19738) already landed. This finishes the remaining item on #15986.
Short hex results are common (small integers, short strings/binaries). Building a `Utf8` `StringArray` for those rows pays for offset buffers and a data buffer even when Arrow can store the digits in the view itself. Returning `Utf8View` avoids that for encodings of at most 12 bytes and still stores longer encodings out-of-line.
## What changes are included in this PR?
- `SparkHex::return_type` is now `Utf8View`, including dictionary values (`Dictionary(K, Utf8View)`).
- Encodings of at most 12 hex digits are inlined via `make_view` and never touch the StringView data buffer.
- Longer encodings are written once into that buffer and referenced by offset.
- Input null buffers are still reused (`ptr_eq` on the inner validity buffer).
- `BinaryView` inputs are handled (the signature already accepted them).
- Criterion bench adds `hex_utf8_inline` for the short-string path.
## Are these changes tested?
Yes.
- Unit tests in `datafusion/spark/src/function/math/hex.rs` cover Utf8View / LargeUtf8 / BinaryView inputs, null reuse, the 12- vs 13-digit inline boundary, and `return_type`.
- SQL logic tests in `datafusion/sqllogictest/test_files/spark/math/hex.slt` assert values plus `arrow_typeof(hex(...)) = Utf8View` (and `Dictionary(Int32, Utf8View)` for dictionary input).
- `cargo test -p datafusion-spark --lib hex`: 19 passed.
- `cargo clippy -p datafusion-spark --all-targets -- -D warnings`: clean.
## Are there any user-facing changes?
`hex` now returns `Utf8View` instead of `Utf8`. Encoded values are unchanged. Callers that match on the physical type (for example `ScalarValue::Utf8`) need to accept `Utf8View`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: return Utf8View from Spark hex with inline short path [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #24494: perf: return Utf8View from Spark hex with inline short path
URL: https://github.com/apache/datafusion/pull/24494
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: return Utf8View from Spark hex with inline short path [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24494:
URL: https://github.com/apache/datafusion/pull/24494#issuecomment-6378318052
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: return Utf8View from Spark hex with inline short path [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #24494:
URL: https://github.com/apache/datafusion/pull/24494#issuecomment-5350196933
for anyone looking at this PR please see my comment here:
- https://github.com/apache/datafusion/pull/24407#issuecomment-5350186733
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: return Utf8View from Spark hex with inline short path [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24494:
URL: https://github.com/apache/datafusion/pull/24494#issuecomment-6506004536
Thanks for the contribution! I am closing this under DataFusion's AI Dump policy.
https://datafusion.apache.org/contributor-guide/ai-policy.html#what-is-an-ai-dump-and-why-it-is-not-helpful
This has the hallmarks of AI code that was not reviewed by the author: a 500-line rewrite with hand-built unsafe array construction, a performance change with no benchmark numbers, and no engagement since the PR was opened.
If you would like to continue, please open a focused PR in your own words that you have reviewed and understand, and we are happy to help with review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org