You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2026/07/20 10:28:26 UTC
[PR] perf: reuse input null buffer and add no-nulls fast path in Spark `hex` [datafusion]
jackylee-ch opened a new pull request, #23721:
URL: https://github.com/apache/datafusion/pull/23721
## Which issue does this PR close?
- Closes #23674.
## Rationale for this change
Follow-on from review feedback on #23473. `hex_encode_bytes` in `datafusion/spark/src/function/math/hex.rs` rebuilt the output `NullBuffer` row by row via `NullBufferBuilder`, even though hex encoding maps rows 1:1 (a null in maps to a null out), so the output nulls are always identical to the input's.
This PR applies both optimizations suggested in the issue:
1. **Reuse the input `NullBuffer`.** The input array's null buffer is now cloned (a cheap `Arc`-based clone) and attached to the output array directly, removing the per-row `append_null`/`append_non_null` bookkeeping.
2. **Special-case the no-nulls path.** When the input has no null buffer, a tight loop encodes every value without the per-row `Option` check.
## What changes are included in this PR?
- `hex_encode_bytes` now takes the input array's `Option<NullBuffer>` and reuses it for the output array instead of rebuilding it with `NullBufferBuilder`; all call sites (top-level arrays and dictionary values) pass `array.nulls().cloned()`.
- The encoding loop is split into a no-nulls fast path and a has-nulls path.
Benchmark results from the existing `datafusion/spark/benches/hex.rs` (null_density = 0.1) against `main` on an M-series MacBook:
```
hex_utf8/size=1024 time: [17.466 µs 17.507 µs 17.552 µs] change: −4.98%
hex_utf8/size=4096 time: [73.478 µs 73.592 µs 73.727 µs] change: −7.16%
hex_utf8/size=8192 time: [147.63 µs 148.19 µs 148.82 µs] change: −6.97%
hex_binary/size=1024 time: [17.583 µs 17.613 µs 17.650 µs] change: −5.48%
hex_binary/size=4096 time: [74.234 µs 74.326 µs 74.435 µs] change: −7.38%
hex_binary/size=8192 time: [148.91 µs 149.04 µs 149.19 µs] change: −7.19%
```
## Are these changes tested?
Yes. Existing unit tests cover the string/binary/dictionary paths, and a new test `test_spark_hex_utf8_with_nulls` exercises the has-nulls path with interleaved nulls and empty strings to verify the reused null buffer stays aligned with the encoded values.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse input null buffer and add no-nulls fast path in Spark `hex` [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23721:
URL: https://github.com/apache/datafusion/pull/23721#issuecomment-5021735321
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/23721?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `50.64935%` with `38 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.69%. Comparing base ([`3b1ce16`](https://app.codecov.io/gh/apache/datafusion/commit/3b1ce16be6beb3377c59ec36fd9f8ae62998c4c0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4467fd1`](https://app.codecov.io/gh/apache/datafusion/commit/4467fd152eb82786e58a99c86976f52d74a2837a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/23721?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/spark/src/function/math/hex.rs](https://app.codecov.io/gh/apache/datafusion/pull/23721?src=pr&el=tree&filepath=datafusion%2Fspark%2Fsrc%2Ffunction%2Fmath%2Fhex.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zcGFyay9zcmMvZnVuY3Rpb24vbWF0aC9oZXgucnM=) | 50.64% | [34 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23721?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #23721 +/- ##
==========================================
- Coverage 80.69% 80.69% -0.01%
==========================================
Files 1088 1088
Lines 367766 367826 +60
Branches 367766 367826 +60
==========================================
+ Hits 296781 296803 +22
- Misses 53276 53311 +35
- Partials 17709 17712 +3
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/23721?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse input null buffer and add no-nulls fast path in Spark `hex` [datafusion]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch closed pull request #23721: perf: reuse input null buffer and add no-nulls fast path in Spark `hex`
URL: https://github.com/apache/datafusion/pull/23721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse input null buffer and add no-nulls fast path in Spark `hex` [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #23721:
URL: https://github.com/apache/datafusion/pull/23721#issuecomment-5023210447
Hi @jackylee-ch we already have an open pr for this https://github.com/apache/datafusion/pull/23688
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse input null buffer and add no-nulls fast path in Spark `hex` [datafusion]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #23721:
URL: https://github.com/apache/datafusion/pull/23721#issuecomment-5029490787
> Hi @jackylee-ch we already have an open pr for this #23688
Thanks, sorry for miss that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org