You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/21 16:35:19 UTC
[PR] perf: optimize to_hex in datafusion-functions [datafusion]
andygrove opened a new pull request, #23766:
URL: https://github.com/apache/datafusion/pull/23766
> This PR was created by an LLM as a draft PR. I will mark it as ready for review after human review.
## Which issue does this PR close?
N/A — autonomous exploratory PR.
## Rationale for this change
Replaced to_hex's one-nibble-per-iteration digit loop with a 256-entry byte lookup table (two nibbles/iteration), halving loop iterations for long integers with bit-identical output.
## What changes are included in this PR?
Replaced to_hex's one-nibble-per-iteration digit loop with a 256-entry byte lookup table (two nibbles/iteration), halving loop iterations for long integers with bit-identical output.
## Are these changes tested?
Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs `main`).
Benchmark (criterion):
- i64_random: 8.38% faster (base 47310ns -> cand 43345ns)
- i64_large_values: 4.001% faster (base 43269ns -> cand 41538ns)
- i32_random: -1.079% faster (base 45437ns -> cand 45927ns)
- to_hex_scalar_i32: 4.416% faster (base 76ns -> cand 73ns)
- i64_random: 8.699% faster (base 23722ns -> cand 21658ns)
- i64_large_values: 9.138% faster (base 21641ns -> cand 19664ns)
- i32_random: -0.082% faster (base 22366ns -> cand 22384ns)
- i64_random: 8.271% faster (base 5887ns -> cand 5400ns)
- i64_large_values: 8.548% faster (base 5447ns -> cand 4981ns)
- i32_random: 0.754% faster (base 5578ns -> cand 5536ns)
- to_hex_scalar_i64: -1.667% faster (base 72ns -> cand 73ns)
Full criterion output:
```text
to_hex/scalar_i32 time: [72.464 ns 73.353 ns 74.313 ns]
change: [−5.6616% −4.4159% −3.2105%] (p = 0.00 < 0.05)
Performance has improved.
to_hex/scalar_i64 time: [72.762 ns 73.347 ns 73.979 ns]
change: [+0.7597% +1.6671% +2.6207%] (p = 0.00 < 0.05)
Change within noise threshold.
to_hex size=1024/i32_random
time: [5.5219 µs 5.5362 µs 5.5519 µs]
change: [−1.1759% −0.7539% −0.3663%] (p = 0.00 < 0.05)
Change within noise threshold.
to_hex size=1024/i64_random
time: [5.3749 µs 5.4004 µs 5.4284 µs]
change: [−8.8588% −8.2708% −7.6124%] (p = 0.00 < 0.05)
Performance has improved.
to_hex size=1024/i64_large_values
time: [4.9640 µs 4.9815 µs 5.0036 µs]
change: [−9.0021% −8.5475% −7.9514%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
to_hex size=4096/i32_random
time: [22.341 µs 22.385 µs 22.445 µs]
change: [−0.3183% +0.0822% +0.4876%] (p = 0.71 > 0.05)
No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
to_hex size=4096/i64_random
time: [21.626 µs 21.659 µs 21.690 µs]
change: [−9.1614% −8.6986% −8.2679%] (p = 0.00 < 0.05)
Performance has improved.
to_hex size=4096/i64_large_values
time: [19.600 µs 19.664 µs 19.728 µs]
change: [−9.6011% −9.1377% −8.5799%] (p = 0.00 < 0.05)
Performance has improved.
to_hex size=8192/i32_random
time: [45.542 µs 45.927 µs 46.341 µs]
change: [−0.1168% +1.0792% +2.3694%] (p = 0.13 > 0.05)
No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
to_hex size=8192/i64_random
time: [43.248 µs 43.345 µs 43.454 µs]
change: [−9.0189% −8.3801% −7.8411%] (p = 0.00 < 0.05)
Performance has improved.
to_hex size=8192/i64_large_values
time: [40.705 µs 41.539 µs 42.339 µs]
change: [−6.1190% −4.0005% −2.0172%] (p = 0.00 < 0.05)
Performance has improved.
```
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#discussion_r3636090442
##########
datafusion/common/src/utils/hex.rs:
##########
@@ -0,0 +1,357 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Hex encoding of bytes and integers.
+//!
+//! [`encode_bytes`] and [`encode_bytes_into`] encode a byte slice into an
+//! owned `String` or an appended `Vec<u8>`, respectively; [`encode_bytes_to_slice`]
+//! writes into a caller-provided, pre-sized buffer. [`encode_u64`] encodes an
+//! integer, trimming leading zeros. All four take a [`HexCase`] to choose
+//! between lowercase and uppercase digits.
+
+use crate::Result;
+use crate::error::_internal_err;
+
+/// Case of the emitted hex digits.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum HexCase {
+ /// Digits `0123456789abcdef`.
+ Lower,
+ /// Digits `0123456789ABCDEF`.
+ Upper,
+}
+
+const LOWER_DIGITS: &[u8; 16] = b"0123456789abcdef";
+const UPPER_DIGITS: &[u8; 16] = b"0123456789ABCDEF";
+
+/// Maps a full byte to its two hex digits, so encoding advances a whole byte
+/// per iteration instead of a nibble.
+const LOOKUP_LOWER: [[u8; 2]; 256] = build_lookup(LOWER_DIGITS);
+const LOOKUP_UPPER: [[u8; 2]; 256] = build_lookup(UPPER_DIGITS);
+
+const fn build_lookup(digits: &[u8; 16]) -> [[u8; 2]; 256] {
+ let mut table = [[0u8; 2]; 256];
+ let mut i = 0;
+ while i < 256 {
+ table[i][0] = digits[i >> 4];
+ table[i][1] = digits[i & 0xF];
+ i += 1;
+ }
+ table
+}
+
+impl HexCase {
+ #[inline]
+ const fn lookup(self) -> &'static [[u8; 2]; 256] {
+ match self {
+ HexCase::Lower => &LOOKUP_LOWER,
+ HexCase::Upper => &LOOKUP_UPPER,
+ }
+ }
+
+ #[inline]
+ const fn digits(self) -> &'static [u8; 16] {
+ match self {
+ HexCase::Lower => LOWER_DIGITS,
+ HexCase::Upper => UPPER_DIGITS,
+ }
+ }
+}
+
+/// Appends the hex encoding of `bytes` to `out`.
+///
+/// Allocates only through `out`'s own growth. Callers that must bound or guard
+/// that growth should reserve capacity in `out` before calling.
+#[inline(always)]
+pub fn encode_bytes_into(bytes: &[u8], case: HexCase, out: &mut Vec<u8>) {
+ let lookup = case.lookup();
+ for &byte in bytes {
+ out.extend_from_slice(&lookup[byte as usize]);
+ }
+}
+
+/// Writes the hex encoding of `bytes` into `out`.
+///
+/// This is for callers that already own a pre-sized buffer (for example a
+/// slice of a larger, pre-allocated output array) and want to write directly
+/// into it rather than appending to a `Vec`.
+///
+/// Returns an internal error if `out` is not exactly `2 * bytes.len()` bytes
+/// long, rather than silently encoding only part of the input.
Review Comment:
```suggestion
/// Returns an internal error if `out` is not exactly `2 * bytes.len()` bytes
/// long, without filling any of the `out` buffer.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5041349942
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23766#issuecomment-5041334503)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5041334503-1218-wk5xp 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing auto-opt/to_hex-datafusion-20260721-094849 (5e7e2112731a9e0810f4555a6fbf5b1b2636ca8e) to eef1017 (merge-base) [diff](https://github.com/apache/datafusion/compare/eef101769d650b2b06ac2bcf298f71113a047c9d..5e7e2112731a9e0810f4555a6fbf5b1b2636ca8e) using: sha2
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5041344901
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23766#issuecomment-5041334503)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5041334503-1219-9h47g 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing auto-opt/to_hex-datafusion-20260721-094849 (5e7e2112731a9e0810f4555a6fbf5b1b2636ca8e) to eef1017 (merge-base) [diff](https://github.com/apache/datafusion/compare/eef101769d650b2b06ac2bcf298f71113a047c9d..5e7e2112731a9e0810f4555a6fbf5b1b2636ca8e) using: crypto
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#discussion_r3627046948
##########
datafusion/common/src/utils/hex.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Hex encoding of bytes and integers.
+//!
+//! [`encode_bytes`] and [`encode_bytes_into`] encode a byte slice into an
+//! owned `String` or an appended `Vec<u8>`, respectively; [`encode_bytes_to_slice`]
+//! writes into a caller-provided, pre-sized buffer. [`encode_u64`] encodes an
+//! integer, trimming leading zeros. All four take a [`HexCase`] to choose
+//! between lowercase and uppercase digits.
+
+/// Case of the emitted hex digits.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum HexCase {
+ /// Digits `0123456789abcdef`.
+ Lower,
+ /// Digits `0123456789ABCDEF`.
+ Upper,
+}
+
+const LOWER_DIGITS: &[u8; 16] = b"0123456789abcdef";
+const UPPER_DIGITS: &[u8; 16] = b"0123456789ABCDEF";
+
+/// Maps a full byte to its two hex digits, so encoding advances a whole byte
+/// per iteration instead of a nibble.
+const LOOKUP_LOWER: [[u8; 2]; 256] = build_lookup(LOWER_DIGITS);
+const LOOKUP_UPPER: [[u8; 2]; 256] = build_lookup(UPPER_DIGITS);
+
+const fn build_lookup(digits: &[u8; 16]) -> [[u8; 2]; 256] {
+ let mut table = [[0u8; 2]; 256];
+ let mut i = 0;
+ while i < 256 {
+ table[i][0] = digits[i >> 4];
+ table[i][1] = digits[i & 0xF];
+ i += 1;
+ }
+ table
+}
+
+impl HexCase {
+ #[inline]
+ const fn lookup(self) -> &'static [[u8; 2]; 256] {
+ match self {
+ HexCase::Lower => &LOOKUP_LOWER,
+ HexCase::Upper => &LOOKUP_UPPER,
+ }
+ }
+
+ #[inline]
+ const fn digits(self) -> &'static [u8; 16] {
+ match self {
+ HexCase::Lower => LOWER_DIGITS,
+ HexCase::Upper => UPPER_DIGITS,
+ }
+ }
+}
+
+/// Appends the hex encoding of `bytes` to `out`.
+///
+/// Allocates only through `out`'s own growth. Callers that must bound or guard
+/// that growth should reserve capacity in `out` before calling.
+#[inline(always)]
+pub fn encode_bytes_into(bytes: &[u8], case: HexCase, out: &mut Vec<u8>) {
+ let lookup = case.lookup();
+ for &byte in bytes {
+ out.extend_from_slice(&lookup[byte as usize]);
+ }
+}
+
+/// Writes the hex encoding of `bytes` into `out`.
+///
+/// `out` must be exactly `2 * bytes.len()` bytes long. This is for callers
+/// that already own a pre-sized buffer (for example a slice of a larger,
+/// pre-allocated output array) and want to write directly into it rather
+/// than appending to a `Vec`.
+///
+/// The length requirement is only checked in debug builds, via
Review Comment:
Why not just check it and return an internal error? It seems better than a silent failure in release builds
##########
datafusion/functions/src/crypto/md5.rs:
##########
@@ -122,13 +107,15 @@ fn md5(args: &[ColumnarValue]) -> Result<ColumnarValue> {
Ok(match value {
ColumnarValue::Array(array) => {
let binary_array = as_binary_array(&array)?;
- let string_array: StringViewArray =
- binary_array.iter().map(|opt| opt.map(hex_encode)).collect();
+ let string_array: StringViewArray = binary_array
+ .iter()
+ .map(|opt| opt.map(|b| encode_bytes(b, HexCase::Lower)))
Review Comment:
This is unfortunate (as it makes a bunch of allocated strings and then copies them again into theStringViewArray -- using a temporary buffer ald a stringViewBuilder could avoid the intermediate allocation
This can be done as a follow on PR (or never) -- the old version dod this too
##########
datafusion/functions/src/string/to_hex.rs:
##########
@@ -79,101 +76,50 @@ where
#[inline]
fn to_hex_scalar<T: ToHex>(value: T) -> String {
let mut hex_buffer = [0u8; 16];
- let hex_len = value.write_hex_to_buffer(&mut hex_buffer);
- // SAFETY: hex_buffer is ASCII hex digits
- unsafe { std::str::from_utf8_unchecked(&hex_buffer[16 - hex_len..]).to_string() }
+ let hex = value.write_hex(&mut hex_buffer);
+ // SAFETY: hex holds only ASCII hex digits.
+ unsafe { std::str::from_utf8_unchecked(hex).to_string() }
}
/// Trait for converting integer types to hexadecimal in a buffer
Review Comment:
I wonder if we should move this code into the common library crate as well
##########
datafusion/common/src/utils/hex.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Hex encoding of bytes and integers.
+//!
+//! [`encode_bytes`] and [`encode_bytes_into`] encode a byte slice into an
+//! owned `String` or an appended `Vec<u8>`, respectively; [`encode_bytes_to_slice`]
+//! writes into a caller-provided, pre-sized buffer. [`encode_u64`] encodes an
+//! integer, trimming leading zeros. All four take a [`HexCase`] to choose
+//! between lowercase and uppercase digits.
+
+/// Case of the emitted hex digits.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum HexCase {
+ /// Digits `0123456789abcdef`.
+ Lower,
+ /// Digits `0123456789ABCDEF`.
+ Upper,
+}
+
+const LOWER_DIGITS: &[u8; 16] = b"0123456789abcdef";
+const UPPER_DIGITS: &[u8; 16] = b"0123456789ABCDEF";
+
+/// Maps a full byte to its two hex digits, so encoding advances a whole byte
+/// per iteration instead of a nibble.
+const LOOKUP_LOWER: [[u8; 2]; 256] = build_lookup(LOWER_DIGITS);
+const LOOKUP_UPPER: [[u8; 2]; 256] = build_lookup(UPPER_DIGITS);
+
+const fn build_lookup(digits: &[u8; 16]) -> [[u8; 2]; 256] {
+ let mut table = [[0u8; 2]; 256];
+ let mut i = 0;
+ while i < 256 {
+ table[i][0] = digits[i >> 4];
+ table[i][1] = digits[i & 0xF];
+ i += 1;
+ }
+ table
+}
+
+impl HexCase {
+ #[inline]
+ const fn lookup(self) -> &'static [[u8; 2]; 256] {
+ match self {
+ HexCase::Lower => &LOOKUP_LOWER,
+ HexCase::Upper => &LOOKUP_UPPER,
+ }
+ }
+
+ #[inline]
+ const fn digits(self) -> &'static [u8; 16] {
+ match self {
+ HexCase::Lower => LOWER_DIGITS,
+ HexCase::Upper => UPPER_DIGITS,
+ }
+ }
+}
+
+/// Appends the hex encoding of `bytes` to `out`.
+///
+/// Allocates only through `out`'s own growth. Callers that must bound or guard
+/// that growth should reserve capacity in `out` before calling.
+#[inline(always)]
+pub fn encode_bytes_into(bytes: &[u8], case: HexCase, out: &mut Vec<u8>) {
+ let lookup = case.lookup();
+ for &byte in bytes {
+ out.extend_from_slice(&lookup[byte as usize]);
+ }
+}
+
+/// Writes the hex encoding of `bytes` into `out`.
+///
+/// `out` must be exactly `2 * bytes.len()` bytes long. This is for callers
+/// that already own a pre-sized buffer (for example a slice of a larger,
+/// pre-allocated output array) and want to write directly into it rather
+/// than appending to a `Vec`.
+///
+/// The length requirement is only checked in debug builds, via
+/// `debug_assert_eq!`. In release builds a mismatched `out` length is not
+/// an error: only `min(bytes.len(), out.len() / 2)` bytes are encoded, so
+/// an `out` that is too short silently drops the remaining input, and an
+/// `out` that is too long is left with unwritten, stale bytes at the end.
+/// Callers are responsible for sizing `out` correctly.
+///
+/// # Example
+///
+/// ```
+/// use datafusion_common::utils::hex::{HexCase, encode_bytes_to_slice};
+///
+/// let mut out = [0u8; 8];
+/// encode_bytes_to_slice(&[0xde, 0xad, 0xbe, 0xef], HexCase::Lower, &mut out);
+/// assert_eq!(&out, b"deadbeef");
+/// ```
+#[inline(always)]
+pub fn encode_bytes_to_slice(bytes: &[u8], case: HexCase, out: &mut [u8]) {
+ debug_assert_eq!(out.len(), bytes.len() * 2);
+ let lookup = case.lookup();
+ for (&b, chunk) in bytes.iter().zip(out.chunks_exact_mut(2)) {
+ chunk.copy_from_slice(&lookup[b as usize]);
Review Comment:
this is probably going to be quite fast
##########
datafusion/common/src/utils/hex.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Hex encoding of bytes and integers.
+//!
+//! [`encode_bytes`] and [`encode_bytes_into`] encode a byte slice into an
+//! owned `String` or an appended `Vec<u8>`, respectively; [`encode_bytes_to_slice`]
+//! writes into a caller-provided, pre-sized buffer. [`encode_u64`] encodes an
+//! integer, trimming leading zeros. All four take a [`HexCase`] to choose
+//! between lowercase and uppercase digits.
+
+/// Case of the emitted hex digits.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum HexCase {
+ /// Digits `0123456789abcdef`.
+ Lower,
+ /// Digits `0123456789ABCDEF`.
+ Upper,
+}
+
+const LOWER_DIGITS: &[u8; 16] = b"0123456789abcdef";
+const UPPER_DIGITS: &[u8; 16] = b"0123456789ABCDEF";
+
+/// Maps a full byte to its two hex digits, so encoding advances a whole byte
+/// per iteration instead of a nibble.
+const LOOKUP_LOWER: [[u8; 2]; 256] = build_lookup(LOWER_DIGITS);
+const LOOKUP_UPPER: [[u8; 2]; 256] = build_lookup(UPPER_DIGITS);
+
+const fn build_lookup(digits: &[u8; 16]) -> [[u8; 2]; 256] {
+ let mut table = [[0u8; 2]; 256];
+ let mut i = 0;
+ while i < 256 {
+ table[i][0] = digits[i >> 4];
+ table[i][1] = digits[i & 0xF];
+ i += 1;
+ }
+ table
+}
+
+impl HexCase {
+ #[inline]
+ const fn lookup(self) -> &'static [[u8; 2]; 256] {
+ match self {
+ HexCase::Lower => &LOOKUP_LOWER,
+ HexCase::Upper => &LOOKUP_UPPER,
+ }
+ }
+
+ #[inline]
+ const fn digits(self) -> &'static [u8; 16] {
+ match self {
+ HexCase::Lower => LOWER_DIGITS,
+ HexCase::Upper => UPPER_DIGITS,
+ }
+ }
+}
+
+/// Appends the hex encoding of `bytes` to `out`.
+///
+/// Allocates only through `out`'s own growth. Callers that must bound or guard
+/// that growth should reserve capacity in `out` before calling.
+#[inline(always)]
+pub fn encode_bytes_into(bytes: &[u8], case: HexCase, out: &mut Vec<u8>) {
+ let lookup = case.lookup();
+ for &byte in bytes {
+ out.extend_from_slice(&lookup[byte as usize]);
+ }
+}
+
+/// Writes the hex encoding of `bytes` into `out`.
+///
+/// `out` must be exactly `2 * bytes.len()` bytes long. This is for callers
+/// that already own a pre-sized buffer (for example a slice of a larger,
+/// pre-allocated output array) and want to write directly into it rather
+/// than appending to a `Vec`.
+///
+/// The length requirement is only checked in debug builds, via
+/// `debug_assert_eq!`. In release builds a mismatched `out` length is not
+/// an error: only `min(bytes.len(), out.len() / 2)` bytes are encoded, so
+/// an `out` that is too short silently drops the remaining input, and an
+/// `out` that is too long is left with unwritten, stale bytes at the end.
+/// Callers are responsible for sizing `out` correctly.
+///
+/// # Example
+///
+/// ```
+/// use datafusion_common::utils::hex::{HexCase, encode_bytes_to_slice};
+///
+/// let mut out = [0u8; 8];
+/// encode_bytes_to_slice(&[0xde, 0xad, 0xbe, 0xef], HexCase::Lower, &mut out);
+/// assert_eq!(&out, b"deadbeef");
+/// ```
+#[inline(always)]
+pub fn encode_bytes_to_slice(bytes: &[u8], case: HexCase, out: &mut [u8]) {
+ debug_assert_eq!(out.len(), bytes.len() * 2);
+ let lookup = case.lookup();
+ for (&b, chunk) in bytes.iter().zip(out.chunks_exact_mut(2)) {
+ chunk.copy_from_slice(&lookup[b as usize]);
+ }
+}
+
+/// Returns the hex encoding of `bytes` as an owned `String`.
+///
+/// # Example
+///
+/// ```
+/// use datafusion_common::utils::hex::{HexCase, encode_bytes};
+///
+/// assert_eq!(encode_bytes(&[0xde, 0xad, 0xbe, 0xef], HexCase::Lower), "deadbeef");
+/// assert_eq!(encode_bytes(&[0xde, 0xad, 0xbe, 0xef], HexCase::Upper), "DEADBEEF");
+/// ```
+#[inline]
+pub fn encode_bytes(bytes: &[u8], case: HexCase) -> String {
Review Comment:
I think this function will perform poorly as it allocates -- it might be a good idea as a follow on PR to remove. this function and rewrite any uses of it so they don't allocate a bunch of strings
##########
datafusion/functions/src/string/to_hex.rs:
##########
@@ -24,16 +24,14 @@ use arrow::datatypes::{
Int64Type, UInt8Type, UInt16Type, UInt32Type, UInt64Type,
};
use datafusion_common::cast::as_primitive_array;
+use datafusion_common::utils::hex::{HexCase, encode_u64};
use datafusion_common::{Result, ScalarValue, exec_err, internal_err};
use datafusion_expr::{
Coercion, ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
TypeSignatureClass, Volatility,
};
use datafusion_macros::user_doc;
-/// Hex lookup table for fast conversion
-const HEX_CHARS: &[u8; 16] = b"0123456789abcdef";
-
/// Converts the number to its equivalent hexadecimal representation.
/// to_hex(2147483647) = '7fffffff'
fn to_hex_array<T: ArrowPrimitiveType>(array: &ArrayRef) -> Result<ArrayRef>
Review Comment:
here is an good example of creating StringArrays directly with hex
##########
datafusion/common/src/utils/hex.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Hex encoding of bytes and integers.
+//!
+//! [`encode_bytes`] and [`encode_bytes_into`] encode a byte slice into an
+//! owned `String` or an appended `Vec<u8>`, respectively; [`encode_bytes_to_slice`]
+//! writes into a caller-provided, pre-sized buffer. [`encode_u64`] encodes an
+//! integer, trimming leading zeros. All four take a [`HexCase`] to choose
+//! between lowercase and uppercase digits.
+
+/// Case of the emitted hex digits.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum HexCase {
+ /// Digits `0123456789abcdef`.
+ Lower,
+ /// Digits `0123456789ABCDEF`.
+ Upper,
+}
+
+const LOWER_DIGITS: &[u8; 16] = b"0123456789abcdef";
+const UPPER_DIGITS: &[u8; 16] = b"0123456789ABCDEF";
+
+/// Maps a full byte to its two hex digits, so encoding advances a whole byte
+/// per iteration instead of a nibble.
+const LOOKUP_LOWER: [[u8; 2]; 256] = build_lookup(LOWER_DIGITS);
+const LOOKUP_UPPER: [[u8; 2]; 256] = build_lookup(UPPER_DIGITS);
+
+const fn build_lookup(digits: &[u8; 16]) -> [[u8; 2]; 256] {
+ let mut table = [[0u8; 2]; 256];
+ let mut i = 0;
+ while i < 256 {
+ table[i][0] = digits[i >> 4];
+ table[i][1] = digits[i & 0xF];
+ i += 1;
+ }
+ table
+}
+
+impl HexCase {
+ #[inline]
+ const fn lookup(self) -> &'static [[u8; 2]; 256] {
+ match self {
+ HexCase::Lower => &LOOKUP_LOWER,
+ HexCase::Upper => &LOOKUP_UPPER,
+ }
+ }
+
+ #[inline]
+ const fn digits(self) -> &'static [u8; 16] {
+ match self {
+ HexCase::Lower => LOWER_DIGITS,
+ HexCase::Upper => UPPER_DIGITS,
+ }
+ }
+}
+
+/// Appends the hex encoding of `bytes` to `out`.
+///
+/// Allocates only through `out`'s own growth. Callers that must bound or guard
+/// that growth should reserve capacity in `out` before calling.
+#[inline(always)]
+pub fn encode_bytes_into(bytes: &[u8], case: HexCase, out: &mut Vec<u8>) {
+ let lookup = case.lookup();
+ for &byte in bytes {
+ out.extend_from_slice(&lookup[byte as usize]);
+ }
+}
+
+/// Writes the hex encoding of `bytes` into `out`.
+///
+/// `out` must be exactly `2 * bytes.len()` bytes long. This is for callers
+/// that already own a pre-sized buffer (for example a slice of a larger,
+/// pre-allocated output array) and want to write directly into it rather
+/// than appending to a `Vec`.
+///
+/// The length requirement is only checked in debug builds, via
+/// `debug_assert_eq!`. In release builds a mismatched `out` length is not
+/// an error: only `min(bytes.len(), out.len() / 2)` bytes are encoded, so
+/// an `out` that is too short silently drops the remaining input, and an
+/// `out` that is too long is left with unwritten, stale bytes at the end.
+/// Callers are responsible for sizing `out` correctly.
+///
+/// # Example
+///
+/// ```
+/// use datafusion_common::utils::hex::{HexCase, encode_bytes_to_slice};
+///
+/// let mut out = [0u8; 8];
+/// encode_bytes_to_slice(&[0xde, 0xad, 0xbe, 0xef], HexCase::Lower, &mut out);
+/// assert_eq!(&out, b"deadbeef");
+/// ```
+#[inline(always)]
+pub fn encode_bytes_to_slice(bytes: &[u8], case: HexCase, out: &mut [u8]) {
+ debug_assert_eq!(out.len(), bytes.len() * 2);
+ let lookup = case.lookup();
+ for (&b, chunk) in bytes.iter().zip(out.chunks_exact_mut(2)) {
+ chunk.copy_from_slice(&lookup[b as usize]);
+ }
+}
+
+/// Returns the hex encoding of `bytes` as an owned `String`.
+///
+/// # Example
+///
+/// ```
+/// use datafusion_common::utils::hex::{HexCase, encode_bytes};
+///
+/// assert_eq!(encode_bytes(&[0xde, 0xad, 0xbe, 0xef], HexCase::Lower), "deadbeef");
+/// assert_eq!(encode_bytes(&[0xde, 0xad, 0xbe, 0xef], HexCase::Upper), "DEADBEEF");
+/// ```
+#[inline]
+pub fn encode_bytes(bytes: &[u8], case: HexCase) -> String {
+ let mut out = Vec::with_capacity(bytes.len() * 2);
+ encode_bytes_into(bytes, case, &mut out);
+ // SAFETY: `out` holds only ASCII hex digits, which are valid UTF-8.
+ unsafe { String::from_utf8_unchecked(out) }
+}
+
+/// Writes `v` as hex into `buf` and returns the written subslice.
+///
+/// Digits are written right-aligned with leading zeros trimmed, so the result
+/// borrows the tail of `buf`. Zero encodes as `"0"`.
+///
+/// Signed values should be cast with `as u64`, which yields the two's
+/// complement representation that both `to_hex` and Spark's `hex` produce for
+/// negative input.
+///
+/// # Example
+///
+/// The caller owns the buffer and can reuse it across calls; each call
+/// returns a fresh subslice of it, borrowed for as long as `buf` is:
+///
+/// ```
+/// use datafusion_common::utils::hex::{HexCase, encode_u64};
+///
+/// let mut buf = [0u8; 16];
+/// assert_eq!(encode_u64(0xAB, HexCase::Lower, &mut buf), b"ab");
+/// assert_eq!(encode_u64(0, HexCase::Lower, &mut buf), b"0");
+/// ```
+#[inline(always)]
+pub fn encode_u64(v: u64, case: HexCase, buf: &mut [u8; 16]) -> &[u8] {
+ let start = write_digits(v, case, buf);
+ &buf[start..]
+}
+
+/// Writes the digits of `v` right-aligned in `buf`, returning the index of the
+/// first digit.
+///
+/// Split out from [`encode_u64`] so the mutable borrow of `buf` ends before the
+/// returned slice reborrows it. The `v == 0` case returns early; if that early
+/// return instead reborrowed `buf` as `&buf[..]` inline in [`encode_u64`], the
+/// borrow checker would extend that reborrow over the rest of the function
+/// body, conflicting with the mutable writes on the non-zero path below.
Review Comment:
I think it would be enough here to describe the borrow checker and not try to summarize the implementation detail too
```suggestion
/// returned slice reborrows it.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#discussion_r3634091608
##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -233,7 +184,7 @@ fn hex_encode_int64(
for v in iter {
if let Some(num) = v {
let mut temp = [0u8; 16];
- let slice = hex_int64(num, &mut temp);
+ let slice = encode_u64(num as u64, HexCase::Upper, &mut temp);
Review Comment:
this just caught my eye when using u64 for hex.
hex still supports negative input
```
scala> spark.sql("select hex(1)").show(false)
+------+
|hex(1)|
+------+
|1 |
+------+
scala> spark.sql("select hex(-2)").show(false)
+----------------+
|hex(-2) |
+----------------+
|FFFFFFFFFFFFFFFE|
+----------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #23766:
URL: https://github.com/apache/datafusion/pull/23766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#discussion_r3650307726
##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -233,7 +184,7 @@ fn hex_encode_int64(
for v in iter {
if let Some(num) = v {
let mut temp = [0u8; 16];
- let slice = hex_int64(num, &mut temp);
+ let slice = encode_u64(num as u64, HexCase::Upper, &mut temp);
Review Comment:
Negative input still works — the `u64` here is the two's-complement bit pattern, not a range restriction.
`num as u64` on an `i64` reinterprets the bits rather than saturating, so `-2` becomes `0xFFFFFFFFFFFFFFFE` and prints as `FFFFFFFFFFFFFFFE`, matching the Spark output you posted. This is the same cast the previous `hex_int64` did, so behaviour is unchanged by this PR — only the digit-writing loop moved.
`test_hex_int64` already pins it, and still passes:
```rust
(i64::MAX, "7FFFFFFFFFFFFFFF"),
(i64::MIN, "8000000000000000"),
(-1, "FFFFFFFFFFFFFFFF"),
```
The `i64::MIN` and `-1` cases are the interesting ones: they only produce those strings if the full 64-bit pattern survives the cast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5045142596
Bench results look good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5041433685
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23766#issuecomment-5041334503)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD auto-opt_to_hex-datafusion-20260721-094849
----- ---- ------------------------------------------
sha2/array_binary_256/size=1024 1.17 136.3±0.40µs ? ?/sec 1.00 116.4±0.35µs ? ?/sec
sha2/array_binary_256/size=4096 1.16 544.5±1.68µs ? ?/sec 1.00 468.0±1.23µs ? ?/sec
sha2/array_binary_256/size=8192 1.15 1085.3±3.56µs ? ?/sec 1.00 940.8±2.63µs ? ?/sec
sha2/array_scalar_binary_256/size=1024 1.16 135.7±0.17µs ? ?/sec 1.00 117.2±0.17µs ? ?/sec
sha2/array_scalar_binary_256/size=4096 1.15 542.0±0.69µs ? ?/sec 1.00 470.1±0.80µs ? ?/sec
sha2/array_scalar_binary_256/size=8192 1.14 1081.3±1.54µs ? ?/sec 1.00 947.5±1.66µs ? ?/sec
sha2/scalar/size=1 1.09 249.7±0.40ns ? ?/sec 1.00 229.8±0.72ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sha2 — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 430.1s |
| Peak memory | 32.2 MiB |
| Avg memory | 4.3 MiB |
| CPU user | 84.9s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**sha2 — branch**
| Metric | Value |
|--------|-------|
| Wall time | 430.1s |
| Peak memory | 30.7 MiB |
| Avg memory | 4.7 MiB |
| CPU user | 92.3s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5041437718
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23766#issuecomment-5041334503)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD auto-opt_to_hex-datafusion-20260721-094849
----- ---- ------------------------------------------
hex_binary/size=1024 1.00 34.4±0.06µs ? ?/sec 1.00 34.2±0.05µs ? ?/sec
hex_binary/size=4096 1.01 145.5±0.32µs ? ?/sec 1.00 144.7±0.21µs ? ?/sec
hex_binary/size=8192 1.01 285.0±0.56µs ? ?/sec 1.00 283.1±0.42µs ? ?/sec
hex_int64/size=1024 1.00 8.8±0.03µs ? ?/sec 1.02 9.0±0.03µs ? ?/sec
hex_int64/size=4096 1.00 33.7±0.11µs ? ?/sec 1.01 34.2±0.10µs ? ?/sec
hex_int64/size=8192 1.01 69.6±0.28µs ? ?/sec 1.00 68.7±0.28µs ? ?/sec
hex_int64_dict/size=1024 1.00 8.8±0.01µs ? ?/sec 1.02 9.0±0.02µs ? ?/sec
hex_int64_dict/size=4096 1.00 33.6±0.07µs ? ?/sec 1.00 33.7±0.08µs ? ?/sec
hex_int64_dict/size=8192 1.00 67.0±0.11µs ? ?/sec 1.00 67.1±0.12µs ? ?/sec
hex_utf8/size=1024 1.00 34.3±0.05µs ? ?/sec 1.00 34.3±0.14µs ? ?/sec
hex_utf8/size=4096 1.01 145.6±0.16µs ? ?/sec 1.00 144.8±0.22µs ? ?/sec
hex_utf8/size=8192 1.00 285.3±0.26µs ? ?/sec 1.00 284.4±0.33µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**hex — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 470.1s |
| Peak memory | 29.9 MiB |
| Avg memory | 6.6 MiB |
| CPU user | 150.7s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**hex — branch**
| Metric | Value |
|--------|-------|
| Wall time | 430.1s |
| Peak memory | 29.8 MiB |
| Avg memory | 7.0 MiB |
| CPU user | 147.4s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5041392109
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23766#issuecomment-5041334503)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD auto-opt_to_hex-datafusion-20260721-094849
----- ---- ------------------------------------------
md5_array 1.09 125.1±5.89µs ? ?/sec 1.00 114.7±0.88µs ? ?/sec
md5_scalar 1.02 323.8±3.34ns ? ?/sec 1.00 318.9±0.86ns ? ?/sec
sha224_array 1.00 55.7±0.30µs ? ?/sec 1.00 55.6±0.32µs ? ?/sec
sha224_scalar 1.00 231.1±2.33ns ? ?/sec 1.00 231.3±2.38ns ? ?/sec
sha256_array 1.00 54.7±0.25µs ? ?/sec 1.00 54.5±0.27µs ? ?/sec
sha256_scalar 1.00 231.8±2.62ns ? ?/sec 1.04 240.9±2.74ns ? ?/sec
sha384_array 1.00 113.2±0.62µs ? ?/sec 1.00 113.2±0.60µs ? ?/sec
sha384_scalar 1.01 315.5±2.86ns ? ?/sec 1.00 313.1±2.25ns ? ?/sec
sha512_array 1.00 119.6±0.19µs ? ?/sec 1.00 119.1±0.19µs ? ?/sec
sha512_scalar 1.01 315.8±3.19ns ? ?/sec 1.00 311.7±2.88ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**crypto — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 225.0s |
| Peak memory | 31.9 MiB |
| Avg memory | 11.4 MiB |
| CPU user | 118.6s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**crypto — branch**
| Metric | Value |
|--------|-------|
| Wall time | 260.1s |
| Peak memory | 26.9 MiB |
| Avg memory | 8.7 MiB |
| CPU user | 119.0s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5049352173
Filed the follow-ons:
- #23810 — avoid the per-row `String` allocation in `md5` / `sha1` / `sha2`, and drop `encode_bytes` once its callers no longer need it. Covers both the `hex.rs` and `md5.rs` comments, since they are the same fix at different call sites.
- #23811 — whether `to_hex`'s Arrow integer type dispatch should move into `datafusion-common` too. I wrote up the case against as well as for: the widening rules are the part most likely to want per-dialect variation, so it may reasonably close as working-as-intended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: optimize to_hex in datafusion-functions [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5036799118
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/23766?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `50.00000%` with `10 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.70%. Comparing base ([`050046a`](https://app.codecov.io/gh/apache/datafusion/commit/050046a7f0c08d64d8dba9ebf0e3544cbda6cd38?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d791b9a`](https://app.codecov.io/gh/apache/datafusion/commit/d791b9a4b976454ef1b603d77df2049db112283e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/23766?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/functions/src/string/to\_hex.rs](https://app.codecov.io/gh/apache/datafusion/pull/23766?src=pr&el=tree&filepath=datafusion%2Ffunctions%2Fsrc%2Fstring%2Fto_hex.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mdW5jdGlvbnMvc3JjL3N0cmluZy90b19oZXgucnM=) | 50.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23766?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #23766 +/- ##
==========================================
- Coverage 80.71% 80.70% -0.01%
==========================================
Files 1089 1089
Lines 368750 368765 +15
Branches 368750 368765 +15
==========================================
- Hits 297631 297626 -5
- Misses 53375 53391 +16
- Partials 17744 17748 +4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/23766?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5049312854
Thanks for the review @alamb.
> I wonder if we have something similar with base64 🤔
I checked, and happily no: there is exactly one hand-written base64 site, `BASE64_ENGINE` / `BASE64_ENGINE_PADDED` in `encoding/inner.rs`. Everything else that greps for base64 is generated proto code going through `pbjson`'s own helper. Hex was the outlier because `to_hex`, Spark's `hex`, `md5`, `sha1` and `sha2` had each independently grown a lookup table, and they had drifted into two different levels of optimization.
Addressed in f30b8e84d:
- `encode_bytes_to_slice` now checks the output length unconditionally and returns an internal error, instead of the `debug_assert` that let release builds silently encode only part of the input. Good catch — the doc comment I had written described that silent truncation as if documenting it made it acceptable.
- Took your suggestion on the `write_digits` comment.
Leaving as follow-ons, as you suggested:
- `encode_bytes` returning an owned `String`, and `md5` building those strings before copying into the `StringViewArray`. Both allocate more than they need to, and both predate this PR — worth a separate change that reworks the callers onto a reusable buffer and a `StringViewBuilder` rather than papering over it here.
- Moving `to_hex`'s scalar path into the common crate.
Happy to file issues for those if you'd like them tracked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5041340758
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23766#issuecomment-5041334503)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5041334503-1216-vcnrm 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing auto-opt/to_hex-datafusion-20260721-094849 (5e7e2112731a9e0810f4555a6fbf5b1b2636ca8e) to eef1017 (merge-base) [diff](https://github.com/apache/datafusion/compare/eef101769d650b2b06ac2bcf298f71113a047c9d..5e7e2112731a9e0810f4555a6fbf5b1b2636ca8e) using: to_hex
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5041334503
run benchmarks to_hex hex sha2 crypto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5041398181
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23766#issuecomment-5041334503)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD auto-opt_to_hex-datafusion-20260721-094849
----- ---- ------------------------------------------
to_hex size=1024/i32_random 1.09 9.3±0.28µs ? ?/sec 1.00 8.6±0.24µs ? ?/sec
to_hex size=1024/i64_large_values 1.05 10.0±0.36µs ? ?/sec 1.00 9.5±0.34µs ? ?/sec
to_hex size=1024/i64_random 1.06 10.0±0.35µs ? ?/sec 1.00 9.5±0.33µs ? ?/sec
to_hex size=4096/i32_random 1.06 35.4±1.12µs ? ?/sec 1.00 33.4±0.97µs ? ?/sec
to_hex size=4096/i64_large_values 1.06 39.3±1.45µs ? ?/sec 1.00 37.0±1.35µs ? ?/sec
to_hex size=4096/i64_random 1.06 39.2±1.38µs ? ?/sec 1.00 37.1±1.33µs ? ?/sec
to_hex size=8192/i32_random 1.07 72.4±2.04µs ? ?/sec 1.00 67.7±1.89µs ? ?/sec
to_hex size=8192/i64_large_values 1.06 78.5±3.83µs ? ?/sec 1.00 73.8±2.69µs ? ?/sec
to_hex size=8192/i64_random 1.05 78.3±2.74µs ? ?/sec 1.00 74.2±2.60µs ? ?/sec
to_hex/scalar_i32 1.02 116.1±2.93ns ? ?/sec 1.00 113.9±1.70ns ? ?/sec
to_hex/scalar_i64 1.02 117.4±2.38ns ? ?/sec 1.00 115.3±1.77ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**to_hex — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 320.1s |
| Peak memory | 27.6 MiB |
| Avg memory | 11.6 MiB |
| CPU user | 167.4s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**to_hex — branch**
| Metric | Value |
|--------|-------|
| Wall time | 275.1s |
| Peak memory | 28.5 MiB |
| Avg memory | 14.2 MiB |
| CPU user | 172.3s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share hex encoding across datafusion-common, functions, and spark [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23766:
URL: https://github.com/apache/datafusion/pull/23766#issuecomment-5041350367
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23766#issuecomment-5041334503)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5041334503-1217-72glc 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing auto-opt/to_hex-datafusion-20260721-094849 (5e7e2112731a9e0810f4555a6fbf5b1b2636ca8e) to eef1017 (merge-base) [diff](https://github.com/apache/datafusion/compare/eef101769d650b2b06ac2bcf298f71113a047c9d..5e7e2112731a9e0810f4555a6fbf5b1b2636ca8e) using: hex
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org