You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/25 19:10:20 UTC
(datafusion) branch gh-readonly-queue/main/pr-23881-68a676e1c1e2719aa8b985616b121c1b1b288ef2 created (now 36c417b6c1)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-23881-68a676e1c1e2719aa8b985616b121c1b1b288ef2
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 36c417b6c1 perf: avoid per-row String allocation in Spark bin and char (#23881)
This branch includes the following new commits:
new 36c417b6c1 perf: avoid per-row String allocation in Spark bin and char (#23881)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: perf: avoid per-row String allocation in Spark bin and char (#23881)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-23881-68a676e1c1e2719aa8b985616b121c1b1b288ef2
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 36c417b6c14244526c81ee5dc4210740ab7b44f8
Author: Andy Grove <ag...@apache.org>
AuthorDate: Sat Jul 25 13:09:56 2026 -0600
perf: avoid per-row String allocation in Spark bin and char (#23881)
## Which issue does this PR close?
N/A
## Rationale for this change
Two Spark functions allocated a `String` per row purely to render a
small,
bounded amount of text:
- `bin` called `format!("{value:b}")` for every row.
- `char` called `ch.to_string()` for every row — a heap allocation for a
single
character.
In both cases the output has a known upper bound (64 binary digits for
an `i64`,
4 bytes for a UTF-8 character), so the rendering fits in a stack buffer
and the
result can be appended straight to a pre-sized builder.
## What changes are included in this PR?
`math/bin.rs`:
- `spark_bin` now writes digits right-aligned into a caller-supplied
`[u8; 64]`
and returns a `&str` borrowed from it, instead of returning an owned
`String`.
- The `collect::<StringArray>()` becomes an explicit loop over a
`StringBuilder::with_capacity`, sized at 8 digits per row.
- Negative values still render as their two's-complement bit pattern,
matching
`{:b}`. The digit loop is a `loop`, not a `while`, so zero renders as
`"0"`
rather than the empty string.
`string/char.rs`:
- `ch.to_string()` becomes `ch.encode_utf8(&mut encoded)` against a
`[u8; 4]`
hoisted out of the loop.
Output is unchanged in both cases.
## Are these changes tested?
Existing coverage pins the behaviour. `spark/math/bin.slt` asserts
concrete
output for the cases the rewrite had to get right: zero, negative
values,
`i64::MIN` (`-9223372036854775808`), `i64::MAX`, and `-2147483648` /
`-32768` widened from narrower integer types. `spark/string/char.slt`
covers
the negative-input empty string, the null path, and characters on both
sides of
the ASCII boundary (`char(256)` and above wrap via `% 256`).
All 119 `spark/math` and `spark/string` sqllogictest files pass, along
with the
258 `datafusion-spark` unit tests.
The `bin` benchmark used below, `datafusion/spark/benches/bin.rs`, is
added
separately in #23882 so the baseline can be measured on `main` before
this
change lands. It covers 1024 and 8192 rows with 20% nulls over two value
distributions: small values that render to a handful of digits, and
full-range
values that render to the maximum 64. `char` already had
`datafusion/spark/benches/char.rs` on `main`, so no benchmark change is
needed
for it.
### Benchmarks
Criterion, `apache/main` @ `f1ab86dad` as baseline. Median of the
reported
change interval.
| Benchmark | 1024 | 8192 |
| --- | --- | --- |
| `bin/small` | −75.7% | −73.7% |
| `bin/wide` | −47.8% | −49.8% |
`char` (1024 rows): −76.1%.
## Are there any user-facing changes?
No. Both functions produce byte-identical output; this is purely an
allocation
change.
---
datafusion/spark/src/function/math/bin.rs | 44 ++++++++++++++++++++++------
datafusion/spark/src/function/string/char.rs | 4 ++-
2 files changed, 38 insertions(+), 10 deletions(-)
diff --git a/datafusion/spark/src/function/math/bin.rs b/datafusion/spark/src/function/math/bin.rs
index 82afd48e8d..e6a0e1a735 100644
--- a/datafusion/spark/src/function/math/bin.rs
+++ b/datafusion/spark/src/function/math/bin.rs
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-use arrow::array::{ArrayRef, AsArray, StringArray};
+use arrow::array::{Array, ArrayRef, AsArray, StringBuilder};
use arrow::datatypes::{DataType, Field, FieldRef, Int64Type};
use datafusion_common::types::{NativeType, logical_int64};
use datafusion_common::utils::take_function_args;
@@ -88,12 +88,20 @@ fn spark_bin_inner(arg: &[ArrayRef]) -> Result<ArrayRef> {
let [array] = take_function_args("bin", arg)?;
match &array.data_type() {
DataType::Int64 => {
- let result: StringArray = array
- .as_primitive::<Int64Type>()
- .iter()
- .map(|opt| opt.map(spark_bin))
- .collect();
- Ok(Arc::new(result))
+ let array = array.as_primitive::<Int64Type>();
+ let len = array.len();
+ // Most values are small, so 8 digits per row is a reasonable estimate;
+ // the buffer grows on its own for wider ones.
+ let mut builder = StringBuilder::with_capacity(len, len * 8);
+ // Digits are rendered into this stack buffer, so no row allocates.
+ let mut digits = [0u8; MAX_BIN_DIGITS];
+ for value in array.iter() {
+ match value {
+ Some(value) => builder.append_value(spark_bin(value, &mut digits)),
+ None => builder.append_null(),
+ }
+ }
+ Ok(Arc::new(builder.finish()))
}
data_type => {
internal_err!("bin does not support: {data_type}")
@@ -101,6 +109,24 @@ fn spark_bin_inner(arg: &[ArrayRef]) -> Result<ArrayRef> {
}
}
-fn spark_bin(value: i64) -> String {
- format!("{value:b}")
+/// An `i64` renders as at most 64 binary digits.
+const MAX_BIN_DIGITS: usize = 64;
+
+/// Renders `value` as binary, right-aligned in `digits`, and returns the digits written.
+///
+/// Negative values render as their two's-complement bit pattern, matching `{:b}`.
+fn spark_bin(value: i64, digits: &mut [u8; MAX_BIN_DIGITS]) -> &str {
+ let mut pos = MAX_BIN_DIGITS;
+ let mut remaining = value as u64;
+ // `while` alone would produce an empty string for zero.
+ loop {
+ pos -= 1;
+ digits[pos] = b'0' + (remaining & 1) as u8;
+ remaining >>= 1;
+ if remaining == 0 {
+ break;
+ }
+ }
+ // SAFETY: every byte written above is an ASCII '0' or '1'.
+ unsafe { std::str::from_utf8_unchecked(&digits[pos..]) }
}
diff --git a/datafusion/spark/src/function/string/char.rs b/datafusion/spark/src/function/string/char.rs
index 15b00ee98f..5d6de3ae36 100644
--- a/datafusion/spark/src/function/string/char.rs
+++ b/datafusion/spark/src/function/string/char.rs
@@ -112,6 +112,8 @@ fn chr(args: &[ArrayRef]) -> Result<ArrayRef> {
integer_array.len(),
);
+ // Each character encodes into this stack buffer, so no row allocates a `String`.
+ let mut encoded = [0u8; 4];
for integer_opt in integer_array {
match integer_opt {
Some(integer) => {
@@ -119,7 +121,7 @@ fn chr(args: &[ArrayRef]) -> Result<ArrayRef> {
builder.append_value(""); // empty string for negative numbers.
} else {
match core::char::from_u32((integer % 256) as u32) {
- Some(ch) => builder.append_value(ch.to_string()),
+ Some(ch) => builder.append_value(ch.encode_utf8(&mut encoded)),
None => {
return exec_err!(
"requested character not compatible for encoding."
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org