You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/08 13:41:06 UTC
(datafusion) branch gh-readonly-queue/main/pr-26121-5c0e73f87034ed3fb26b2ef52bf50a15942923dd created (now 55b2f093bb)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-26121-5c0e73f87034ed3fb26b2ef52bf50a15942923dd
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 55b2f093bb perf: Use `sub_view` in `trim`, `substr`, `split_part`, and `substring_index` (#26121)
This branch includes the following new commits:
new 55b2f093bb perf: Use `sub_view` in `trim`, `substr`, `split_part`, and `substring_index` (#26121)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: perf: Use `sub_view` in `trim`, `substr`, `split_part`, and `substring_index` (#26121)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-26121-5c0e73f87034ed3fb26b2ef52bf50a15942923dd
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 55b2f093bb80780b8ce253873ff816cc4740fbe2
Author: Neil Conway <ne...@gmail.com>
AuthorDate: Thu Oct 8 13:40:37 2026 +0000
perf: Use `sub_view` in `trim`, `substr`, `split_part`, and `substring_index` (#26121)
## Which issue does this PR close?
- N/A
## Rationale for this change
`sub_view` was introduced in #26039; it efficiently builds a view that
is a substring of another view. Several string functions had private
helpers that did the same thing or used `append_view`. This PR refactors
those places to use `sub_view` instead.
Both `append_view` and the private helpers were built on Arrow's
`make_view`. `make_view` has some shortcomings:
* For strings of up to 12 bytes, it does an indirect jump on the
string's length. For real-world data with variable length strings, this
jump is often poorly predicted.
* Perhaps because it has a 13-arm match statement, `make_view` was
marked as `inline(never)`, to avoid bloating the code at each call-site
In contrast, `sub_view` replaces the indirect jump with shifts and
masks, and is small enough to inline. Benchmarks show that adopting
`sub_view` improves performance by 5-40%, depending on the workload.
Along the way, simplify the `Trimmer` interface because `append_view` is
no longer used.
Benchmarks: (x86, AMD EPYC Milan):
- ltrim/spaces: 52.4µs → 37.5µs, −28%
- ltrim/heavy_padding: 285.2µs → 215.6µs, −24%
- ltrim/nothing_to_trim: 51.8µs → 29.3µs, −43%
- ltrim/char_set: 2.80ms → 2.80ms, 0%
- rtrim/spaces: 53.6µs → 37.6µs, −30%
- rtrim/heavy_padding: 322.0µs → 253.0µs, −21%
- rtrim/nothing_to_trim: 51.4µs → 29.0µs, −44%
- rtrim/char_set: 2.90ms → 2.80ms, −3%
- btrim/spaces: 87.3µs → 56.6µs, −35%
- btrim/heavy_padding: 458.6µs → 398.3µs, −13%
- btrim/nothing_to_trim: 58.1µs → 42.0µs, −28%
- btrim/char_set: 5.70ms → 5.50ms, −4%
- substr/short_result: 114.6µs → 100.5µs, −12%
- substr/long_result: 198.4µs → 181.8µs, −8%
- substr/short_result_long_input: 93.5µs → 95.5µs, +2%
- substr/no_count: 84.0µs → 62.3µs, −26%
- substr/per_row_start: 149.0µs → 119.5µs, −20%
- split_part/short_fields: 140.9µs → 113.7µs, −19%
- split_part/long_fields: 137.6µs → 113.3µs, −18%
- split_part/many_fields: 497.9µs → 427.6µs, −14%
- split_part/negative_position: 98.6µs → 89.0µs, −10%
- split_part/multi_char_delimiter: 163.8µs → 152.6µs, −7%
- split_part/per_row_position: 458.5µs → 460.2µs, 0%
- substr_index/array_single_delimiter: 295.1µs → 292.7µs, −1%
- substr_index/array_long_delimiter: 471.7µs → 466.5µs, −1%
- substr_index/scalar_single_delimiter_pos: 75.0µs → 72.0µs, −4%
- substr_index/scalar_single_delimiter_neg: 79.3µs → 75.7µs, −5%
- substr_index/scalar_long_delimiter_pos: 103.1µs → 104.5µs, +1%
- substr_index/scalar_long_delimiter_neg: 219.2µs → 207.3µs, −5%
- left/short_result: 37.2µs → 37.7µs, +1%
- left/long_result: 38.2µs → 38.2µs, 0%
- left/short_result_long_input: 50.4µs → 50.3µs, 0%
- left/n_exceeds_len: 36.4µs → 34.6µs, −5%
- left/negative_n: 34.7µs → 34.6µs, 0%
- left/per_row_n: 33.5µs → 33.8µs, +1%
- right/short_result: 48.1µs → 48.3µs, 0%
- right/long_result: 52.2µs → 45.0µs, −14%
- right/short_result_long_input: 63.1µs → 61.1µs, −3%
- right/n_exceeds_len: 46.4µs → 41.9µs, −10%
- right/negative_n: 45.7µs → 41.0µs, −10%
- right/per_row_n: 45.3µs → 43.7µs, −4%
## What changes are included in this PR?
* Refactor `trim`, `substr`, `split_part`, and `substring_index` to use
`sub_view`
* Remove `append_view` (no longer used)
* Simplify `Trimmer` interface (byte offset no longer needed)
* Optimize `sub_view` for long result strings
## What is the testing strategy for this PR?
Covered by existing tests.
## Are there any user-facing changes?
No.
---
.../benches/string_expressions/split_part.rs | 374 ++++++---------
.../functions/benches/string_expressions/trim.rs | 531 ++++++---------------
.../benches/unicode_expressions/substr.rs | 387 +++++----------
.../benches/unicode_expressions/substr_index.rs | 122 +++--
datafusion/functions/src/string/common.rs | 61 +--
datafusion/functions/src/string/split_part.rs | 26 +-
datafusion/functions/src/strings.rs | 79 +--
datafusion/functions/src/unicode/substr.rs | 6 +-
datafusion/functions/src/unicode/substrindex.rs | 31 +-
9 files changed, 530 insertions(+), 1087 deletions(-)
diff --git a/datafusion/functions/benches/string_expressions/split_part.rs b/datafusion/functions/benches/string_expressions/split_part.rs
index 910814d8e3..052ca17b71 100644
--- a/datafusion/functions/benches/string_expressions/split_part.rs
+++ b/datafusion/functions/benches/string_expressions/split_part.rs
@@ -15,254 +15,170 @@
// specific language governing permissions and limitations
// under the License.
-use arrow::array::{ArrayRef, Int64Array, StringArray, StringViewArray};
-use arrow::datatypes::{DataType, Field};
-use criterion::{BenchmarkId, Criterion, criterion_group};
+use std::hint::black_box;
+use std::ops::{Range, RangeInclusive};
+use std::sync::Arc;
+
+use arrow::array::{StringArray, StringViewArray};
+use arrow::datatypes::{Field, Int64Type};
+use arrow::util::bench_util::create_primitive_array_range;
+use criterion::{Criterion, criterion_group};
use datafusion_common::ScalarValue;
use datafusion_common::config::ConfigOptions;
-use datafusion_expr::{ColumnarValue, ScalarFunctionArgs, ScalarUDF};
+use datafusion_expr::{ColumnarValue, ReturnFieldArgs, ScalarFunctionArgs};
use datafusion_functions::string::split_part;
use rand::distr::Alphanumeric;
-use rand::prelude::StdRng;
+use rand::rngs::StdRng;
use rand::{Rng, SeedableRng};
-use std::hint::black_box;
-use std::sync::Arc;
-const N_ROWS: usize = 8192;
+const BATCH_SIZE: usize = 8192;
-/// Creates an array of strings with `num_parts` random alphanumeric segments
-/// of `part_len` bytes each, joined by `delimiter`.
-fn gen_string_array(
- n_rows: usize,
- num_parts: usize,
- part_len: usize,
+/// How the position argument is passed.
+enum Position {
+ /// The same position for every row, as in `split_part(s, '.', 2)`.
+ Scalar(i64),
+ /// A different position for each row, drawn from the range.
+ PerRow(Range<i64>),
+}
+
+/// Returns strings of alphanumeric fields joined by `delimiter`. Each row's
+/// number of fields and each field's length are drawn from the given ranges.
+fn create_strings(
+ field_len: &RangeInclusive<usize>,
+ num_fields: &RangeInclusive<usize>,
delimiter: &str,
- use_string_view: bool,
-) -> ColumnarValue {
+) -> Vec<Option<String>> {
let mut rng = StdRng::seed_from_u64(42);
-
- let mut strings: Vec<String> = Vec::with_capacity(n_rows);
- for _ in 0..n_rows {
- let mut parts: Vec<String> = Vec::with_capacity(num_parts);
- for _ in 0..num_parts {
- let part: String = (&mut rng)
- .sample_iter(&Alphanumeric)
- .take(part_len)
- .map(char::from)
+ (0..BATCH_SIZE)
+ .map(|_| {
+ if rng.random::<f32>() < 0.1 {
+ return None;
+ }
+ let fields: Vec<String> = (0..rng.random_range(num_fields.clone()))
+ .map(|_| {
+ let len = rng.random_range(field_len.clone());
+ (&mut rng)
+ .sample_iter(&Alphanumeric)
+ .take(len)
+ .map(char::from)
+ .collect()
+ })
.collect();
- parts.push(part);
- }
- strings.push(parts.join(delimiter));
- }
+ Some(fields.join(delimiter))
+ })
+ .collect()
+}
- if use_string_view {
- let string_array: StringViewArray = strings.into_iter().map(Some).collect();
- ColumnarValue::Array(Arc::new(string_array) as ArrayRef)
+fn create_args(
+ field_len: &RangeInclusive<usize>,
+ num_fields: &RangeInclusive<usize>,
+ delimiter: &str,
+ position: &Position,
+ is_string_view: bool,
+) -> Vec<ColumnarValue> {
+ let strings = create_strings(field_len, num_fields, delimiter);
+ let string_arg = if is_string_view {
+ ColumnarValue::Array(Arc::new(strings.into_iter().collect::<StringViewArray>()))
} else {
- let string_array: StringArray = strings.into_iter().map(Some).collect();
- ColumnarValue::Array(Arc::new(string_array) as ArrayRef)
- }
-}
+ ColumnarValue::Array(Arc::new(strings.into_iter().collect::<StringArray>()))
+ };
-#[expect(clippy::too_many_arguments)]
-fn bench_split_part(
- group: &mut criterion::BenchmarkGroup<'_, criterion::measurement::WallTime>,
- func: &ScalarUDF,
- config_options: &Arc<ConfigOptions>,
- name: &str,
- tag: &str,
- strings: ColumnarValue,
- delimiter: ColumnarValue,
- position: ColumnarValue,
-) {
- let args = vec![strings, delimiter, position];
- let arg_fields: Vec<_> = args
- .iter()
- .enumerate()
- .map(|(idx, arg)| Field::new(format!("arg_{idx}"), arg.data_type(), true).into())
- .collect();
- let return_type = match args[0].data_type() {
- DataType::Utf8View => DataType::Utf8View,
- _ => DataType::Utf8,
+ let position_arg = match position {
+ Position::Scalar(n) => ColumnarValue::Scalar(ScalarValue::Int64(Some(*n))),
+ Position::PerRow(range) => ColumnarValue::Array(Arc::new(
+ create_primitive_array_range::<Int64Type>(BATCH_SIZE, 0.0, range.clone()),
+ )),
};
- let return_field = Field::new("f", return_type, true).into();
- group.bench_function(BenchmarkId::new(name, tag), |b| {
- b.iter(|| {
- black_box(
- func.invoke_with_args(ScalarFunctionArgs {
- args: args.clone(),
- arg_fields: arg_fields.clone(),
- number_rows: N_ROWS,
- return_field: Arc::clone(&return_field),
- config_options: Arc::clone(config_options),
- })
- .expect("split_part should work"),
- )
- })
- });
+ vec![
+ string_arg,
+ ColumnarValue::Scalar(ScalarValue::from(delimiter)),
+ position_arg,
+ ]
}
fn criterion_benchmark(c: &mut Criterion) {
- let split_part_func = split_part();
+ // Field lengths and counts vary within each case, as in real data: with
+ // fixed lengths, per-row work that depends on them is unrealistically
+ // predictable.
+ let cases = [
+ // Fields of up to 8 bytes, stored inline in StringView arrays (≤12 bytes).
+ ("short_fields", 1..=8, 3..=6, ".", Position::Scalar(2)),
+ // Fields of more than 12 bytes, stored out of line.
+ ("long_fields", 16..=48, 3..=6, ".", Position::Scalar(2)),
+ // A field near the middle of many fields.
+ ("many_fields", 1..=16, 20..=50, ".", Position::Scalar(10)),
+ // A negative position counts fields from the end.
+ (
+ "negative_position",
+ 1..=32,
+ 3..=6,
+ ".",
+ Position::Scalar(-1),
+ ),
+ // A delimiter of more than one character.
+ (
+ "multi_char_delimiter",
+ 1..=32,
+ 3..=6,
+ "~@~",
+ Position::Scalar(2),
+ ),
+ // The position computed per row.
+ (
+ "per_row_position",
+ 1..=32,
+ 3..=6,
+ ".",
+ Position::PerRow(1..4),
+ ),
+ ];
+ let function = split_part();
let config_options = Arc::new(ConfigOptions::default());
- let mut group = c.benchmark_group("split_part");
-
- // ── Scalar delimiter and position ────────────────
-
- // Utf8, single-char delimiter, scalar args
- {
- let strings = gen_string_array(N_ROWS, 10, 8, ".", false);
- let delimiter = ColumnarValue::Scalar(ScalarValue::Utf8(Some(".".into())));
- let position = ColumnarValue::Scalar(ScalarValue::Int64(Some(1)));
- bench_split_part(
- &mut group,
- &split_part_func,
- &config_options,
- "scalar_utf8_single_char",
- "pos_first",
- strings,
- delimiter,
- position,
- );
- }
-
- {
- let strings = gen_string_array(N_ROWS, 10, 8, ".", false);
- let delimiter = ColumnarValue::Scalar(ScalarValue::Utf8(Some(".".into())));
- let position = ColumnarValue::Scalar(ScalarValue::Int64(Some(5)));
- bench_split_part(
- &mut group,
- &split_part_func,
- &config_options,
- "scalar_utf8_single_char",
- "pos_middle",
- strings,
- delimiter,
- position,
- );
- }
-
- {
- let strings = gen_string_array(N_ROWS, 10, 8, ".", false);
- let delimiter = ColumnarValue::Scalar(ScalarValue::Utf8(Some(".".into())));
- let position = ColumnarValue::Scalar(ScalarValue::Int64(Some(-1)));
- bench_split_part(
- &mut group,
- &split_part_func,
- &config_options,
- "scalar_utf8_single_char",
- "pos_negative",
- strings,
- delimiter,
- position,
- );
- }
-
- // Utf8, multi-char delimiter, scalar args
- {
- let strings = gen_string_array(N_ROWS, 10, 8, "~@~", false);
- let delimiter = ColumnarValue::Scalar(ScalarValue::Utf8(Some("~@~".into())));
- let position = ColumnarValue::Scalar(ScalarValue::Int64(Some(5)));
- bench_split_part(
- &mut group,
- &split_part_func,
- &config_options,
- "scalar_utf8_multi_char",
- "pos_middle",
- strings,
- delimiter,
- position,
- );
- }
-
- // Utf8, long strings, scalar args
- {
- let strings = gen_string_array(N_ROWS, 50, 16, ".", false);
- let delimiter = ColumnarValue::Scalar(ScalarValue::Utf8(Some(".".into())));
- let position = ColumnarValue::Scalar(ScalarValue::Int64(Some(25)));
- bench_split_part(
- &mut group,
- &split_part_func,
- &config_options,
- "scalar_utf8_long_strings",
- "pos_middle",
- strings,
- delimiter,
- position,
- );
- }
-
- // Utf8View, long parts, scalar args
- {
- let strings = gen_string_array(N_ROWS, 10, 32, ".", true);
- let delimiter = ColumnarValue::Scalar(ScalarValue::Utf8View(Some(".".into())));
- let position = ColumnarValue::Scalar(ScalarValue::Int64(Some(5)));
- bench_split_part(
- &mut group,
- &split_part_func,
- &config_options,
- "scalar_utf8view_long_parts",
- "pos_middle",
- strings,
- delimiter,
- position,
- );
- }
-
- // Utf8View, very long parts (256 bytes), position 1
- {
- let strings = gen_string_array(N_ROWS, 5, 256, ".", true);
- let delimiter = ColumnarValue::Scalar(ScalarValue::Utf8View(Some(".".into())));
- let position = ColumnarValue::Scalar(ScalarValue::Int64(Some(1)));
- bench_split_part(
- &mut group,
- &split_part_func,
- &config_options,
- "scalar_utf8view_very_long_parts",
- "pos_first",
- strings,
- delimiter,
- position,
- );
- }
-
- // ── Array delimiter and position ─────────────────
-
- // Utf8, single-char delimiter, array args
- {
- let strings = gen_string_array(N_ROWS, 10, 8, ".", false);
- let delimiters: StringArray = vec![Some("."); N_ROWS].into_iter().collect();
- let delimiter = ColumnarValue::Array(Arc::new(delimiters) as ArrayRef);
- let positions = ColumnarValue::Array(Arc::new(Int64Array::from(vec![5; N_ROWS])));
- bench_split_part(
- &mut group,
- &split_part_func,
- &config_options,
- "array_utf8_single_char",
- "pos_middle",
- strings,
- delimiter,
- positions,
- );
- }
-
- // Utf8, multi-char delimiter, array args
- {
- let strings = gen_string_array(N_ROWS, 10, 8, "~@~", false);
- let delimiters: StringArray = vec![Some("~@~"); N_ROWS].into_iter().collect();
- let delimiter = ColumnarValue::Array(Arc::new(delimiters) as ArrayRef);
- let positions = ColumnarValue::Array(Arc::new(Int64Array::from(vec![5; N_ROWS])));
- bench_split_part(
- &mut group,
- &split_part_func,
- &config_options,
- "array_utf8_multi_char",
- "pos_middle",
- strings,
- delimiter,
- positions,
- );
+ let mut group = c.benchmark_group(function.name().to_string());
+
+ for is_string_view in [false, true] {
+ let array_type = if is_string_view {
+ "string_view"
+ } else {
+ "string"
+ };
+
+ for (case_name, field_len, num_fields, delimiter, position) in &cases {
+ let args =
+ create_args(field_len, num_fields, delimiter, position, is_string_view);
+
+ let arg_fields: Vec<_> = args
+ .iter()
+ .enumerate()
+ .map(|(idx, arg)| {
+ Field::new(format!("arg_{idx}"), arg.data_type(), true).into()
+ })
+ .collect();
+ let scalar_arguments = vec![None; arg_fields.len()];
+ let return_field = function
+ .return_field_from_args(ReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &scalar_arguments,
+ })
+ .expect("should resolve return field");
+
+ group.bench_function(format!("{array_type} {case_name}"), |b| {
+ b.iter(|| {
+ black_box(
+ function
+ .invoke_with_args(ScalarFunctionArgs {
+ args: args.clone(),
+ arg_fields: arg_fields.clone(),
+ number_rows: BATCH_SIZE,
+ return_field: Arc::clone(&return_field),
+ config_options: Arc::clone(&config_options),
+ })
+ .expect("should work"),
+ )
+ })
+ });
+ }
}
group.finish();
diff --git a/datafusion/functions/benches/string_expressions/trim.rs b/datafusion/functions/benches/string_expressions/trim.rs
index 9970cf2f80..bc318758f9 100644
--- a/datafusion/functions/benches/string_expressions/trim.rs
+++ b/datafusion/functions/benches/string_expressions/trim.rs
@@ -15,418 +15,161 @@
// specific language governing permissions and limitations
// under the License.
-use arrow::array::{ArrayRef, LargeStringArray, StringArray, StringViewArray};
-use arrow::datatypes::{DataType, Field};
-use criterion::{
- BenchmarkGroup, Criterion, SamplingMode, criterion_group, measurement::Measurement,
-};
-use datafusion_common::ScalarValue;
-use datafusion_common::config::ConfigOptions;
-use datafusion_expr::{ColumnarValue, ScalarFunctionArgs, ScalarUDF};
-use datafusion_functions::string;
-use rand::{Rng, SeedableRng, distr::Alphanumeric, rngs::StdRng};
use std::hint::black_box;
-use std::{fmt, sync::Arc};
-
-#[derive(Clone, Copy)]
-pub enum StringArrayType {
- Utf8View,
- Utf8,
- LargeUtf8,
-}
-
-impl fmt::Display for StringArrayType {
- fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
- match self {
- StringArrayType::Utf8View => f.write_str("string_view"),
- StringArrayType::Utf8 => f.write_str("string"),
- StringArrayType::LargeUtf8 => f.write_str("large_string"),
- }
- }
-}
+use std::ops::RangeInclusive;
+use std::sync::Arc;
-#[derive(Clone, Copy)]
-pub enum TrimType {
- Ltrim,
- Rtrim,
- Btrim,
+use arrow::array::{StringArray, StringViewArray};
+use arrow::datatypes::Field;
+use criterion::{Criterion, criterion_group};
+use datafusion_common::ScalarValue;
+use datafusion_common::config::ConfigOptions;
+use datafusion_expr::{ColumnarValue, ReturnFieldArgs, ScalarFunctionArgs};
+use datafusion_functions::string::{btrim, ltrim, rtrim};
+use rand::distr::Alphanumeric;
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+
+const BATCH_SIZE: usize = 8192;
+
+/// The characters to trim.
+enum Pattern {
+ /// Spaces, as in `trim(s)`.
+ Spaces,
+ /// Any of the given characters, as in `trim(s, ',!()')`.
+ Chars(&'static str),
}
-impl fmt::Display for TrimType {
- fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
- match self {
- TrimType::Ltrim => f.write_str("ltrim"),
- TrimType::Rtrim => f.write_str("rtrim"),
- TrimType::Btrim => f.write_str("btrim"),
- }
+/// Returns strings of alphanumeric content, with padding to trim at the start
+/// and/or end. Each row's content and padding lengths are drawn from the given
+/// ranges.
+fn create_strings(
+ content_len: &RangeInclusive<usize>,
+ pad_len: &RangeInclusive<usize>,
+ pattern: &Pattern,
+ pad_start: bool,
+ pad_end: bool,
+) -> Vec<Option<String>> {
+ let mut rng = StdRng::seed_from_u64(42);
+ let pad_chars = match pattern {
+ Pattern::Spaces => " ",
+ Pattern::Chars(chars) => chars,
}
-}
-
-/// Returns an array of strings with trim characters positioned according to trim type,
-/// and `characters` as a ScalarValue.
-///
-/// For ltrim: trim characters are at the start (prefix)
-/// For rtrim: trim characters are at the end (suffix)
-/// For btrim: trim characters are at both start and end
-fn create_string_array_and_characters(
- size: usize,
- characters: &str,
- trimmed: &str,
- remaining_len: usize,
- string_array_type: StringArrayType,
- trim_type: TrimType,
-) -> (ArrayRef, ScalarValue) {
- let rng = &mut StdRng::seed_from_u64(42);
+ .as_bytes();
+ let padding = |rng: &mut StdRng| -> String {
+ let len = rng.random_range(pad_len.clone());
+ (0..len)
+ .map(|_| pad_chars[rng.random_range(0..pad_chars.len())] as char)
+ .collect()
+ };
- // Create `size` rows:
- // - 10% rows will be `None`
- // - Other 90% will be strings with `remaining_len` content length
- let string_iter = (0..size).map(|_| {
- if rng.random::<f32>() < 0.1 {
- None
- } else {
- let content: String = rng
+ (0..BATCH_SIZE)
+ .map(|_| {
+ if rng.random::<f32>() < 0.1 {
+ return None;
+ }
+ let len = rng.random_range(content_len.clone());
+ let content: String = (&mut rng)
.sample_iter(&Alphanumeric)
- .take(remaining_len)
+ .take(len)
.map(char::from)
.collect();
-
- let value = match trim_type {
- TrimType::Ltrim => format!("{trimmed}{content}"),
- TrimType::Rtrim => format!("{content}{trimmed}"),
- TrimType::Btrim => format!("{trimmed}{content}{trimmed}"),
+ let start = if pad_start {
+ padding(&mut rng)
+ } else {
+ String::new()
};
- Some(value)
- }
- });
-
- // Build the target `string array` and `characters` according to `string_array_type`
- match string_array_type {
- StringArrayType::Utf8View => (
- Arc::new(string_iter.collect::<StringViewArray>()),
- ScalarValue::Utf8View(Some(characters.to_string())),
- ),
- StringArrayType::Utf8 => (
- Arc::new(string_iter.collect::<StringArray>()),
- ScalarValue::Utf8(Some(characters.to_string())),
- ),
- StringArrayType::LargeUtf8 => (
- Arc::new(string_iter.collect::<LargeStringArray>()),
- ScalarValue::LargeUtf8(Some(characters.to_string())),
- ),
- }
-}
-
-/// Create args for the trim benchmark
-fn create_args(
- size: usize,
- characters: &str,
- trimmed: &str,
- remaining_len: usize,
- string_array_type: StringArrayType,
- trim_type: TrimType,
-) -> Vec<ColumnarValue> {
- let (string_array, pattern) = create_string_array_and_characters(
- size,
- characters,
- trimmed,
- remaining_len,
- string_array_type,
- trim_type,
- );
- vec![
- ColumnarValue::Array(string_array),
- ColumnarValue::Scalar(pattern),
- ]
-}
-
-/// Create args for trim benchmark where space characters are being trimmed
-fn create_space_trim_args(
- size: usize,
- pad_len: usize,
- remaining_len: usize,
- string_array_type: StringArrayType,
- trim_type: TrimType,
-) -> Vec<ColumnarValue> {
- let rng = &mut StdRng::seed_from_u64(42);
- let spaces = " ".repeat(pad_len);
-
- let string_iter = (0..size).map(|_| {
- if rng.random::<f32>() < 0.1 {
- None
- } else {
- let content: String = rng
- .sample_iter(&Alphanumeric)
- .take(remaining_len)
- .map(char::from)
- .collect();
-
- let value = match trim_type {
- TrimType::Ltrim => format!("{spaces}{content}"),
- TrimType::Rtrim => format!("{content}{spaces}"),
- TrimType::Btrim => format!("{spaces}{content}{spaces}"),
+ let end = if pad_end {
+ padding(&mut rng)
+ } else {
+ String::new()
};
- Some(value)
- }
- });
-
- let string_array: ArrayRef = match string_array_type {
- StringArrayType::Utf8View => Arc::new(string_iter.collect::<StringViewArray>()),
- StringArrayType::Utf8 => Arc::new(string_iter.collect::<StringArray>()),
- StringArrayType::LargeUtf8 => Arc::new(string_iter.collect::<LargeStringArray>()),
- };
-
- vec![ColumnarValue::Array(string_array)]
-}
-
-#[expect(clippy::too_many_arguments)]
-fn run_with_string_type<M: Measurement>(
- group: &mut BenchmarkGroup<'_, M>,
- trim_func: &ScalarUDF,
- trim_type: TrimType,
- size: usize,
- total_len: usize,
- characters: &str,
- trimmed: &str,
- remaining_len: usize,
- string_type: StringArrayType,
-) {
- let args = create_args(
- size,
- characters,
- trimmed,
- remaining_len,
- string_type,
- trim_type,
- );
- let arg_fields = args
- .iter()
- .enumerate()
- .map(|(idx, arg)| Field::new(format!("arg_{idx}"), arg.data_type(), true).into())
- .collect::<Vec<_>>();
- let config_options = Arc::new(ConfigOptions::default());
-
- group.bench_function(
- format!(
- "{trim_type} {string_type} [size={size}, len={total_len}, remaining={remaining_len}]",
- ),
- |b| {
- b.iter(|| {
- let args_cloned = args.clone();
- black_box(trim_func.invoke_with_args(ScalarFunctionArgs {
- args: args_cloned,
- arg_fields: arg_fields.clone(),
- number_rows: size,
- return_field: Field::new("f", DataType::Utf8, true).into(),
- config_options: Arc::clone(&config_options),
- }))
- })
- },
- );
-}
-
-#[expect(clippy::too_many_arguments)]
-fn run_trim_benchmark(
- c: &mut Criterion,
- group_name: &str,
- trim_func: &ScalarUDF,
- trim_type: TrimType,
- string_types: &[StringArrayType],
- size: usize,
- total_len: usize,
- characters: &str,
- trimmed: &str,
- remaining_len: usize,
-) {
- let mut group = c.benchmark_group(group_name);
- group.sampling_mode(SamplingMode::Flat);
- group.sample_size(10);
-
- for string_type in string_types {
- run_with_string_type(
- &mut group,
- trim_func,
- trim_type,
- size,
- total_len,
- characters,
- trimmed,
- remaining_len,
- *string_type,
- );
- }
-
- group.finish();
-}
-
-#[expect(clippy::too_many_arguments)]
-fn run_space_trim_benchmark(
- c: &mut Criterion,
- group_name: &str,
- trim_func: &ScalarUDF,
- trim_type: TrimType,
- string_types: &[StringArrayType],
- size: usize,
- pad_len: usize,
- remaining_len: usize,
-) {
- let mut group = c.benchmark_group(group_name);
- group.sampling_mode(SamplingMode::Flat);
- group.sample_size(10);
-
- let total_len = match trim_type {
- TrimType::Btrim => 2 * pad_len + remaining_len,
- _ => pad_len + remaining_len,
- };
-
- for string_type in string_types {
- let args =
- create_space_trim_args(size, pad_len, remaining_len, *string_type, trim_type);
- let arg_fields = args
- .iter()
- .enumerate()
- .map(|(idx, arg)| {
- Field::new(format!("arg_{idx}"), arg.data_type(), true).into()
- })
- .collect::<Vec<_>>();
- let config_options = Arc::new(ConfigOptions::default());
-
- group.bench_function(
- format!(
- "{trim_type} {string_type} [size={size}, len={total_len}, pad={pad_len}]",
- ),
- |b| {
- b.iter(|| {
- let args_cloned = args.clone();
- black_box(trim_func.invoke_with_args(ScalarFunctionArgs {
- args: args_cloned,
- arg_fields: arg_fields.clone(),
- number_rows: size,
- return_field: Field::new("f", DataType::Utf8, true).into(),
- config_options: Arc::clone(&config_options),
- }))
- })
- },
- );
- }
-
- group.finish();
+ Some(format!("{start}{content}{end}"))
+ })
+ .collect()
}
fn criterion_benchmark(c: &mut Criterion) {
- let ltrim = string::ltrim();
- let rtrim = string::rtrim();
- let btrim = string::btrim();
-
- let characters = ",!()";
-
- let string_types = [
- StringArrayType::Utf8View,
- StringArrayType::Utf8,
- StringArrayType::LargeUtf8,
- ];
-
- let trim_funcs = [
- (<rim, TrimType::Ltrim),
- (&rtrim, TrimType::Rtrim),
- (&btrim, TrimType::Btrim),
+ // Content and padding lengths vary within each case, as in real data: with
+ // fixed lengths, per-row work that depends on them is unrealistically
+ // predictable.
+ let cases = [
+ // A little padding. Results of up to 12 bytes are stored inline in
+ // StringView arrays, and longer ones out of line.
+ ("spaces", 1..=64, 0..=8, Pattern::Spaces),
+ // Short values padded to a long width, as in `CHAR(n)` data.
+ ("heavy_padding", 1..=12, 32..=64, Pattern::Spaces),
+ // Values that are already trimmed.
+ ("nothing_to_trim", 1..=64, 0..=0, Pattern::Spaces),
+ // Short values with long padding made of several different characters.
+ ("char_set", 1..=12, 32..=64, Pattern::Chars(",!()")),
];
+ let config_options = Arc::new(ConfigOptions::default());
- for size in [4096] {
- for (trim_func, trim_type) in &trim_funcs {
- // Scenario 1: Short strings (len <= 12, inline in StringView)
- // trimmed_len=4, remaining_len=8
- let total_len = 12;
- let trimmed = characters;
- let remaining_len = total_len - trimmed.len();
- run_trim_benchmark(
- c,
- "short strings (len <= 12)",
- trim_func,
- *trim_type,
- &string_types,
- size,
- total_len,
- characters,
- trimmed,
- remaining_len,
- );
-
- // Scenario 2: Long strings, short trim (len > 12, output > 12)
- // trimmed_len=4, remaining_len=60
- let total_len = 64;
- let trimmed = characters;
- let remaining_len = total_len - trimmed.len();
- run_trim_benchmark(
- c,
- "long strings, short trim",
- trim_func,
- *trim_type,
- &string_types,
- size,
- total_len,
- characters,
- trimmed,
- remaining_len,
- );
-
- // Scenario 3: Long strings, long trim (len > 12, output <= 12)
- // trimmed_len=56, remaining_len=8
- let total_len = 64;
- let trimmed = characters.repeat(14);
- let remaining_len = total_len - trimmed.len();
- run_trim_benchmark(
- c,
- "long strings, long trim",
- trim_func,
- *trim_type,
- &string_types,
- size,
- total_len,
- characters,
- &trimmed,
- remaining_len,
- );
-
- // Scenario 4: Trim spaces, short strings (len <= 12)
- // pad_len=4, remaining_len=8
- run_space_trim_benchmark(
- c,
- "trim spaces, short strings (len <= 12)",
- trim_func,
- *trim_type,
- &string_types,
- size,
- 4,
- 8,
- );
-
- // Scenario 5: Trim spaces, long strings (len > 12)
- // pad_len=4, remaining_len=60
- run_space_trim_benchmark(
- c,
- "trim spaces, long strings",
- trim_func,
- *trim_type,
- &string_types,
- size,
- 4,
- 60,
- );
+ for (function, pad_start, pad_end) in [
+ (ltrim(), true, false),
+ (rtrim(), false, true),
+ (btrim(), true, true),
+ ] {
+ let mut group = c.benchmark_group(function.name().to_string());
+
+ for is_string_view in [false, true] {
+ let array_type = if is_string_view {
+ "string_view"
+ } else {
+ "string"
+ };
- // Scenario 6: Trim spaces, long strings, heavy padding
- // pad_len=56, remaining_len=8
- run_space_trim_benchmark(
- c,
- "trim spaces, heavy padding",
- trim_func,
- *trim_type,
- &string_types,
- size,
- 56,
- 8,
- );
+ for (case_name, content_len, pad_len, pattern) in &cases {
+ let strings =
+ create_strings(content_len, pad_len, pattern, pad_start, pad_end);
+ let mut args = vec![if is_string_view {
+ ColumnarValue::Array(Arc::new(
+ strings.into_iter().collect::<StringViewArray>(),
+ ))
+ } else {
+ ColumnarValue::Array(Arc::new(
+ strings.into_iter().collect::<StringArray>(),
+ ))
+ }];
+ if let Pattern::Chars(chars) = pattern {
+ args.push(ColumnarValue::Scalar(ScalarValue::from(*chars)));
+ }
+
+ let arg_fields: Vec<_> = args
+ .iter()
+ .enumerate()
+ .map(|(idx, arg)| {
+ Field::new(format!("arg_{idx}"), arg.data_type(), true).into()
+ })
+ .collect();
+ let scalar_arguments = vec![None; arg_fields.len()];
+ let return_field = function
+ .return_field_from_args(ReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &scalar_arguments,
+ })
+ .expect("should resolve return field");
+
+ group.bench_function(format!("{array_type} {case_name}"), |b| {
+ b.iter(|| {
+ black_box(
+ function
+ .invoke_with_args(ScalarFunctionArgs {
+ args: args.clone(),
+ arg_fields: arg_fields.clone(),
+ number_rows: BATCH_SIZE,
+ return_field: Arc::clone(&return_field),
+ config_options: Arc::clone(&config_options),
+ })
+ .expect("should work"),
+ )
+ })
+ });
+ }
}
+
+ group.finish();
}
}
diff --git a/datafusion/functions/benches/unicode_expressions/substr.rs b/datafusion/functions/benches/unicode_expressions/substr.rs
index a59dd7a007..571eee39a3 100644
--- a/datafusion/functions/benches/unicode_expressions/substr.rs
+++ b/datafusion/functions/benches/unicode_expressions/substr.rs
@@ -15,279 +15,152 @@
// specific language governing permissions and limitations
// under the License.
-use arrow::array::{ArrayRef, Int64Array, OffsetSizeTrait};
-use arrow::datatypes::{DataType, Field};
-use arrow::util::bench_util::{
- create_string_array_with_len, create_string_view_array_with_len,
-};
-use criterion::{Criterion, SamplingMode, criterion_group};
-use datafusion_common::config::ConfigOptions;
-use datafusion_common::{DataFusionError, ScalarValue};
-use datafusion_expr::{ColumnarValue, ScalarFunctionArgs};
-use datafusion_functions::unicode;
use std::hint::black_box;
+use std::ops::{Range, RangeInclusive};
use std::sync::Arc;
-fn make_i64_arg(value: i64, size: usize, as_scalar: bool) -> ColumnarValue {
- if as_scalar {
- ColumnarValue::Scalar(ScalarValue::from(value))
- } else {
- ColumnarValue::Array(Arc::new(Int64Array::from(vec![value; size])))
- }
-}
+use arrow::array::StringViewArray;
+use arrow::datatypes::{Field, Int64Type};
+use arrow::util::bench_util::{
+ create_primitive_array_range, create_string_array_with_len_range_and_prefix_and_seed,
+};
+use criterion::{Criterion, criterion_group};
+use datafusion_common::ScalarValue;
+use datafusion_common::config::ConfigOptions;
+use datafusion_expr::{ColumnarValue, ReturnFieldArgs, ScalarFunctionArgs};
+use datafusion_functions::unicode::substr;
-fn create_args_without_count<O: OffsetSizeTrait>(
- size: usize,
- str_len: usize,
- start_half_way: bool,
- force_view_types: bool,
- scalar_start: bool,
-) -> Vec<ColumnarValue> {
- let start_val = if start_half_way {
- (str_len / 2) as i64
- } else {
- 1i64
- };
- let start = make_i64_arg(start_val, size, scalar_start);
+const BATCH_SIZE: usize = 8192;
- let string_array: ArrayRef = if force_view_types {
- Arc::new(create_string_view_array_with_len(size, 0.1, str_len, false))
- } else {
- Arc::new(create_string_array_with_len::<O>(size, 0.1, str_len))
- };
+/// How an integer argument is passed.
+enum IntArg {
+ /// The same value for every row, as in `substr(s, 1, 5)`.
+ Scalar(i64),
+ /// A different value for each row, drawn from the range.
+ PerRow(Range<i64>),
+}
- vec![ColumnarValue::Array(string_array), start]
+impl IntArg {
+ fn to_columnar_value(&self) -> ColumnarValue {
+ match self {
+ IntArg::Scalar(value) => {
+ ColumnarValue::Scalar(ScalarValue::Int64(Some(*value)))
+ }
+ IntArg::PerRow(range) => ColumnarValue::Array(Arc::new(
+ create_primitive_array_range::<Int64Type>(BATCH_SIZE, 0.0, range.clone()),
+ )),
+ }
+ }
}
-fn create_args_with_count<O: OffsetSizeTrait>(
- size: usize,
- str_len: usize,
- count_max: usize,
- force_view_types: bool,
- scalar_args: bool,
+fn create_args(
+ str_len: &RangeInclusive<usize>,
+ start: &IntArg,
+ count: Option<&IntArg>,
+ is_string_view: bool,
) -> Vec<ColumnarValue> {
- let count = count_max.min(str_len) as i64;
- let start = make_i64_arg(1i64, size, scalar_args);
- let count = make_i64_arg(count, size, scalar_args);
-
- let string_array: ArrayRef = if force_view_types {
- Arc::new(create_string_view_array_with_len(size, 0.1, str_len, false))
+ let strings = create_string_array_with_len_range_and_prefix_and_seed::<i32>(
+ BATCH_SIZE,
+ 0.1,
+ *str_len.start(),
+ *str_len.end(),
+ "",
+ 42,
+ );
+ let string_arg = if is_string_view {
+ ColumnarValue::Array(Arc::new(strings.iter().collect::<StringViewArray>()))
} else {
- Arc::new(create_string_array_with_len::<O>(size, 0.1, str_len))
+ ColumnarValue::Array(Arc::new(strings))
};
- vec![ColumnarValue::Array(string_array), start, count]
-}
-
-#[expect(clippy::needless_pass_by_value)]
-fn invoke_substr_with_args(
- args: Vec<ColumnarValue>,
- number_rows: usize,
-) -> Result<ColumnarValue, DataFusionError> {
- let arg_fields = args
- .iter()
- .enumerate()
- .map(|(idx, arg)| Field::new(format!("arg_{idx}"), arg.data_type(), true).into())
- .collect::<Vec<_>>();
- let config_options = Arc::new(ConfigOptions::default());
-
- unicode::substr().invoke_with_args(ScalarFunctionArgs {
- args: args.clone(),
- arg_fields,
- number_rows,
- return_field: Field::new("f", DataType::Utf8View, true).into(),
- config_options: Arc::clone(&config_options),
- })
+ let mut args = vec![string_arg, start.to_columnar_value()];
+ args.extend(count.map(IntArg::to_columnar_value));
+ args
}
fn criterion_benchmark(c: &mut Criterion) {
- for size in [1024, 4096] {
- // string_len = 12, substring_len=6 (see `create_args_without_count`)
- let len = 12;
- let mut group = c.benchmark_group("substr, no count, short strings");
- group.sampling_mode(SamplingMode::Flat);
- group.sample_size(10);
-
- let args = create_args_without_count::<i32>(size, len, true, true, false);
- group.bench_function(
- format!("substr_string_view [size={size}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_without_count::<i32>(size, len, false, false, false);
- group.bench_function(format!("substr_string [size={size}, strlen={len}]"), |b| {
- b.iter(|| black_box(invoke_substr_with_args(args.clone(), size)))
- });
-
- let args = create_args_without_count::<i64>(size, len, true, false, false);
- group.bench_function(
- format!("substr_large_string [size={size}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- group.finish();
-
- // string_len = 128, start=1, count=64, substring_len=64
- let len = 128;
- let count = 64;
- let mut group = c.benchmark_group("substr, with count, long strings");
- group.sampling_mode(SamplingMode::Flat);
- group.sample_size(10);
-
- let args = create_args_with_count::<i32>(size, len, count, true, false);
- group.bench_function(
- format!("substr_string_view [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_with_count::<i32>(size, len, count, false, false);
- group.bench_function(
- format!("substr_string [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_with_count::<i64>(size, len, count, false, false);
- group.bench_function(
- format!("substr_large_string [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- group.finish();
-
- // string_len = 128, start=1, count=6, substring_len=6
- let len = 128;
- let count = 6;
- let mut group = c.benchmark_group("substr, short count, long strings");
- group.sampling_mode(SamplingMode::Flat);
- group.sample_size(10);
-
- let args = create_args_with_count::<i32>(size, len, count, true, false);
- group.bench_function(
- format!("substr_string_view [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_with_count::<i32>(size, len, count, false, false);
- group.bench_function(
- format!("substr_string [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_with_count::<i64>(size, len, count, false, false);
- group.bench_function(
- format!("substr_large_string [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- group.finish();
-
- // Scalar start, no count, short strings
- let len = 12;
- let mut group =
- c.benchmark_group("substr, scalar start, no count, short strings");
- group.sampling_mode(SamplingMode::Flat);
- group.sample_size(10);
-
- let args = create_args_without_count::<i32>(size, len, true, true, true);
- group.bench_function(
- format!("substr_string_view [size={size}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_without_count::<i32>(size, len, false, false, true);
- group.bench_function(format!("substr_string [size={size}, strlen={len}]"), |b| {
- b.iter(|| black_box(invoke_substr_with_args(args.clone(), size)))
- });
-
- group.finish();
-
- // Scalar start, no count, long strings, start near middle
- let len = 128;
- let mut group = c.benchmark_group("substr, scalar start, no count, long strings");
- group.sampling_mode(SamplingMode::Flat);
- group.sample_size(10);
-
- let args = create_args_without_count::<i32>(size, len, true, true, true);
- group.bench_function(
- format!("substr_string_view [size={size}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_without_count::<i32>(size, len, false, false, true);
- group.bench_function(format!("substr_string [size={size}, strlen={len}]"), |b| {
- b.iter(|| black_box(invoke_substr_with_args(args.clone(), size)))
- });
-
- group.finish();
-
- // Scalar start=1, no count, long strings
- let len = 128;
- let mut group =
- c.benchmark_group("substr, scalar start=1, no count, long strings");
- group.sampling_mode(SamplingMode::Flat);
- group.sample_size(10);
-
- let args = create_args_without_count::<i32>(size, len, false, true, true);
- group.bench_function(
- format!("substr_string_view [size={size}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_without_count::<i32>(size, len, false, false, true);
- group.bench_function(format!("substr_string [size={size}, strlen={len}]"), |b| {
- b.iter(|| black_box(invoke_substr_with_args(args.clone(), size)))
- });
-
- group.finish();
-
- // Scalar start and count, short strings
- let len = 12;
- let count = 6;
- let mut group = c.benchmark_group("substr, scalar args, short strings");
- group.sampling_mode(SamplingMode::Flat);
- group.sample_size(10);
-
- let args = create_args_with_count::<i32>(size, len, count, true, true);
- group.bench_function(
- format!("substr_string_view [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_with_count::<i32>(size, len, count, false, true);
- group.bench_function(
- format!("substr_string [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- group.finish();
-
- // Scalar start and count, long strings
- let len = 128;
- let count = 64;
- let mut group = c.benchmark_group("substr, scalar args, long strings");
- group.sampling_mode(SamplingMode::Flat);
- group.sample_size(10);
-
- let args = create_args_with_count::<i32>(size, len, count, true, true);
- group.bench_function(
- format!("substr_string_view [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_with_count::<i32>(size, len, count, false, true);
- group.bench_function(
- format!("substr_string [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- let args = create_args_with_count::<i64>(size, len, count, false, true);
- group.bench_function(
- format!("substr_large_string [size={size}, count={count}, strlen={len}]"),
- |b| b.iter(|| black_box(invoke_substr_with_args(args.clone(), size))),
- );
-
- group.finish();
+ // Input lengths vary within each case, as in real data: with fixed-length
+ // inputs, per-row work that depends on the input length is unrealistically
+ // predictable.
+ let cases = [
+ // Results of up to 5 chars, stored inline in StringView arrays (≤12 bytes).
+ (
+ "short_result",
+ 1..=32,
+ IntArg::Scalar(1),
+ Some(IntArg::Scalar(5)),
+ ),
+ // 25-char results, stored out of line.
+ (
+ "long_result",
+ 32..=64,
+ IntArg::Scalar(5),
+ Some(IntArg::Scalar(25)),
+ ),
+ // Short results from long inputs.
+ (
+ "short_result_long_input",
+ 96..=256,
+ IntArg::Scalar(1),
+ Some(IntArg::Scalar(5)),
+ ),
+ // No count, so each result runs to the end of the input.
+ ("no_count", 1..=64, IntArg::Scalar(5), None),
+ // `start` computed per row, as in `substr(s, strpos(s, ':') + 1, 8)`.
+ (
+ "per_row_start",
+ 1..=32,
+ IntArg::PerRow(1..11),
+ Some(IntArg::Scalar(8)),
+ ),
+ ];
+ let function = substr();
+ let config_options = Arc::new(ConfigOptions::default());
+ let mut group = c.benchmark_group(function.name().to_string());
+
+ for is_string_view in [false, true] {
+ let array_type = if is_string_view {
+ "string_view"
+ } else {
+ "string"
+ };
+
+ for (case_name, str_len, start, count) in &cases {
+ let args = create_args(str_len, start, count.as_ref(), is_string_view);
+ let arg_fields: Vec<_> = args
+ .iter()
+ .enumerate()
+ .map(|(idx, arg)| {
+ Field::new(format!("arg_{idx}"), arg.data_type(), true).into()
+ })
+ .collect();
+ let scalar_arguments = vec![None; arg_fields.len()];
+ let return_field = function
+ .return_field_from_args(ReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &scalar_arguments,
+ })
+ .expect("should resolve return field");
+
+ group.bench_function(format!("{array_type} {case_name}"), |b| {
+ b.iter(|| {
+ black_box(
+ function
+ .invoke_with_args(ScalarFunctionArgs {
+ args: args.clone(),
+ arg_fields: arg_fields.clone(),
+ number_rows: BATCH_SIZE,
+ return_field: Arc::clone(&return_field),
+ config_options: Arc::clone(&config_options),
+ })
+ .expect("should work"),
+ )
+ })
+ });
+ }
}
+
+ group.finish();
}
criterion_group!(benches, criterion_benchmark);
diff --git a/datafusion/functions/benches/unicode_expressions/substr_index.rs b/datafusion/functions/benches/unicode_expressions/substr_index.rs
index fb5e77cfa0..432d2536bb 100644
--- a/datafusion/functions/benches/unicode_expressions/substr_index.rs
+++ b/datafusion/functions/benches/unicode_expressions/substr_index.rs
@@ -30,6 +30,7 @@ use rand::distr::{Alphanumeric, Uniform};
use rand::prelude::Distribution;
use rand::rngs::StdRng;
+const BATCH_SIZE: usize = 8192;
const ARRAY_DATA_SEED: u64 = 0x5EED_AAAA;
const SCALAR_DATA_SEED: u64 = 0x5EED_BBBB;
@@ -101,20 +102,17 @@ fn random_token<R: Rng + ?Sized>(rng: &mut R, len: usize) -> String {
.collect()
}
-fn array_data(
- batch_size: usize,
- single_char_delimiter: bool,
-) -> (Vec<String>, Vec<String>, Vec<i64>) {
+fn array_data(single_char_delimiter: bool) -> (Vec<String>, Vec<String>, Vec<i64>) {
let count_dist = Filter {
dist: Uniform::new(-4, 5).expect("valid count distribution"),
test: |x: &i64| x != &0,
};
let mut rng = StdRng::seed_from_u64(ARRAY_DATA_SEED);
- let mut strings = Vec::with_capacity(batch_size);
- let mut delimiters = Vec::with_capacity(batch_size);
- let mut counts = Vec::with_capacity(batch_size);
+ let mut strings = Vec::with_capacity(BATCH_SIZE);
+ let mut delimiters = Vec::with_capacity(BATCH_SIZE);
+ let mut counts = Vec::with_capacity(BATCH_SIZE);
- for _ in 0..batch_size {
+ for _ in 0..BATCH_SIZE {
let length = rng.random_range(20..50);
let base = random_token(&mut rng, length);
@@ -146,11 +144,11 @@ fn array_data(
(strings, delimiters, counts)
}
-fn scalar_data(batch_size: usize, delimiter: &str) -> Vec<String> {
+fn scalar_data(delimiter: &str) -> Vec<String> {
let mut rng = StdRng::seed_from_u64(SCALAR_DATA_SEED);
- let mut strings = Vec::with_capacity(batch_size);
+ let mut strings = Vec::with_capacity(BATCH_SIZE);
- for _ in 0..batch_size {
+ for _ in 0..BATCH_SIZE {
let left_len = rng.random_range(12..24);
let middle_len = rng.random_range(12..24);
let right_len = rng.random_range(12..24);
@@ -167,7 +165,6 @@ fn run_benchmark(
b: &mut criterion::Bencher,
args: &[ColumnarValue],
return_type: &DataType,
- number_rows: usize,
) {
let arg_fields = args
.iter()
@@ -184,7 +181,7 @@ fn run_benchmark(
.invoke_with_args(ScalarFunctionArgs {
args: args.to_vec(),
arg_fields: arg_fields.clone(),
- number_rows,
+ number_rows: BATCH_SIZE,
return_field: Field::new("f", return_type.clone(), true).into(),
config_options: Arc::clone(&config_options),
})
@@ -195,61 +192,50 @@ fn run_benchmark(
fn criterion_benchmark(c: &mut Criterion) {
let mut group = c.benchmark_group("substr_index");
- let batch_sizes = [100, 1000, 10_000];
-
- for batch_size in batch_sizes {
- for rep in [StringRep::Utf8, StringRep::Utf8View] {
- let rep_name = rep.name();
-
- group.bench_function(
- format!("substr_index_{rep_name}_{batch_size}_array_single_delimiter"),
- |b| {
- let (strings, delimiters, counts) = array_data(batch_size, true);
- let args = vec![
- ColumnarValue::Array(rep.array(&strings)),
- ColumnarValue::Array(rep.array(&delimiters)),
- ColumnarValue::Array(
- Arc::new(Int64Array::from(counts)) as ArrayRef
- ),
- ];
- run_benchmark(b, &args, &rep.data_type(), batch_size);
- },
- );
-
- group.bench_function(
- format!("substr_index_{rep_name}_{batch_size}_array_long_delimiter"),
- |b| {
- let (strings, delimiters, counts) = array_data(batch_size, false);
- let args = vec![
- ColumnarValue::Array(rep.array(&strings)),
- ColumnarValue::Array(rep.array(&delimiters)),
- ColumnarValue::Array(
- Arc::new(Int64Array::from(counts)) as ArrayRef
- ),
- ];
- run_benchmark(b, &args, &rep.data_type(), batch_size);
- },
- );
-
- for (name, delimiter, count) in [
- ("single_delimiter_pos", ".", 1_i64),
- ("single_delimiter_neg", ".", -1_i64),
- ("long_delimiter_pos", "|||", 1_i64),
- ("long_delimiter_neg", "|||", -1_i64),
- ] {
- group.bench_function(
- format!("substr_index_{rep_name}_{batch_size}_scalar_{name}"),
- |b| {
- let strings = scalar_data(batch_size, delimiter);
- let args = vec![
- ColumnarValue::Array(rep.array(&strings)),
- ColumnarValue::Scalar(rep.scalar(delimiter)),
- ColumnarValue::Scalar(ScalarValue::Int64(Some(count))),
- ];
- run_benchmark(b, &args, &rep.data_type(), batch_size);
- },
- );
- }
+ for rep in [StringRep::Utf8, StringRep::Utf8View] {
+ let rep_name = rep.name();
+
+ group.bench_function(
+ format!("substr_index_{rep_name}_array_single_delimiter"),
+ |b| {
+ let (strings, delimiters, counts) = array_data(true);
+ let args = vec![
+ ColumnarValue::Array(rep.array(&strings)),
+ ColumnarValue::Array(rep.array(&delimiters)),
+ ColumnarValue::Array(Arc::new(Int64Array::from(counts)) as ArrayRef),
+ ];
+ run_benchmark(b, &args, &rep.data_type());
+ },
+ );
+
+ group.bench_function(
+ format!("substr_index_{rep_name}_array_long_delimiter"),
+ |b| {
+ let (strings, delimiters, counts) = array_data(false);
+ let args = vec![
+ ColumnarValue::Array(rep.array(&strings)),
+ ColumnarValue::Array(rep.array(&delimiters)),
+ ColumnarValue::Array(Arc::new(Int64Array::from(counts)) as ArrayRef),
+ ];
+ run_benchmark(b, &args, &rep.data_type());
+ },
+ );
+
+ for (name, delimiter, count) in [
+ ("single_delimiter_pos", ".", 1_i64),
+ ("single_delimiter_neg", ".", -1_i64),
+ ("long_delimiter_pos", "|||", 1_i64),
+ ("long_delimiter_neg", "|||", -1_i64),
+ ] {
+ group.bench_function(format!("substr_index_{rep_name}_scalar_{name}"), |b| {
+ let strings = scalar_data(delimiter);
+ let args = vec![
+ ColumnarValue::Array(rep.array(&strings)),
+ ColumnarValue::Scalar(rep.scalar(delimiter)),
+ ColumnarValue::Scalar(ScalarValue::Int64(Some(count))),
+ ];
+ run_benchmark(b, &args, &rep.data_type());
+ });
}
}
diff --git a/datafusion/functions/src/string/common.rs b/datafusion/functions/src/string/common.rs
index 0bc63afa6a..f2544fa386 100644
--- a/datafusion/functions/src/string/common.rs
+++ b/datafusion/functions/src/string/common.rs
@@ -21,7 +21,7 @@ use std::sync::Arc;
use crate::strings::{
GenericStringArrayBuilder, STRING_VIEW_INIT_BLOCK_SIZE, STRING_VIEW_MAX_BLOCK_SIZE,
- StringViewArrayBuilder, StringWriter, append_view,
+ StringViewArrayBuilder, StringWriter, substr_view,
};
use arrow::array::{
Array, ArrayRef, AsArray, GenericStringArray, NullBufferBuilder, OffsetSizeTrait,
@@ -37,15 +37,14 @@ use datafusion_expr::ColumnarValue;
/// Trait for trim operations, allowing compile-time dispatch instead of runtime matching.
///
-/// Each implementation performs its specific trim operation and returns
-/// (trimmed_str, start_offset) where start_offset is the byte offset
-/// from the beginning of the input string where the trimmed result starts.
+/// Each implementation performs its specific trim operation and returns the
+/// trimmed slice of the input.
pub(crate) trait Trimmer {
- fn trim<'a>(input: &'a str, pattern: &[char]) -> (&'a str, u32);
+ fn trim<'a>(input: &'a str, pattern: &[char]) -> &'a str;
/// Optimized trim for a single ASCII byte.
/// Uses byte-level scanning instead of char-level iteration.
- fn trim_ascii_char(input: &str, byte: u8) -> (&str, u32);
+ fn trim_ascii_char(input: &str, byte: u8) -> &str;
}
/// Returns the number of leading bytes matching `byte`
@@ -65,19 +64,16 @@ pub(crate) struct TrimLeft;
impl Trimmer for TrimLeft {
#[inline]
- fn trim<'a>(input: &'a str, pattern: &[char]) -> (&'a str, u32) {
+ fn trim<'a>(input: &'a str, pattern: &[char]) -> &'a str {
if pattern.len() == 1 && pattern[0].is_ascii() {
return Self::trim_ascii_char(input, pattern[0] as u8);
}
- let trimmed = input.trim_start_matches(pattern);
- let offset = (input.len() - trimmed.len()) as u32;
- (trimmed, offset)
+ input.trim_start_matches(pattern)
}
#[inline]
- fn trim_ascii_char(input: &str, byte: u8) -> (&str, u32) {
- let start = leading_bytes(input.as_bytes(), byte);
- (&input[start..], start as u32)
+ fn trim_ascii_char(input: &str, byte: u8) -> &str {
+ &input[leading_bytes(input.as_bytes(), byte)..]
}
}
@@ -86,19 +82,17 @@ pub(crate) struct TrimRight;
impl Trimmer for TrimRight {
#[inline]
- fn trim<'a>(input: &'a str, pattern: &[char]) -> (&'a str, u32) {
+ fn trim<'a>(input: &'a str, pattern: &[char]) -> &'a str {
if pattern.len() == 1 && pattern[0].is_ascii() {
return Self::trim_ascii_char(input, pattern[0] as u8);
}
- let trimmed = input.trim_end_matches(pattern);
- (trimmed, 0)
+ input.trim_end_matches(pattern)
}
#[inline]
- fn trim_ascii_char(input: &str, byte: u8) -> (&str, u32) {
+ fn trim_ascii_char(input: &str, byte: u8) -> &str {
let bytes = input.as_bytes();
- let end = bytes.len() - trailing_bytes(bytes, byte);
- (&input[..end], 0)
+ &input[..bytes.len() - trailing_bytes(bytes, byte)]
}
}
@@ -107,22 +101,19 @@ pub(crate) struct TrimBoth;
impl Trimmer for TrimBoth {
#[inline]
- fn trim<'a>(input: &'a str, pattern: &[char]) -> (&'a str, u32) {
+ fn trim<'a>(input: &'a str, pattern: &[char]) -> &'a str {
if pattern.len() == 1 && pattern[0].is_ascii() {
return Self::trim_ascii_char(input, pattern[0] as u8);
}
- let left_trimmed = input.trim_start_matches(pattern);
- let offset = (input.len() - left_trimmed.len()) as u32;
- let trimmed = left_trimmed.trim_end_matches(pattern);
- (trimmed, offset)
+ input.trim_matches(pattern)
}
#[inline]
- fn trim_ascii_char(input: &str, byte: u8) -> (&str, u32) {
+ fn trim_ascii_char(input: &str, byte: u8) -> &str {
let bytes = input.as_bytes();
let start = leading_bytes(bytes, byte);
let end = bytes.len() - trailing_bytes(&bytes[start..], byte);
- (&input[start..end], start as u32)
+ &input[start..end]
}
}
@@ -155,8 +146,8 @@ fn string_view_trim<Tr: Trimmer>(args: &[ArrayRef]) -> Result<ArrayRef> {
.zip(string_view_array.views().iter())
{
if let Some(src_str) = src_str_opt {
- let (trimmed, offset) = Tr::trim_ascii_char(src_str, b' ');
- append_view(&mut views_buf, raw_view, trimmed, offset);
+ let trimmed = Tr::trim_ascii_char(src_str, b' ');
+ views_buf.push(substr_view(*raw_view, src_str, trimmed));
null_builder.append_non_null();
} else {
null_builder.append_null();
@@ -202,8 +193,8 @@ fn string_view_trim<Tr: Trimmer>(args: &[ArrayRef]) -> Result<ArrayRef> {
{
pattern.clear();
pattern.extend(characters.chars());
- let (trimmed, offset) = Tr::trim(src_str, &pattern);
- append_view(&mut views_buf, raw_view, trimmed, offset);
+ let trimmed = Tr::trim(src_str, &pattern);
+ views_buf.push(substr_view(*raw_view, src_str, trimmed));
null_builder.append_non_null();
} else {
null_builder.append_null();
@@ -254,8 +245,8 @@ fn trim_and_append_view<Tr: Trimmer>(
original_view: &u128,
) {
if let Some(src_str) = src_str_opt {
- let (trimmed, offset) = Tr::trim(src_str, pattern);
- append_view(views_buf, original_view, trimmed, offset);
+ let trimmed = Tr::trim(src_str, pattern);
+ views_buf.push(substr_view(*original_view, src_str, trimmed));
null_builder.append_non_null();
} else {
null_builder.append_null();
@@ -332,7 +323,7 @@ fn string_trim<T: OffsetSizeTrait, Tr: Trimmer>(args: &[ArrayRef]) -> Result<Arr
Ok(build_trimmed(
string_array,
string_array.nulls().cloned(),
- |_, s| Tr::trim_ascii_char(s, b' ').0,
+ |_, s| Tr::trim_ascii_char(s, b' '),
))
}
2 => {
@@ -351,7 +342,7 @@ fn string_trim<T: OffsetSizeTrait, Tr: Trimmer>(args: &[ArrayRef]) -> Result<Arr
return Ok(build_trimmed(
string_array,
string_array.nulls().cloned(),
- |_, s| Tr::trim(s, &pattern).0,
+ |_, s| Tr::trim(s, &pattern),
));
}
@@ -371,7 +362,7 @@ fn string_trim<T: OffsetSizeTrait, Tr: Trimmer>(args: &[ArrayRef]) -> Result<Arr
Ok(build_trimmed(string_array, nulls, |i, s| {
pattern.clear();
pattern.extend(characters_array.value(i).chars());
- Tr::trim(s, &pattern).0
+ Tr::trim(s, &pattern)
}))
}
other => {
diff --git a/datafusion/functions/src/string/split_part.rs b/datafusion/functions/src/string/split_part.rs
index 401f248a24..9293574d16 100644
--- a/datafusion/functions/src/string/split_part.rs
+++ b/datafusion/functions/src/string/split_part.rs
@@ -17,11 +17,12 @@
use crate::strings::{
BulkNullStringArrayBuilder, GenericStringArrayBuilder, StringViewArrayBuilder,
+ substr_view,
};
use crate::utils::utf8_to_str_type;
use arrow::array::{
- Array, ArrayRef, AsArray, ByteView, Int64Array, StringArrayType, StringViewArray,
- make_view, new_null_array,
+ Array, ArrayRef, AsArray, Int64Array, StringArrayType, StringViewArray, make_view,
+ new_null_array,
};
use arrow::buffer::{NullBuffer, ScalarBuffer};
use arrow::datatypes::DataType;
@@ -503,22 +504,6 @@ fn split_part_scalar_view(
}
}
-/// Creates a `StringView` referencing a substring of an existing view's buffer.
-/// For substrings ≤ 12 bytes, creates an inline view instead.
-#[inline]
-fn substr_view(original_view: &u128, substr: &str, start_offset: u32) -> u128 {
- if substr.len() > 12 {
- let view = ByteView::from(*original_view);
- make_view(
- substr.as_bytes(),
- view.buffer_index,
- view.offset + start_offset,
- )
- } else {
- make_view(substr.as_bytes(), 0, 0)
- }
-}
-
/// Applies `split_fn` to each non-null string and appends the resulting view to
/// `views_buf`.
#[inline(always)]
@@ -538,10 +523,7 @@ fn split_view_loop<F>(
}
let string = string_view_array.value(i);
match split_fn(string) {
- Some(substr) => {
- let start_offset = substr.as_ptr() as usize - string.as_ptr() as usize;
- views_buf.push(substr_view(raw_view, substr, start_offset as u32));
- }
+ Some(substr) => views_buf.push(substr_view(*raw_view, string, substr)),
None => views_buf.push(empty_view),
}
}
diff --git a/datafusion/functions/src/strings.rs b/datafusion/functions/src/strings.rs
index 96084474ff..ed99680ec7 100644
--- a/datafusion/functions/src/strings.rs
+++ b/datafusion/functions/src/strings.rs
@@ -1181,44 +1181,6 @@ impl BulkNullStringArrayBuilder for StringViewArrayBuilder {
}
}
-/// Append a new view to the views buffer with the given substr.
-///
-/// Callers are responsible for their own null tracking.
-///
-/// # Safety
-///
-/// original_view must be a valid view (the format described on
-/// [`GenericByteViewArray`](arrow::array::GenericByteViewArray).
-///
-/// # Arguments
-/// - views_buffer: The buffer to append the new view to
-/// - original_view: The original view value
-/// - substr: The substring to append. Must be a valid substring of the original view
-/// - start_offset: The start offset of the substring in the view
-///
-/// LLVM is apparently overly eager to inline this function into some hot loops,
-/// which bloats them and regresses performance, so we disable inlining for now.
-#[inline(never)]
-pub(crate) fn append_view(
- views_buffer: &mut Vec<u128>,
- original_view: &u128,
- substr: &str,
- start_offset: u32,
-) {
- let substr_len = substr.len();
- let sub_view = if substr_len > 12 {
- let view = ByteView::from(*original_view);
- make_view(
- substr.as_bytes(),
- view.buffer_index,
- view.offset + start_offset,
- )
- } else {
- make_view(substr.as_bytes(), 0, 0)
- };
- views_buffer.push(sub_view);
-}
-
/// Values of at most this many bytes are stored inline in their view.
pub(crate) const MAX_INLINE_LEN: usize = MAX_INLINE_VIEW_LEN as usize;
@@ -1239,9 +1201,23 @@ pub(crate) const MAX_INLINE_LEN: usize = MAX_INLINE_VIEW_LEN as usize;
pub(crate) fn sub_view(view: u128, source: &[u8], range: Range<usize>) -> u128 {
debug_assert!(range.start <= range.end && range.end <= source.len());
- // The substring's first bytes, in the low-order bits. Any bits past the
- // end of the substring are masked off by `inline_view`.
- let leading_bytes = if source.len() <= MAX_INLINE_LEN {
+ let len = range.end - range.start;
+ if len > MAX_INLINE_LEN {
+ // The substring has more than 12 bytes, so its 4-byte prefix is in bounds.
+ let prefix = source[range.start..range.start + 4].try_into().unwrap();
+ let original = ByteView::from(view);
+ return ByteView {
+ length: len as u32,
+ prefix: u32::from_le_bytes(prefix),
+ offset: original.offset + range.start as u32,
+ ..original
+ }
+ .as_u128();
+ }
+
+ // The substring's bytes, in the low-order bits. Any bits past the end of
+ // the substring are masked off by `inline_view`.
+ let bytes = if source.len() <= MAX_INLINE_LEN {
// `source` is stored in `view` itself, after its 4-byte length.
(view >> 32) >> (8 * range.start)
} else {
@@ -1254,20 +1230,15 @@ pub(crate) fn sub_view(view: u128, source: &[u8], range: Range<usize>) -> u128 {
.unwrap();
read_12_bytes(window) >> (8 * (range.start - window_start))
};
+ inline_view(bytes, len)
+}
- let len = range.len();
- if len <= MAX_INLINE_LEN {
- inline_view(leading_bytes, len)
- } else {
- let original = ByteView::from(view);
- ByteView {
- length: len as u32,
- prefix: leading_bytes as u32,
- offset: original.offset + range.start as u32,
- ..original
- }
- .as_u128()
- }
+/// Returns the view for `substr`, which must be a slice of `string`, the value
+/// `view` refers to. See [`sub_view`].
+#[inline]
+pub(crate) fn substr_view(view: u128, string: &str, substr: &str) -> u128 {
+ let start = substr.as_ptr() as usize - string.as_ptr() as usize;
+ sub_view(view, string.as_bytes(), start..start + substr.len())
}
/// Reads `bytes` as a little-endian integer.
diff --git a/datafusion/functions/src/unicode/substr.rs b/datafusion/functions/src/unicode/substr.rs
index 53f6080088..55da0e801a 100644
--- a/datafusion/functions/src/unicode/substr.rs
+++ b/datafusion/functions/src/unicode/substr.rs
@@ -17,7 +17,7 @@
use std::sync::Arc;
-use crate::strings::append_view;
+use crate::strings::sub_view;
use crate::utils::make_scalar_function;
use arrow::array::{
Array, ArrayRef, AsArray, GenericStringArray, Int64Array, OffsetSizeTrait,
@@ -297,9 +297,7 @@ fn string_view_substr(
let count = count_array_opt.map(|a| a.value(i));
let (byte_start, byte_end) = get_true_start_end(string, start, count, is_ascii)?;
- let substr = &string[byte_start..byte_end];
-
- append_view(&mut views_buf, raw_view, substr, byte_start as u32);
+ views_buf.push(sub_view(*raw_view, string.as_bytes(), byte_start..byte_end));
}
let views_buf = ScalarBuffer::from(views_buf);
diff --git a/datafusion/functions/src/unicode/substrindex.rs b/datafusion/functions/src/unicode/substrindex.rs
index 33da6e8fc2..ada3141a9b 100644
--- a/datafusion/functions/src/unicode/substrindex.rs
+++ b/datafusion/functions/src/unicode/substrindex.rs
@@ -18,14 +18,14 @@
use std::sync::Arc;
use arrow::array::{
- Array, ArrayRef, AsArray, ByteView, OffsetSizeTrait, PrimitiveArray, StringArrayType,
+ Array, ArrayRef, AsArray, OffsetSizeTrait, PrimitiveArray, StringArrayType,
StringViewArray, make_view, new_null_array,
};
use arrow::buffer::ScalarBuffer;
use arrow::datatypes::{DataType, Int64Type};
use arrow_buffer::NullBuffer;
-use crate::strings::GenericStringArrayBuilder;
+use crate::strings::{GenericStringArrayBuilder, MAX_INLINE_LEN, substr_view};
use crate::utils::make_scalar_function;
use datafusion_common::utils::offset_span_len;
use datafusion_common::{
@@ -324,7 +324,7 @@ fn substr_index_view(
// Safety: each appended view is either:
// (1) a copied null sentinel,
// (2) the original valid input view, or
- // (3) built by `append_view` for a contiguous substring of the input row.
+ // (3) built by `substr_view` for a contiguous substring of the input row.
unsafe {
Ok(Arc::new(StringViewArray::new_unchecked(
ScalarBuffer::from(views_buf),
@@ -457,7 +457,7 @@ fn substr_index_scalar_view(
// (1) a copied null sentinel,
// (2) the original valid input view,
// (3) an inline empty string view, or
- // (4) built by `append_view` for a contiguous substring of the input row.
+ // (4) built by `substr_view` for a contiguous substring of the input row.
unsafe {
Ok(Arc::new(StringViewArray::new_unchecked(
ScalarBuffer::from(views_buf),
@@ -579,20 +579,6 @@ fn substr_index_rslice_finder<'a>(
}
}
-#[inline]
-fn substr_view(original_view: &u128, substr: &str, start_offset: u32) -> u128 {
- if substr.len() > 12 {
- let view = ByteView::from(*original_view);
- make_view(
- substr.as_bytes(),
- view.buffer_index,
- view.offset + start_offset,
- )
- } else {
- make_view(substr.as_bytes(), 0, 0)
- }
-}
-
#[inline]
fn append_substr_view(
views_buf: &mut Vec<u128>,
@@ -602,7 +588,7 @@ fn append_substr_view(
) -> bool {
if substr.len() == string.len() {
views_buf.push(*raw_view);
- return substr.len() > 12;
+ return substr.len() > MAX_INLINE_LEN;
}
if substr.is_empty() {
@@ -610,11 +596,8 @@ fn append_substr_view(
return false;
}
- let start_offset = substr.as_ptr() as usize - string.as_ptr() as usize;
- let start_offset =
- u32::try_from(start_offset).expect("string view offsets fit in u32");
- views_buf.push(substr_view(raw_view, substr, start_offset));
- substr.len() > 12
+ views_buf.push(substr_view(*raw_view, string, substr));
+ substr.len() > MAX_INLINE_LEN
}
#[cfg(test)]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org