You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/07/01 22:07:35 UTC
[PR] Add validated row decode benchmark [arrow-rs]
alamb opened a new pull request, #10259:
URL: https://github.com/apache/arrow-rs/pull/10259
# Which issue does this PR close?
- Closes #NNN.
# Rationale for this change
The existing row format benchmark measures `convert_rows` using rows created directly by `RowConverter::convert_columns`, which skips UTF-8 validation during decode.
# What changes are included in this PR?
This adds a `convert_rows_validated` benchmark that decodes rows parsed through `RowParser`, exercising the UTF-8 validation path. It also derives `Clone` for `Rows` so the benchmark setup can reuse the prepared rows when converting to binary.
# Are these changes tested?
CI.
# Are there any user-facing changes?
No breaking changes. `Rows` now implements `Clone`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add validated row decode benchmark [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10259:
URL: https://github.com/apache/arrow-rs/pull/10259#discussion_r3509725836
##########
arrow/benches/row_format.rs:
##########
@@ -61,6 +61,25 @@ fn do_bench(c: &mut Criterion, name: &str, cols: Vec<ArrayRef>) {
b.iter(|| hint::black_box(converter.convert_rows(&rows).unwrap()));
});
+ let binary_rows = rows.clone().try_into_binary().expect("reasonable size");
+ let parser = converter.parser();
+ // RowParser marks rows as requiring UTF-8 validation when they are decoded
+ // back into Arrow arrays by RowConverter::convert_rows.
+ let parsed_rows: Vec<_> = binary_rows
+ .iter()
+ .flatten()
Review Comment:
does this mean nulls are ignored in this bench?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add validated row decode benchmark [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10259:
URL: https://github.com/apache/arrow-rs/pull/10259#discussion_r3509344472
##########
arrow/benches/row_format.rs:
##########
@@ -61,6 +61,25 @@ fn do_bench(c: &mut Criterion, name: &str, cols: Vec<ArrayRef>) {
b.iter(|| hint::black_box(converter.convert_rows(&rows).unwrap()));
});
+ let binary_rows = rows.clone().try_into_binary().expect("reasonable size");
+ let parser = converter.parser();
+ // RowParser marks rows as requiring UTF-8 validation when they are decoded
+ // back into Arrow arrays by RowConverter::convert_rows.
+ let parsed_rows: Vec<_> = binary_rows
+ .iter()
+ .flatten()
+ .map(|row| parser.parse(row).owned())
+ .collect();
+ c.bench_function(&format!("convert_rows_validated {name}"), |b| {
+ b.iter(|| {
+ hint::black_box(
Review Comment:
I profiled these benchmarks locally and the profile looks reasonable to me (it is validating the parsed rows)
<img width="1394" height="968" alt="Image" src="https://github.com/user-attachments/assets/0673be15-9acb-4ba4-bdf5-0dfd86ffea20" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org