You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/08/01 17:56:50 UTC
[PR] bench(parquet): cover DELTA_BYTE_ARRAY large values in the arrow_writer benchmark [arrow-rs]
adriangb opened a new pull request, #10512:
URL: https://github.com/apache/arrow-rs/pull/10512
# Which issue does this PR close?
- Part of #10511
# Rationale for this change
No existing benchmark writes `DELTA_BYTE_ARRAY` through the writer: the `arrow_writer` property matrix varies writer version, compression, bloom filters, and CDC, all on the default encoding, and `encoding.rs` benches encoders in isolation.
That gap matters for the large-value write path specifically: the byte-budget sub-batching in `write_batch_internal` (#9972) measures raw payload bytes, and page-boundary behavior interacts with the encoding's cross-value state (#10489, #10505). None of that is visible to any current benchmark. #10511 tracks whether making the byte budget encoded-size-aware is worthwhile; these benchmarks are the measurement for that question.
# What changes are included in this PR?
Two new batches and one new bench group in `parquet/benches/arrow_writer.rs`:
- `large_string_shared_prefix`: 128 rows x 2 MiB, long common prefix with a short distinct suffix (the case `DELTA_BYTE_ARRAY` exists for)
- `large_string_distinct`: same shape, values differing from byte 0 (the adversarial case, prefix length ~0)
Values are sized so one value alone exceeds the default 1 MiB page limit, the regime of #10489. Each batch runs under `plain` and `delta_byte_array` properties (dictionary disabled), so the delta-vs-plain gap on identical data separates inherent encoding cost from writer overhead.
Results on an Apple M-series laptop, current `main`:
| group | `plain` | `delta_byte_array` |
| --- | --- | --- |
| `large_string_shared_prefix` | 59.0 ms (4.2 GiB/s) | 61.4 ms (4.1 GiB/s) |
| `large_string_distinct` | 37.2 ms (6.7 GiB/s) | 44.7 ms (5.6 GiB/s) |
Note these numbers shift when #10505 lands, in both directions: shared-prefix delta gets slower in CPU time because the encoder starts doing real prefix comparisons instead of degenerating to per-page `PLAIN` (while the output shrinks ~128x), and distinct delta gets faster from halving the page count. That sensitivity is the point of having the benchmark.
# Are these changes tested?
The change is itself a benchmark; it compiles and runs under `cargo bench -p parquet --bench arrow_writer`.
# Are there any user-facing changes?
No.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): cover DELTA_BYTE_ARRAY large values in the arrow_writer benchmark [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10512:
URL: https://github.com/apache/arrow-rs/pull/10512#issuecomment-5166878654
Thanks for the review @alamb. Addressed your comment suggestions. Would appreciate a merge so I can rebase https://github.com/apache/arrow-rs/pull/10505 and run these benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): cover DELTA_BYTE_ARRAY large values in the arrow_writer benchmark [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #10512:
URL: https://github.com/apache/arrow-rs/pull/10512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): cover DELTA_BYTE_ARRAY large values in the arrow_writer benchmark [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10512:
URL: https://github.com/apache/arrow-rs/pull/10512#discussion_r3703309989
##########
parquet/benches/arrow_writer.rs:
##########
@@ -651,5 +676,55 @@ fn bench_all_writers(c: &mut Criterion) {
}
}
-criterion_group!(benches, bench_all_writers);
+/// Large-value BYTE_ARRAY columns under `DELTA_BYTE_ARRAY`, which the
+/// property matrix above never exercises (it only varies writer version,
+/// compression, bloom filters, and CDC — all on the default encoding).
+///
+/// Values are sized so a single value exceeds the default 1 MiB data page
+/// limit, putting the writer in the byte-budget sub-batching regime of
+/// `write_batch_internal`. That budget measures raw payload bytes, so under
+/// `DELTA_BYTE_ARRAY` it sub-batches by the values' *pre-dedup* size however
+/// well they compress; benchmarking against `PLAIN` on the same data bounds
+/// what an encoded-size-aware budget could recover. The shared-prefix and
+/// distinct batches bracket the encoding's best and worst case.
+fn bench_delta_byte_array_writers(c: &mut Criterion) {
+ // 128 rows × 2 MiB: the same total bytes as `large_string_non_null`
+ // above, but with each value alone exceeding the default 1 MiB page
Review Comment:
I think these comments will go stale as soon as this PR is merged (this are basically comments for the PR, not the code)
##########
parquet/benches/arrow_writer.rs:
##########
@@ -651,5 +676,55 @@ fn bench_all_writers(c: &mut Criterion) {
}
}
-criterion_group!(benches, bench_all_writers);
+/// Large-value BYTE_ARRAY columns under `DELTA_BYTE_ARRAY`, which the
+/// property matrix above never exercises (it only varies writer version,
+/// compression, bloom filters, and CDC — all on the default encoding).
+///
+/// Values are sized so a single value exceeds the default 1 MiB data page
+/// limit, putting the writer in the byte-budget sub-batching regime of
Review Comment:
this is talking about the internal implementation of `write_batch_internal` which is not in this meethog -- I think it would be clearer if it talked about the properties of the data instead.
For example, perhaps give some examples of the data strings being compressed (`xxxx.....001`, etc) and note that the shared prefix case is the case that DLBA is designed to handle well
##########
parquet/benches/arrow_writer.rs:
##########
@@ -651,5 +676,55 @@ fn bench_all_writers(c: &mut Criterion) {
}
}
-criterion_group!(benches, bench_all_writers);
+/// Large-value BYTE_ARRAY columns under `DELTA_BYTE_ARRAY`, which the
+/// property matrix above never exercises (it only varies writer version,
Review Comment:
referring to the current state of the above matrix may get stale eventually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): cover DELTA_BYTE_ARRAY large values in the arrow_writer benchmark [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10512:
URL: https://github.com/apache/arrow-rs/pull/10512#issuecomment-5168625973
Thanks @adriangb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org