You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "neilconway (via GitHub)" <gi...@apache.org> on 2026/10/06 15:09:50 UTC
[PR] perf(parquet): optimize level counting [arrow-rs]
neilconway opened a new pull request, #11408:
URL: https://github.com/apache/arrow-rs/pull/11408
# Which issue does this PR close?
- N/A
# Rationale for this change
This PR makes several related optimizations when decoding definition and repetition levels:
* `DefinitionLevelDecoderImpl::read_def_levels` counts how many definition levels have the maximum value. That was previously done with `filter(|x| **x == self.max_level).count()`, which does not vectorize well: levels are `i16` but `count` returns `usize`, so we can look at many levels at once (e.g., 8 with ARM NEON registers), but we then need to widen those comparison results to sum them into 64-bit `usize` counters. It is more efficient to sum into `u16` counters first and then aggregate the partial sums. We count chunks of <= `u16::MAX` levels at a time, to avoid overflowing the `u16` counters. We introduce a helper `count_levels_eq` that does this.
* `RepetitionLevelDecoderImpl::count_records` looks for 0s in the repetition levels to identify when new records start. That was previously done one level at a time, stopping once the requested # of records has been reached. That means the loop can stop at any level, which is hard to vectorize. Instead, use `count_levels_eq` to count 0s in chunks of 32 levels at a time; once we reach or exceed the requested # of records, we go back and precisely count the last chunk.
* `DefinitionLevelBufferDecoder::read_def_levels` reads the definition levels of a nested column and builds a null mask with one bit per level. When every level is at the maximum (no nulls, empty lists, or null lists), we can append an all-valid run instead of building the null-mask bit-by-bit.
`arrow_reader` benchmarks, Apple M4 Max / AMD EPYC (x86):
- `ListArray` cases: -12% to -29% / -12% to -26%
- `Int32List`, skip and read in runs of 32: -24% / -24%
- `Int32List`, skip and read in runs of 1: -3% to +4% / -2% to +1%
- optional struct with optional data: -51% / -70% with no NULLs, -10% / -10% with half NULLs
# What changes are included in this PR?
See above.
# Are these changes tested?
Yes; existing tests pass, new tests added.
# Are there any significant user-facing changes?
No functional changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#issuecomment-6081640625
Thanks for the reviews! @etseidl @jhorstmann
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#issuecomment-6023280008
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11408#issuecomment-6021480575)
**Target:** arrow_reader
Comparing neilc/parquet-count-levels (63d9cad5b144795dad5e3793a800253316d9fc3f) to d18ea16 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/d18ea16f6fc9af0ba7e8ea76f6934c6e6cb57a26..63d9cad5b144795dad5e3793a800253316d9fc3f)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_reader
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 835.8±9.78µs ? ?/sec 1.00 834.5±10.13µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 949.6±6.31µs ? ?/sec 1.01 954.5±5.07µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 842.2±9.55µs ? ?/sec 1.01 847.5±12.83µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs 1.00 284.9±0.20µs ? ?/sec 1.00 285.8±0.18µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs, large values 1.05 44.5±0.18ms ? ?/sec 1.00 42.3±0.39ms ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, half NULLs 1.00 392.4±0.59µs ? ?/sec 1.01 397.8±0.79µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, no NULLs 1.00 288.8±0.20µs ? ?/sec 1.00 289.0±0.17µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, mandatory, no NULLs 1.00 401.6±5.45µs ? ?/sec 1.02 408.3±5.54µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, half NULLs 1.00 464.3±2.62µs ? ?/sec 1.02 473.1±3.17µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, no NULLs 1.00 399.1±4.95µs ? ?/sec 1.03 411.4±4.37µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, mandatory, no NULLs 1.00 80.3±0.04µs ? ?/sec 1.00 80.7±0.05µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, half NULLs 1.00 111.5±0.12µs ? ?/sec 1.00 111.6±0.09µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, no NULLs 1.00 84.2±0.10µs ? ?/sec 1.00 84.4±0.06µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs 1.07 157.8±1.19µs ? ?/sec 1.00 147.4±0.81µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs, short string 1.00 142.5±0.38µs ? ?/sec 1.00 142.7±0.64µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, half NULLs 1.00 151.3±1.86µs ? ?/sec 1.05 159.3±3.20µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, no NULLs 1.09 165.9±3.86µs ? ?/sec 1.00 152.4±0.63µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, mandatory, no NULLs 1.00 211.0±1.67µs ? ?/sec 1.01 213.3±2.31µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, optional, half NULLs 1.00 230.1±2.70µs ? ?/sec 1.00 229.2±1.90µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, optional, no NULLs 1.00 216.0±1.74µs ? ?/sec 1.01 218.8±1.96µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 968.5±0.74µs ? ?/sec 1.06 1024.1±36.27µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 673.2±0.41µs ? ?/sec 1.00 673.2±1.98µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 972.3±0.54µs ? ?/sec 1.00 973.7±3.26µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 188.5±0.24µs ? ?/sec 1.01 189.7±0.63µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 294.8±0.23µs ? ?/sec 1.00 294.8±1.03µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 193.5±0.20µs ? ?/sec 1.02 196.5±0.95µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 129.3±0.21µs ? ?/sec 1.01 130.8±0.39µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, half NULLs 1.00 181.9±0.27µs ? ?/sec 1.01 183.6±0.23µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, no NULLs 1.00 131.4±0.43µs ? ?/sec 1.01 133.4±0.30µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, mandatory, no NULLs 1.00 60.2±0.05µs ? ?/sec 1.01 60.7±0.10µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, half NULLs 1.00 146.9±0.08µs ? ?/sec 1.01 148.1±0.51µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, no NULLs 1.00 63.0±0.04µs ? ?/sec 1.01 63.5±0.19µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, mandatory, no NULLs 1.00 803.9±0.69µs ? ?/sec 1.00 803.7±1.49µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, half NULLs 1.00 512.5±0.34µs ? ?/sec 1.00 510.1±0.49µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, no NULLs 1.00 808.2±0.54µs ? ?/sec 1.00 808.9±2.20µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, mandatory, no NULLs 1.03 23.3±0.05µs ? ?/sec 1.00 22.7±0.02µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, half NULLs 1.00 131.7±0.16µs ? ?/sec 1.00 131.6±0.41µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, no NULLs 1.00 27.0±0.05µs ? ?/sec 1.00 27.1±1.45µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, mandatory, no NULLs 1.00 74.9±0.75µs ? ?/sec 1.00 75.1±0.21µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, half NULLs 1.00 128.9±0.15µs ? ?/sec 1.00 129.1±0.26µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, no NULLs 1.00 77.7±1.20µs ? ?/sec 1.02 79.3±0.28µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, mandatory, no NULLs 1.00 5.5±0.02µs ? ?/sec 1.00 5.5±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, half NULLs 1.00 93.5±0.20µs ? ?/sec 1.00 93.7±0.09µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, no NULLs 1.02 8.9±0.04µs ? ?/sec 1.00 8.8±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, mandatory, no NULLs 1.08 160.7±25.24µs ? ?/sec 1.00 148.7±0.51µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, half NULLs 1.00 199.7±0.22µs ? ?/sec 1.00 199.4±0.29µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, no NULLs 1.00 151.0±0.54µs ? ?/sec 1.02 153.8±0.36µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, mandatory, no NULLs 1.00 7.5±0.05µs ? ?/sec 1.02 7.6±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, half NULLs 1.00 128.5±0.10µs ? ?/sec 1.00 128.2±0.11µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, no NULLs 1.00 11.1±0.04µs ? ?/sec 1.00 11.0±0.02µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, mandatory, no NULLs 1.11 330.5±44.37µs ? ?/sec 1.00 298.4±1.13µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, half NULLs 1.00 268.6±0.56µs ? ?/sec 1.01 270.2±0.37µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, no NULLs 1.00 303.9±0.95µs ? ?/sec 1.02 309.1±0.71µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, mandatory, no NULLs 1.00 12.0±0.03µs ? ?/sec 1.04 12.5±0.02µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, half NULLs 1.00 124.7±0.12µs ? ?/sec 1.00 124.1±0.11µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, no NULLs 1.01 16.3±0.05µs ? ?/sec 1.00 16.1±0.01µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed increasing value 1.00 88.0±0.50µs ? ?/sec 1.00 88.1±0.73µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed single value 1.00 79.8±0.68µs ? ?/sec 1.01 80.8±0.45µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip increasing value 1.00 50.7±0.29µs ? ?/sec 1.00 50.7±0.37µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip single value 1.00 46.3±0.40µs ? ?/sec 1.01 46.6±0.20µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip stepped increasing value 1.01 75.9±0.24µs ? ?/sec 1.00 75.3±0.23µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 89.8±0.88µs ? ?/sec 1.00 89.9±0.94µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, half NULLs 1.01 89.1±0.46µs ? ?/sec 1.00 87.9±0.44µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, no NULLs 1.00 91.7±0.89µs ? ?/sec 1.00 91.8±0.90µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed stepped increasing value 1.00 108.7±0.14µs ? ?/sec 1.01 109.9±1.28µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, mandatory, no NULLs 1.00 126.9±1.12µs ? ?/sec 1.00 127.2±0.98µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, half NULLs 1.01 146.2±0.39µs ? ?/sec 1.00 144.9±0.52µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, no NULLs 1.00 130.4±0.83µs ? ?/sec 1.00 130.1±0.91µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.02 55.1±0.04µs ? ?/sec 1.00 54.2±0.05µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.01 109.0±0.17µs ? ?/sec 1.00 107.5±0.19µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 57.6±0.03µs ? ?/sec 1.00 57.7±0.05µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 84.8±0.06µs ? ?/sec 1.00 84.7±0.06µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, half NULLs 1.01 126.9±0.18µs ? ?/sec 1.00 125.6±0.32µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 87.7±0.06µs ? ?/sec 1.01 88.8±0.13µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, mandatory, no NULLs 1.01 48.4±0.05µs ? ?/sec 1.00 47.9±0.08µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, half NULLs 1.01 106.0±0.17µs ? ?/sec 1.00 104.8±0.20µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, no NULLs 1.00 50.5±0.07µs ? ?/sec 1.00 50.7±0.06µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed increasing value 1.00 79.0±0.10µs ? ?/sec 1.00 78.7±0.13µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed single value 1.00 74.9±0.09µs ? ?/sec 1.00 74.9±0.13µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip increasing value 1.01 43.9±0.04µs ? ?/sec 1.00 43.5±0.12µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip single value 1.00 41.7±0.05µs ? ?/sec 1.00 41.8±0.23µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip stepped increasing value 1.00 68.0±0.08µs ? ?/sec 1.00 68.3±0.05µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 79.6±0.06µs ? ?/sec 1.00 79.7±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, half NULLs 1.00 83.3±0.11µs ? ?/sec 1.01 83.7±0.11µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, no NULLs 1.00 81.4±0.08µs ? ?/sec 1.00 81.5±0.05µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed stepped increasing value 1.00 100.6±0.14µs ? ?/sec 1.00 101.0±0.17µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, mandatory, no NULLs 1.00 115.4±0.54µs ? ?/sec 1.00 115.3±0.25µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, half NULLs 1.00 139.9±0.20µs ? ?/sec 1.01 141.1±0.29µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, no NULLs 1.00 118.2±0.20µs ? ?/sec 1.01 118.9±0.18µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 83.3±0.06µs ? ?/sec 1.00 83.2±0.24µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 124.7±0.22µs ? ?/sec 1.00 124.7±0.40µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 87.2±0.16µs ? ?/sec 1.01 87.8±0.23µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 86.8±0.06µs ? ?/sec 1.00 87.1±0.12µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, half NULLs 1.00 127.5±0.19µs ? ?/sec 1.01 128.5±0.22µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 90.3±0.13µs ? ?/sec 1.01 90.8±0.34µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 54.3±0.07µs ? ?/sec 1.00 54.2±0.20µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, half NULLs 1.00 109.6±0.16µs ? ?/sec 1.00 109.6±0.39µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, no NULLs 1.00 58.1±0.04µs ? ?/sec 1.00 58.2±0.09µs ? ?/sec
arrow_array_reader/Int16Array/binary packed increasing value 1.00 54.2±0.71µs ? ?/sec 1.02 55.3±0.52µs ? ?/sec
arrow_array_reader/Int16Array/binary packed single value 1.01 46.7±0.29µs ? ?/sec 1.00 46.3±0.56µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip increasing value 1.00 33.5±0.36µs ? ?/sec 1.01 33.8±0.35µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip single value 1.00 29.1±0.19µs ? ?/sec 1.00 28.9±0.30µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip stepped increasing value 1.01 58.5±0.25µs ? ?/sec 1.00 58.2±0.22µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, mandatory, no NULLs 1.00 72.8±0.92µs ? ?/sec 1.00 72.6±1.01µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, half NULLs 1.02 72.0±0.46µs ? ?/sec 1.00 70.5±0.53µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, no NULLs 1.00 74.6±0.97µs ? ?/sec 1.00 74.3±0.97µs ? ?/sec
arrow_array_reader/Int16Array/binary packed stepped increasing value 1.00 75.4±0.16µs ? ?/sec 1.00 75.6±0.15µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, mandatory, no NULLs 1.00 92.4±0.30µs ? ?/sec 1.00 92.7±0.35µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, half NULLs 1.01 112.4±0.28µs ? ?/sec 1.00 110.8±0.26µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, no NULLs 1.00 96.0±0.37µs ? ?/sec 1.00 95.9±0.31µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 21.0±0.02µs ? ?/sec 1.00 21.1±0.02µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, half NULLs 1.01 75.5±0.16µs ? ?/sec 1.00 74.5±0.16µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, no NULLs 1.00 24.4±0.07µs ? ?/sec 1.00 24.5±0.02µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, mandatory, no NULLs 1.00 51.4±0.04µs ? ?/sec 1.01 51.6±0.02µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, half NULLs 1.01 93.6±0.20µs ? ?/sec 1.00 92.3±0.19µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, no NULLs 1.00 55.0±0.07µs ? ?/sec 1.00 55.1±0.05µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, mandatory, no NULLs 1.01 14.0±0.16µs ? ?/sec 1.00 13.9±0.03µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, half NULLs 1.02 73.4±0.18µs ? ?/sec 1.00 72.3±0.14µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, no NULLs 1.00 17.4±0.08µs ? ?/sec 1.01 17.6±0.02µs ? ?/sec
arrow_array_reader/Int32Array/binary packed increasing value 1.00 51.0±0.72µs ? ?/sec 1.00 51.1±0.65µs ? ?/sec
arrow_array_reader/Int32Array/binary packed single value 1.00 43.0±0.33µs ? ?/sec 1.03 44.2±0.40µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip increasing value 1.00 31.6±0.32µs ? ?/sec 1.00 31.6±0.35µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip single value 1.00 27.3±0.16µs ? ?/sec 1.01 27.6±0.23µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip stepped increasing value 1.00 56.2±0.22µs ? ?/sec 1.00 56.4±0.22µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, mandatory, no NULLs 1.00 71.1±0.95µs ? ?/sec 1.00 70.9±0.95µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, half NULLs 1.01 70.4±0.45µs ? ?/sec 1.00 69.3±0.56µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, no NULLs 1.00 73.1±0.96µs ? ?/sec 1.00 73.0±0.93µs ? ?/sec
arrow_array_reader/Int32Array/binary packed stepped increasing value 1.00 71.5±0.13µs ? ?/sec 1.00 71.8±0.16µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, mandatory, no NULLs 1.00 89.4±0.84µs ? ?/sec 1.00 89.6±0.82µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, half NULLs 1.01 109.8±0.48µs ? ?/sec 1.00 108.4±0.51µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, no NULLs 1.00 93.0±0.85µs ? ?/sec 1.00 93.5±0.80µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, mandatory, no NULLs 1.00 17.2±0.02µs ? ?/sec 1.01 17.4±0.02µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, half NULLs 1.02 72.9±0.14µs ? ?/sec 1.00 71.4±0.17µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, no NULLs 1.00 20.6±0.06µs ? ?/sec 1.01 20.9±0.01µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, mandatory, no NULLs 1.00 47.5±0.05µs ? ?/sec 1.01 48.1±0.02µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, half NULLs 1.01 90.2±0.28µs ? ?/sec 1.00 89.2±0.24µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, no NULLs 1.00 51.1±0.08µs ? ?/sec 1.01 51.5±0.04µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, mandatory, no NULLs 1.00 10.0±0.05µs ? ?/sec 1.00 9.9±0.02µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, half NULLs 1.02 69.4±0.18µs ? ?/sec 1.00 68.2±0.15µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, no NULLs 1.00 13.6±0.06µs ? ?/sec 1.00 13.6±0.02µs ? ?/sec
arrow_array_reader/Int64Array/binary packed increasing value 1.00 41.6±0.20µs ? ?/sec 1.00 41.8±0.13µs ? ?/sec
arrow_array_reader/Int64Array/binary packed single value 1.00 38.2±0.06µs ? ?/sec 1.01 38.5±0.09µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip increasing value 1.00 24.7±0.08µs ? ?/sec 1.00 24.7±0.06µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip single value 1.00 22.6±0.04µs ? ?/sec 1.01 22.8±0.05µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip stepped increasing value 1.00 49.0±0.07µs ? ?/sec 1.00 49.2±0.07µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, mandatory, no NULLs 1.00 60.7±0.05µs ? ?/sec 1.00 60.8±0.07µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, half NULLs 1.00 64.6±0.12µs ? ?/sec 1.01 65.1±0.12µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, no NULLs 1.00 62.4±0.06µs ? ?/sec 1.00 62.7±0.05µs ? ?/sec
arrow_array_reader/Int64Array/binary packed stepped increasing value 1.00 63.5±0.15µs ? ?/sec 1.01 64.2±0.08µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, mandatory, no NULLs 1.00 77.6±0.15µs ? ?/sec 1.01 78.0±0.14µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, half NULLs 1.00 104.2±0.28µs ? ?/sec 1.00 104.4±0.23µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, no NULLs 1.00 81.9±0.14µs ? ?/sec 1.00 82.0±0.13µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, mandatory, no NULLs 1.01 46.2±0.07µs ? ?/sec 1.00 45.9±0.05µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, half NULLs 1.00 87.5±0.25µs ? ?/sec 1.00 87.5±0.18µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, no NULLs 1.01 50.6±0.07µs ? ?/sec 1.00 50.1±0.04µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, mandatory, no NULLs 1.00 49.7±0.08µs ? ?/sec 1.01 50.1±0.02µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, half NULLs 1.00 90.4±0.26µs ? ?/sec 1.01 90.9±0.17µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, no NULLs 1.00 53.5±0.06µs ? ?/sec 1.00 53.6±0.05µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, mandatory, no NULLs 1.00 16.1±0.02µs ? ?/sec 1.00 16.2±0.03µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, half NULLs 1.00 72.8±0.23µs ? ?/sec 1.01 73.5±0.17µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, no NULLs 1.00 20.0±0.06µs ? ?/sec 1.01 20.2±0.04µs ? ?/sec
arrow_array_reader/Int8Array/binary packed increasing value 1.00 54.4±0.77µs ? ?/sec 1.03 55.8±0.54µs ? ?/sec
arrow_array_reader/Int8Array/binary packed single value 1.00 46.5±0.32µs ? ?/sec 1.00 46.5±0.68µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip increasing value 1.00 33.5±0.35µs ? ?/sec 1.02 34.1±0.35µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip single value 1.00 29.0±0.19µs ? ?/sec 1.00 29.0±0.43µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip stepped increasing value 1.00 58.1±0.24µs ? ?/sec 1.01 58.4±0.21µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, mandatory, no NULLs 1.00 72.3±0.76µs ? ?/sec 1.00 72.4±0.46µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, half NULLs 1.01 71.5±0.38µs ? ?/sec 1.00 70.7±0.31µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, no NULLs 1.00 74.4±0.75µs ? ?/sec 1.00 74.3±0.48µs ? ?/sec
arrow_array_reader/Int8Array/binary packed stepped increasing value 1.00 75.2±0.15µs ? ?/sec 1.01 75.9±0.16µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, mandatory, no NULLs 1.00 92.2±0.23µs ? ?/sec 1.00 92.3±0.13µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, half NULLs 1.01 112.3±0.26µs ? ?/sec 1.00 110.7±0.23µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, no NULLs 1.00 95.8±0.28µs ? ?/sec 1.00 95.6±0.11µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, mandatory, no NULLs 1.00 20.9±0.06µs ? ?/sec 1.00 20.9±0.02µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, half NULLs 1.02 75.4±0.15µs ? ?/sec 1.00 73.9±0.19µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, no NULLs 1.00 24.5±0.09µs ? ?/sec 1.00 24.5±0.02µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, mandatory, no NULLs 1.00 51.0±0.03µs ? ?/sec 1.00 51.2±0.03µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, half NULLs 1.01 92.9±0.22µs ? ?/sec 1.00 92.2±0.20µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, no NULLs 1.00 55.1±0.07µs ? ?/sec 1.00 54.9±0.04µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, mandatory, no NULLs 1.00 13.8±0.06µs ? ?/sec 1.00 13.8±0.03µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, half NULLs 1.02 73.1±0.25µs ? ?/sec 1.00 71.7±0.13µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, no NULLs 1.02 17.8±0.07µs ? ?/sec 1.00 17.5±0.03µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/90pct NULLs 1.17 874.6±1.47µs ? ?/sec 1.00 746.0±1.24µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/99pct NULLs 1.18 447.5±3.44µs ? ?/sec 1.00 380.0±5.00µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/half NULLs 1.23 2.2±0.00ms ? ?/sec 1.00 1756.3±10.78µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/no NULLs 1.20 3.0±0.01ms ? ?/sec 1.00 2.5±0.05ms ? ?/sec
arrow_array_reader/ListArray/Int32List/90pct NULLs 1.18 862.5±1.06µs ? ?/sec 1.00 728.0±0.91µs ? ?/sec
arrow_array_reader/ListArray/Int32List/99pct NULLs 1.17 446.4±3.21µs ? ?/sec 1.00 380.5±5.21µs ? ?/sec
arrow_array_reader/ListArray/Int32List/half NULLs 1.30 1961.2±4.64µs ? ?/sec 1.00 1512.3±3.19µs ? ?/sec
arrow_array_reader/ListArray/Int32List/no NULLs 1.27 2.7±0.01ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
arrow_array_reader/ListArray/Int32List/no NULLs or empty lists 1.38 2.6±0.00ms ? ?/sec 1.00 1887.4±3.55µs ? ?/sec
arrow_array_reader/ListArray/Int32List/no NULLs, skip and read in runs of 1 1.00 8.4±0.01ms ? ?/sec
arrow_array_reader/ListArray/Int32List/no NULLs, skip and read in runs of 32 1.00 1914.6±1.84µs ? ?/sec
arrow_array_reader/ListArray/StringList/90pct NULLs 1.17 933.4±1.24µs ? ?/sec 1.00 796.8±1.20µs ? ?/sec
arrow_array_reader/ListArray/StringList/99pct NULLs 1.17 455.4±4.74µs ? ?/sec 1.00 389.2±4.49µs ? ?/sec
arrow_array_reader/ListArray/StringList/half NULLs 1.20 3.1±0.02ms ? ?/sec 1.00 2.6±0.00ms ? ?/sec
arrow_array_reader/ListArray/StringList/no NULLs 1.17 5.0±0.02ms ? ?/sec 1.00 4.3±0.01ms ? ?/sec
arrow_array_reader/PlainStringToDictionary/high cardinality 1.01 1101.4±8.18µs ? ?/sec 1.00 1088.4±12.64µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/low cardinality 1.01 1030.9±7.58µs ? ?/sec 1.00 1019.6±9.28µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/medium cardinality 1 1.01 1101.6±7.91µs ? ?/sec 1.00 1087.5±12.35µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/medium cardinality 2 1.01 1051.0±7.56µs ? ?/sec 1.00 1039.2±9.00µs ? ?/sec
arrow_array_reader/StringArray/const delta byte array encoded, mandatory, no NULLs 1.00 527.7±2.90µs ? ?/sec 1.04 547.6±4.24µs ? ?/sec
arrow_array_reader/StringArray/const delta length byte array encoded, mandatory, no NULLs 1.00 209.3±0.79µs ? ?/sec 1.06 221.2±0.93µs ? ?/sec
arrow_array_reader/StringArray/const prefix delta byte array encoded, mandatory, no NULLs 1.00 768.6±33.74µs ? ?/sec 1.04 795.7±32.70µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, mandatory, no NULLs 1.02 284.2±0.21µs ? ?/sec 1.00 279.2±0.16µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, half NULLs 1.00 387.1±0.82µs ? ?/sec 1.02 396.7±0.61µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, no NULLs 1.00 280.5±0.53µs ? ?/sec 1.03 289.6±0.18µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, mandatory, no NULLs 1.00 449.6±2.72µs ? ?/sec 1.03 461.2±5.29µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, half NULLs 1.00 495.3±3.44µs ? ?/sec 1.02 503.6±3.81µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, no NULLs 1.00 450.0±4.39µs ? ?/sec 1.04 470.3±6.77µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, mandatory, no NULLs 1.00 195.6±0.72µs ? ?/sec 1.00 194.8±0.70µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, half NULLs 1.00 244.5±0.61µs ? ?/sec 1.00 245.3±0.79µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, no NULLs 1.00 198.0±0.67µs ? ?/sec 1.00 198.5±0.69µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, mandatory, no NULLs 1.00 80.5±0.04µs ? ?/sec 1.00 80.7±0.05µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, half NULLs 1.00 111.1±0.11µs ? ?/sec 1.00 111.5±0.12µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, no NULLs 1.00 84.3±0.06µs ? ?/sec 1.00 84.6±0.06µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, mandatory, no NULLs 1.01 224.0±0.69µs ? ?/sec 1.00 220.8±0.82µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, half NULLs 1.00 186.7±0.33µs ? ?/sec 1.00 186.2±2.10µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, no NULLs 1.01 232.5±0.82µs ? ?/sec 1.00 229.6±0.87µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed increasing value 1.00 54.3±0.77µs ? ?/sec 1.01 55.0±0.57µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed single value 1.01 46.5±0.30µs ? ?/sec 1.00 46.1±0.60µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip increasing value 1.00 33.5±0.35µs ? ?/sec 1.01 33.8±0.34µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip single value 1.01 29.1±0.18µs ? ?/sec 1.00 28.9±0.31µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip stepped increasing value 1.00 58.2±0.24µs ? ?/sec 1.00 58.2±0.23µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, mandatory, no NULLs 1.00 76.1±0.16µs ? ?/sec 1.00 75.8±0.20µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, half NULLs 1.01 73.8±0.15µs ? ?/sec 1.00 72.9±0.19µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, no NULLs 1.00 78.0±0.16µs ? ?/sec 1.00 77.9±0.20µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed stepped increasing value 1.00 75.4±0.15µs ? ?/sec 1.00 75.4±0.19µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, mandatory, no NULLs 1.00 97.4±0.29µs ? ?/sec 1.00 97.6±0.13µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, half NULLs 1.01 115.1±0.24µs ? ?/sec 1.00 114.4±0.24µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, no NULLs 1.00 101.0±0.16µs ? ?/sec 1.00 101.0±0.13µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, mandatory, no NULLs 1.03 21.8±0.04µs ? ?/sec 1.00 21.1±0.03µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, half NULLs 1.02 75.6±0.17µs ? ?/sec 1.00 74.3±0.17µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, no NULLs 1.00 24.5±0.07µs ? ?/sec 1.00 24.5±0.02µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, mandatory, no NULLs 1.00 51.2±0.05µs ? ?/sec 1.01 51.6±0.02µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, half NULLs 1.01 93.7±0.17µs ? ?/sec 1.00 92.5±0.19µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, no NULLs 1.00 54.9±0.05µs ? ?/sec 1.01 55.3±0.03µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, mandatory, no NULLs 1.00 13.8±0.02µs ? ?/sec 1.00 13.9±0.02µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, half NULLs 1.03 73.7±0.18µs ? ?/sec 1.00 71.8±0.13µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, no NULLs 1.01 17.7±0.08µs ? ?/sec 1.00 17.5±0.02µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed increasing value 1.00 51.0±0.79µs ? ?/sec 1.02 51.9±0.55µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed single value 1.00 43.1±0.31µs ? ?/sec 1.00 43.2±0.58µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip increasing value 1.00 31.7±0.34µs ? ?/sec 1.01 32.0±0.32µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip single value 1.00 27.3±0.18µs ? ?/sec 1.00 27.3±0.31µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip stepped increasing value 1.00 56.4±0.24µs ? ?/sec 1.00 56.6±0.24µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, mandatory, no NULLs 1.00 71.2±0.96µs ? ?/sec 1.00 71.4±0.99µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, half NULLs 1.01 70.4±0.46µs ? ?/sec 1.00 69.4±0.52µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, no NULLs 1.00 73.1±0.97µs ? ?/sec 1.00 73.2±0.94µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed stepped increasing value 1.00 71.7±0.14µs ? ?/sec 1.01 72.2±0.19µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, mandatory, no NULLs 1.00 89.7±0.93µs ? ?/sec 1.00 90.1±0.87µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, half NULLs 1.01 109.1±0.52µs ? ?/sec 1.00 108.4±0.45µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, no NULLs 1.00 93.4±0.96µs ? ?/sec 1.00 93.7±0.85µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, mandatory, no NULLs 1.00 17.5±0.02µs ? ?/sec 1.00 17.6±0.01µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, half NULLs 1.01 72.5±0.16µs ? ?/sec 1.00 71.5±0.17µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, no NULLs 1.00 20.9±0.08µs ? ?/sec 1.00 21.0±0.03µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, mandatory, no NULLs 1.00 47.9±0.04µs ? ?/sec 1.01 48.2±0.05µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, half NULLs 1.01 89.9±0.20µs ? ?/sec 1.00 88.7±0.21µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, no NULLs 1.00 51.3±0.06µs ? ?/sec 1.01 51.6±0.05µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, mandatory, no NULLs 1.00 10.1±0.02µs ? ?/sec 1.02 10.3±0.02µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, half NULLs 1.01 69.6±0.18µs ? ?/sec 1.00 69.0±0.19µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, no NULLs 1.00 13.7±0.06µs ? ?/sec 1.01 13.8±0.01µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed increasing value 1.00 41.9±0.16µs ? ?/sec 1.00 42.0±0.14µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed single value 1.00 38.4±0.07µs ? ?/sec 1.01 38.7±0.09µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip increasing value 1.00 24.7±0.09µs ? ?/sec 1.00 24.8±0.06µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip single value 1.00 22.8±0.04µs ? ?/sec 1.00 22.9±0.05µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip stepped increasing value 1.00 49.1±0.08µs ? ?/sec 1.00 49.3±0.06µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, mandatory, no NULLs 1.00 60.5±0.09µs ? ?/sec 1.01 60.9±0.06µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, half NULLs 1.00 64.5±0.13µs ? ?/sec 1.01 65.4±0.12µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, no NULLs 1.00 62.5±0.07µs ? ?/sec 1.01 62.9±0.07µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed stepped increasing value 1.00 63.6±0.11µs ? ?/sec 1.01 64.1±0.09µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, mandatory, no NULLs 1.00 77.7±0.15µs ? ?/sec 1.01 78.1±0.15µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, half NULLs 1.00 103.3±0.24µs ? ?/sec 1.01 104.8±0.26µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, no NULLs 1.00 81.4±0.14µs ? ?/sec 1.01 82.0±0.16µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, mandatory, no NULLs 1.01 46.5±0.03µs ? ?/sec 1.00 46.0±0.03µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, half NULLs 1.00 87.8±0.21µs ? ?/sec 1.00 88.0±0.14µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, no NULLs 1.01 50.5±0.08µs ? ?/sec 1.00 49.9±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, mandatory, no NULLs 1.00 50.2±0.04µs ? ?/sec 1.00 50.2±0.06µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, half NULLs 1.00 90.8±0.24µs ? ?/sec 1.01 91.3±0.18µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, no NULLs 1.00 53.6±0.13µs ? ?/sec 1.00 53.9±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, mandatory, no NULLs 1.00 16.2±0.03µs ? ?/sec 1.01 16.4±0.02µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, half NULLs 1.00 72.9±0.21µs ? ?/sec 1.01 73.9±0.15µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, no NULLs 1.00 19.8±0.06µs ? ?/sec 1.02 20.3±0.05µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed increasing value 1.00 54.3±0.81µs ? ?/sec 1.00 54.4±0.56µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed single value 1.00 46.2±0.27µs ? ?/sec 1.02 47.1±0.40µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip increasing value 1.00 33.4±0.34µs ? ?/sec 1.00 33.3±0.35µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip single value 1.00 29.0±0.19µs ? ?/sec 1.01 29.4±0.25µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip stepped increasing value 1.00 58.3±0.24µs ? ?/sec 1.00 58.1±0.24µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, mandatory, no NULLs 1.00 75.2±0.31µs ? ?/sec 1.01 75.8±0.43µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, half NULLs 1.01 73.2±0.21µs ? ?/sec 1.00 72.4±0.25µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, no NULLs 1.00 77.2±0.32µs ? ?/sec 1.01 77.7±0.29µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed stepped increasing value 1.00 75.3±0.14µs ? ?/sec 1.00 75.3±0.17µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, mandatory, no NULLs 1.00 97.2±0.22µs ? ?/sec 1.00 97.5±0.17µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, half NULLs 1.00 114.1±0.22µs ? ?/sec 1.00 114.0±0.20µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, no NULLs 1.00 100.9±0.27µs ? ?/sec 1.00 100.4±0.11µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, mandatory, no NULLs 1.00 20.7±0.03µs ? ?/sec 1.01 20.9±0.02µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, half NULLs 1.02 75.3±0.18µs ? ?/sec 1.00 74.1±0.22µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, no NULLs 1.00 24.2±0.07µs ? ?/sec 1.01 24.4±0.02µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, mandatory, no NULLs 1.00 50.9±0.05µs ? ?/sec 1.01 51.2±0.04µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, half NULLs 1.01 92.9±0.28µs ? ?/sec 1.00 92.1±0.17µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, no NULLs 1.00 54.6±0.07µs ? ?/sec 1.01 55.3±0.05µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, mandatory, no NULLs 1.00 13.7±0.02µs ? ?/sec 1.00 13.7±0.03µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, half NULLs 1.00 72.5±0.17µs ? ?/sec 1.00 72.2±0.15µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, no NULLs 1.00 17.3±0.05µs ? ?/sec 1.00 17.4±0.02µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, mandatory struct, optional data, half NULLs 1.02 69.9±0.15µs ? ?/sec 1.00 68.6±0.18µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, mandatory struct, optional data, no NULLs 1.00 13.7±0.01µs ? ?/sec 1.01 13.8±0.04µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, half NULLs 1.14 130.3±0.14µs ? ?/sec 1.00 114.7±0.16µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, no NULLs 2.92 65.7±0.05µs ? ?/sec 1.00 22.5±0.07µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>arrow_reader — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1ph4zneg-31424f9b-2g92
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6021480575-3099-r755r 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 3015.7s |
| Peak memory | 174.1 MiB |
| Avg memory | 43.1 MiB |
| CPU user | 2996.3s |
| CPU sys | 13.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 3035.7s |
| Peak memory | 193.3 MiB |
| Avg memory | 44.3 MiB |
| CPU user | 3019.3s |
| CPU sys | 12.4s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6021480575:finish:c265ac06cdb53737e8d03122983b33f6 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#issuecomment-6021458917
🤖 Benchmark failed or incomplete (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11408#issuecomment-6019251386)
**Target:** arrow_reader
Comparing neilc/parquet-count-levels (63d9cad5b144795dad5e3793a800253316d9fc3f) to d18ea16 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/d18ea16f6fc9af0ba7e8ea76f6934c6e6cb57a26..63d9cad5b144795dad5e3793a800253316d9fc3f)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_reader
shards: 1
```
</details>
<details open><summary>Errors / missing shards</summary>
````text
arrow_reader — shard 1/1: Benchmark for [this request](https://github.com/apache/arrow-rs/pull/11408#issuecomment-6019251386) hit the 7200s job deadline before finishing.
Benchmarks requested: `arrow_reader`
<details><summary>Kubernetes message</summary>
```
Job was active longer than specified deadline
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
````
</details>
<details><summary>Per-runner information</summary>
<details><summary>arrow_reader — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1ph4zneg-31424f9b-2g92
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6019251386-3097-pwqch 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 3020.6s |
| Peak memory | 189.4 MiB |
| Avg memory | 43.2 MiB |
| CPU user | 3004.7s |
| CPU sys | 12.8s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6019251386:finish:5c74d9ddde2f9155bb4d692455e63090 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#issuecomment-6021481952
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11408#issuecomment-6021480575)
**Target:** arrow_reader
**Sharding factor: 1** (1 workers).
Comparing neilc/parquet-count-levels (63d9cad5b144795dad5e3793a800253316d9fc3f) to d18ea16 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/d18ea16f6fc9af0ba7e8ea76f6934c6e6cb57a26..63d9cad5b144795dad5e3793a800253316d9fc3f)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_reader
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6021480575:start:11c2090b5ba7a573f9916b4b37d21d0e -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#issuecomment-6065418629
Nice optimization, `count_levels_eq` seems to indeed generate better code than a naive counting version. The optimization for `values_read == levels_read` also looks good to me. I'm not so familiar with `count_records`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#issuecomment-6019251386
run benchmark arrow_reader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#issuecomment-6060905830
@jhorstmann If you have any availability to review, I would love any feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#issuecomment-6019253808
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11408#issuecomment-6019251386)
**Target:** arrow_reader
**Sharding factor: 1** (1 workers).
Comparing neilc/parquet-count-levels (63d9cad5b144795dad5e3793a800253316d9fc3f) to d18ea16 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/d18ea16f6fc9af0ba7e8ea76f6934c6e6cb57a26..63d9cad5b144795dad5e3793a800253316d9fc3f)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_reader
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6019251386:start:f5f52daedf0e6b25ea4848054bcb75d9 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#discussion_r4231413063
##########
parquet/src/arrow/record_reader/definition_levels.rs:
##########
@@ -257,10 +257,16 @@ impl DefinitionLevelDecoder for DefinitionLevelBufferDecoder {
let start = levels.len();
let (values_read, levels_read) = decoder.read_def_levels(levels, num_levels)?;
- // Safety: slice iterator has a trusted length
- unsafe {
- nulls
- .extend_trusted_len(levels[start..].iter().map(|level| level == max_level));
+ if values_read == levels_read {
+ // Every level is at `max_level`
+ nulls.append_n(levels_read, true);
+ } else {
+ // Safety: slice iterator has a trusted length
+ unsafe {
+ nulls.extend_trusted_len(
+ levels[start..].iter().map(|level| level == max_level),
+ );
+ }
Review Comment:
Good idea! Added to the PR, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on code in PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#discussion_r4230678431
##########
parquet/src/column/reader/decoder.rs:
##########
@@ -393,12 +409,28 @@ impl RepetitionLevelDecoderImpl {
///
/// A "complete" record is one where the buffer contains a subsequent repetition level of 0
fn count_records(&self, records_to_read: usize, num_levels: usize) -> (bool, usize, usize) {
Review Comment:
Thanks, the description is much clearer now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "HippoBaro (via GitHub)" <gi...@apache.org>.
HippoBaro commented on code in PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#discussion_r4230906177
##########
parquet/src/arrow/record_reader/definition_levels.rs:
##########
@@ -257,10 +257,16 @@ impl DefinitionLevelDecoder for DefinitionLevelBufferDecoder {
let start = levels.len();
let (values_read, levels_read) = decoder.read_def_levels(levels, num_levels)?;
- // Safety: slice iterator has a trusted length
- unsafe {
- nulls
- .extend_trusted_len(levels[start..].iter().map(|level| level == max_level));
+ if values_read == levels_read {
+ // Every level is at `max_level`
+ nulls.append_n(levels_read, true);
+ } else {
+ // Safety: slice iterator has a trusted length
+ unsafe {
+ nulls.extend_trusted_len(
+ levels[start..].iter().map(|level| level == max_level),
+ );
+ }
Review Comment:
You're think the symnetric case also works here:
```suggestion
if values_read == levels_read {
// Every level is at `max_level`
nulls.append_n(levels_read, true);
} else if values_read == 0 {
nulls.append_n(levels_read, false);
} else {
// Safety: slice iterator has a trusted length
unsafe {
nulls.extend_trusted_len(
levels[start..].iter().map(|level| level == max_level),
);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#issuecomment-6155024869
Thanks everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on code in PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#discussion_r4222208527
##########
parquet/src/column/reader/decoder.rs:
##########
@@ -393,12 +409,28 @@ impl RepetitionLevelDecoderImpl {
///
/// A "complete" record is one where the buffer contains a subsequent repetition level of 0
fn count_records(&self, records_to_read: usize, num_levels: usize) -> (bool, usize, usize) {
Review Comment:
The documentation for this method might need an update, I had to lookup that the bool meant partial records. And is it still "complete" records that we count if the partial flag is true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "HippoBaro (via GitHub)" <gi...@apache.org>.
HippoBaro commented on code in PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#discussion_r4230906177
##########
parquet/src/arrow/record_reader/definition_levels.rs:
##########
@@ -257,10 +257,16 @@ impl DefinitionLevelDecoder for DefinitionLevelBufferDecoder {
let start = levels.len();
let (values_read, levels_read) = decoder.read_def_levels(levels, num_levels)?;
- // Safety: slice iterator has a trusted length
- unsafe {
- nulls
- .extend_trusted_len(levels[start..].iter().map(|level| level == max_level));
+ if values_read == levels_read {
+ // Every level is at `max_level`
+ nulls.append_n(levels_read, true);
+ } else {
+ // Safety: slice iterator has a trusted length
+ unsafe {
+ nulls.extend_trusted_len(
+ levels[start..].iter().map(|level| level == max_level),
+ );
+ }
Review Comment:
I think the symnetric case also works here:
```suggestion
if values_read == levels_read {
// Every level is at `max_level`
nulls.append_n(levels_read, true);
} else if values_read == 0 {
nulls.append_n(levels_read, false);
} else {
// Safety: slice iterator has a trusted length
unsafe {
nulls.extend_trusted_len(
levels[start..].iter().map(|level| level == max_level),
);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#discussion_r4230467193
##########
parquet/src/column/reader/decoder.rs:
##########
@@ -393,12 +409,28 @@ impl RepetitionLevelDecoderImpl {
///
/// A "complete" record is one where the buffer contains a subsequent repetition level of 0
fn count_records(&self, records_to_read: usize, num_levels: usize) -> (bool, usize, usize) {
Review Comment:
I took a pass over the comments in this part of the code; let me know if what I wrote makes things more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): optimize level counting [arrow-rs]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #11408:
URL: https://github.com/apache/arrow-rs/pull/11408#issuecomment-6021480575
run benchmark arrow_reader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org