You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2025/12/04 14:28:21 UTC

[PR] Make filtered coalescing faster for primitive types [arrow-rs]

Dandandan opened a new pull request, #8951:
URL: https://github.com/apache/arrow-rs/pull/8951

   
   # Which issue does this PR close?
   
   - Closes #NNN.
   
   # Rationale for this change
   
   Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
   Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
   
   # What changes are included in this PR?
   
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   
   # Are these changes tested?
   
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   
   # Are there any user-facing changes?
   
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   
   If there are any breaking changes to public APIs, please call them out.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3619751135

   @alamb seems we can play with the filter threshold value, probably a value with >=0.9 will give nice speedups, we might even go further based on some benchmark results


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3619745121

   run benchmarks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 10x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3613212415

   run benchmark filter_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763267983

   run benchmark filter_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763795059

   I am about out of time today, but here is what I was thinking both for testing and to support multi-column batches (it doesn't pass tests yet)
   - https://github.com/Dandandan/arrow-rs/pull/13
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763860926

   Ah, NM I see you did it here [e76a5d7](https://github.com/apache/arrow-rs/pull/8951/commits/e76a5d73e9d5f5cd26011c5ed5a10a916642899c)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3764348856

   run benchmarks coalesce_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2698422312


##########
arrow-select/src/coalesce/primitive.rs:
##########
@@ -95,6 +96,145 @@ impl<T: ArrowPrimitiveType + Debug> InProgressArray for InProgressPrimitiveArray
         Ok(())
     }
 
+    /// Copy rows using a predicate

Review Comment:
   The refactor looks nice, but what parts could be reused? I think it is not possible to reuse the filtering directly, as the existing code allocates a new vec / buffer each time. I think the other way around could work (existing filter code uses some form of this code).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3765271988

   run benchmark coalesce_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3627282606

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (d2b5d296471710319ecc8fa0d0662ea466f5ab80) to ed9efe78e4cc958cc96707557818e754419debb0 [diff](https://github.com/apache/arrow-rs/compare/ed9efe78e4cc958cc96707557818e754419debb0..d2b5d296471710319ecc8fa0d0662ea466f5ab80)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3654531991

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    257.2±2.50ms        ? ?/sec    1.00    256.6±3.74ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.04      8.8±0.18ms        ? ?/sec    1.00      8.5±0.10ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.03      4.2±0.13ms        ? ?/sec    1.00      4.1±0.12ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.3±0.04ms        ? ?/sec    1.08      3.5±0.03ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    244.5±3.19ms        ? ?/sec    1.27    310.3±4.52ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.01      9.4±0.17ms        ? ?/sec    1.00      9.3±0.24ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.03      4.6±0.11ms        ? ?/sec    1.00      4.5±0.10ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.7±0.07ms        ? ?/sec    1.23      4.6±0.09ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     59.7±0.49ms        ? ?/sec    1.00     59.5±0.60ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     11.5±0.19ms        ? ?/sec    1.01     11.5±0.15ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.04      9.5±0.33ms        ? ?/sec    1.00      9.2±0.40ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.7±0.15ms        ? ?/sec    1.33     10.3±0.21ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.01     70.3±0.37ms        ? ?/sec    1.00     69.6±2.61ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.8±0.15ms        ? ?/sec    1.00     12.8±0.23ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.01      9.6±0.33ms        ? ?/sec    1.00      9.5±0.21ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      8.3±0.22ms        ? ?/sec    1.17      9.8±0.21ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.02     49.5±0.41ms        ? ?/sec    1.00     48.4±0.46ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      5.9±0.06ms        ? ?/sec    1.00      5.9±0.10ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.03      4.6±0.25ms        ? ?/sec    1.00      4.5±0.23ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00      2.6±0.05ms        ? ?/sec    1.14      3.0±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.01     59.2±0.47ms        ? ?/sec    1.00     58.5±0.79ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      8.0±0.25ms        ? ?/sec    1.01      8.1±0.23ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.00      5.5±0.25ms        ? ?/sec    1.03      5.7±0.22ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.3±0.02ms        ? ?/sec    1.73      3.9±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.02     43.5±0.42ms        ? ?/sec    1.00     42.6±0.28ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      4.7±0.03ms        ? ?/sec    1.00      4.7±0.10ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.10      2.5±0.19ms        ? ?/sec    1.00      2.3±0.18ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1176.8±34.76µs        ? ?/sec    1.28   1506.1±8.68µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.02     53.5±0.65ms        ? ?/sec    1.00     52.4±0.43ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      7.0±0.04ms        ? ?/sec    1.01      7.1±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.03      3.8±0.19ms        ? ?/sec    1.00      3.7±0.12ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.4±0.06ms        ? ?/sec    1.64      3.9±0.07ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     55.4±0.82ms        ? ?/sec    1.76     97.3±1.34ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      5.8±0.02ms        ? ?/sec    1.60      9.3±0.07ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.6±0.41ms        ? ?/sec    1.10      4.0±0.37ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1681.5±15.51µs        ? ?/sec    1.86      3.1±0.06ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00     62.1±0.50ms        ? ?/sec    2.02    125.4±0.57ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     10.1±0.15ms        ? ?/sec    1.50     15.0±0.07ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.4±0.25ms        ? ?/sec    1.16      7.4±0.36ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.5±0.03ms        ? ?/sec    2.02      9.1±0.05ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.02     67.3±1.39ms        ? ?/sec    1.00     66.2±0.99ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.04      7.6±0.04ms        ? ?/sec    1.00      7.3±0.12ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.00      4.1±0.31ms        ? ?/sec    1.02      4.2±0.36ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.00   1335.7±6.87µs        ? ?/sec    1.06   1412.2±9.47µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.07     89.5±0.97ms        ? ?/sec    1.00     83.9±0.28ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.04     11.6±0.28ms        ? ?/sec    1.00     11.2±0.12ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.01      5.5±0.36ms        ? ?/sec    1.00      5.5±0.31ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      2.7±0.01ms        ? ?/sec    1.46      3.9±0.04ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3768906969

   
   run benchmark coalesce_kernels
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3769669673

   
   run benchmark coalesce_kernels
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771628159

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    248.4±2.81ms        ? ?/sec    1.01    251.1±2.78ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      7.8±0.18ms        ? ?/sec    1.06      8.2±0.18ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      3.9±0.15ms        ? ?/sec    1.02      4.0±0.12ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.1±0.03ms        ? ?/sec    1.10      3.4±0.10ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    236.1±2.81ms        ? ?/sec    1.00    235.4±2.42ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.7±0.07ms        ? ?/sec    1.02      8.9±0.09ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.02      4.4±0.11ms        ? ?/sec    1.00      4.3±0.13ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.5±0.03ms        ? ?/sec    1.26      4.4±0.08ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     52.2±0.70ms        ? ?/sec    1.10     57.3±0.71ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     10.7±0.15ms        ? ?/sec    1.05     11.2±0.30ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.1±0.39ms        ? ?/sec    1.02      9.3±0.36ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.4±0.12ms        ? ?/sec    1.37     10.1±0.17ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     64.9±0.45ms        ? ?/sec    1.03     67.1±1.09ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.3±0.12ms        ? ?/sec    1.01     12.4±0.24ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.4±0.28ms        ? ?/sec    1.03      9.7±0.33ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      7.9±0.13ms        ? ?/sec    1.19      9.4±0.15ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     36.1±0.21ms        ? ?/sec    1.27     45.8±0.63ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.03      5.8±0.14ms        ? ?/sec    1.00      5.6±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.32      5.8±0.18ms        ? ?/sec    1.00      4.4±0.23ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.11      3.1±0.04ms        ? ?/sec    1.00      2.8±0.08ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     54.6±0.76ms        ? ?/sec    1.02     56.0±0.36ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.13      8.5±0.13ms        ? ?/sec    1.00      7.5±0.06ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.31      6.9±0.21ms        ? ?/sec    1.00      5.2±0.25ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.6±0.02ms        ? ?/sec    1.35      3.6±0.07ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     35.0±0.26ms        ? ?/sec    1.18     41.4±0.33ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.11      4.9±0.08ms        ? ?/sec    1.00      4.4±0.02ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.65      3.7±0.14ms        ? ?/sec    1.00      2.2±0.21ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.25  1637.1±16.61µs        ? ?/sec    1.00  1306.1±10.33µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.03     52.8±1.19ms        ? ?/sec    1.00     51.1±0.58ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.17      7.7±0.04ms        ? ?/sec    1.00      6.5±0.15ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.48      4.9±0.14ms        ? ?/sec    1.00      3.3±0.20ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.8±0.01ms        ? ?/sec    1.27      3.6±0.09ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     67.4±0.71ms        ? ?/sec    1.38     92.8±2.28ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.7±0.12ms        ? ?/sec    1.31      8.8±0.07ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.4±0.39ms        ? ?/sec    1.04      3.6±0.34ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1627.8±13.52µs        ? ?/sec    1.76      2.9±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    105.1±0.87ms        ? ?/sec    1.15    120.6±0.94ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     12.8±0.07ms        ? ?/sec    1.10     14.2±0.19ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.04      6.7±0.47ms        ? ?/sec    1.00      6.4±0.34ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.4±0.03ms        ? ?/sec    1.91      8.4±0.06ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.04     66.3±0.30ms        ? ?/sec    1.00     64.0±0.64ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.29      9.5±0.19ms        ? ?/sec    1.00      7.3±0.05ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.78      7.0±0.17ms        ? ?/sec    1.00      3.9±0.31ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            3.34      4.3±0.05ms        ? ?/sec    1.00  1291.8±14.03µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.30    107.7±1.06ms        ? ?/sec    1.00     82.6±0.75ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.30     14.2±0.09ms        ? ?/sec    1.00     10.9±0.12ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.56      8.6±0.09ms        ? ?/sec    1.00      5.5±0.37ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.64      6.0±0.10ms        ? ?/sec    1.00      3.6±0.02ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037

   run benchmarks coalesce_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771672443

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771677714

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771672905

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771680772

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771701230

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771702954

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771711638

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771702486

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771716408

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771686292

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771718917

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771756351

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771760846

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771791168

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771777458

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771768015

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771788358

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771801447

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3627376515

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    263.9±2.33ms        ? ?/sec    1.00    263.2±2.58ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      8.5±0.08ms        ? ?/sec    1.00      8.5±0.11ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.01      4.1±0.10ms        ? ?/sec    1.00      4.0±0.05ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.3±0.06ms        ? ?/sec    1.07      3.5±0.05ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    248.2±1.98ms        ? ?/sec    1.28    318.8±4.18ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      9.4±0.10ms        ? ?/sec    1.00      9.4±0.10ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.01      4.5±0.10ms        ? ?/sec    1.00      4.5±0.06ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.7±0.05ms        ? ?/sec    1.26      4.6±0.06ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     59.1±0.55ms        ? ?/sec    1.00     59.3±0.77ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     11.6±0.15ms        ? ?/sec    1.00     11.6±0.11ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.01      9.4±0.14ms        ? ?/sec    1.00      9.3±0.15ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      8.1±0.17ms        ? ?/sec    1.34     10.8±0.28ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     69.6±0.48ms        ? ?/sec    1.02     70.8±1.08ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.7±0.10ms        ? ?/sec    1.02     13.0±0.18ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.8±0.20ms        ? ?/sec    1.00      9.8±0.22ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      8.6±0.21ms        ? ?/sec    1.18     10.1±0.28ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     48.6±0.43ms        ? ?/sec    1.00     48.7±0.30ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.01      6.0±0.08ms        ? ?/sec    1.00      6.0±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.02      4.6±0.12ms        ? ?/sec    1.00      4.5±0.06ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00      2.6±0.03ms        ? ?/sec    1.17      3.1±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     58.4±0.76ms        ? ?/sec    1.00     58.5±0.46ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      7.9±0.12ms        ? ?/sec    1.02      8.1±0.10ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.00      5.5±0.12ms        ? ?/sec    1.02      5.7±0.16ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.2±0.01ms        ? ?/sec    1.75      3.9±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     42.5±0.61ms        ? ?/sec    1.00     42.7±0.65ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      4.7±0.10ms        ? ?/sec    1.01      4.7±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.01      2.3±0.07ms        ? ?/sec    1.00      2.3±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1137.8±16.65µs        ? ?/sec    1.32  1498.0±15.99µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     52.0±0.28ms        ? ?/sec    1.00     52.2±1.36ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      7.0±0.11ms        ? ?/sec    1.01      7.1±0.07ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.02      3.7±0.16ms        ? ?/sec    1.00      3.6±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.3±0.03ms        ? ?/sec    1.68      3.9±0.03ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     54.2±1.34ms        ? ?/sec    1.79     97.2±1.12ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      5.9±0.03ms        ? ?/sec    1.58      9.3±0.06ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.3±0.22ms        ? ?/sec    1.14      3.8±0.03ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1689.2±38.67µs        ? ?/sec    1.81      3.1±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00     59.2±1.38ms        ? ?/sec    2.14    126.3±1.98ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00      8.1±0.08ms        ? ?/sec    1.86     15.1±0.42ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.4±0.11ms        ? ?/sec    1.12      7.1±0.27ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.9±0.07ms        ? ?/sec    1.84      9.0±0.05ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     64.7±0.37ms        ? ?/sec    1.02     65.9±0.32ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.01      7.4±0.08ms        ? ?/sec    1.00      7.3±0.12ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.02      4.0±0.28ms        ? ?/sec    1.00      3.9±0.09ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.00  1287.1±14.90µs        ? ?/sec    1.11  1429.1±15.07µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.07     89.7±0.95ms        ? ?/sec    1.00     84.2±0.38ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.02     11.5±0.11ms        ? ?/sec    1.00     11.2±0.16ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.04      5.3±0.14ms        ? ?/sec    1.00      5.1±0.08ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      2.7±0.02ms        ? ?/sec    1.46      4.0±0.03ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3634120510

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (1acccc7d04b97d59bae4cddeab847f7bc3ce8b77) to ed9efe78e4cc958cc96707557818e754419debb0 [diff](https://github.com/apache/arrow-rs/compare/ed9efe78e4cc958cc96707557818e754419debb0..1acccc7d04b97d59bae4cddeab847f7bc3ce8b77)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3634119941

   
   run benchmark coalesce_kernels filter_kernels
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2616171034


##########
arrow-select/src/coalesce.rs:
##########
@@ -571,6 +788,15 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
     /// Return an error if the source array is not set
     fn copy_rows(&mut self, offset: usize, len: usize) -> Result<(), ArrowError>;
 
+    /// Copy rows at the given indices from the current source array into the in-progress array
+    fn copy_rows_by_filter(&mut self, filter: &FilterPredicate) -> Result<(), ArrowError> {
+        // Default implementation: iterate over indices from the filter

Review Comment:
   Correct, for views & bytes array



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2615576833


##########
arrow-select/src/coalesce.rs:
##########
@@ -526,6 +631,118 @@ impl BatchCoalescer {
     }
 }
 
+/// Find the position after the n-th set bit in a boolean array starting from `start`.
+/// Returns the position after the n-th set bit, or the end of the array if fewer than n bits are set.

Review Comment:
   I recommend we move this code into BooleanBuffer as well so it is easier to find / reuse



##########
arrow-buffer/src/builder/null.rs:
##########
@@ -402,4 +481,43 @@ mod tests {
 
         assert_eq!(builder.finish(), None);
     }
+
+    #[test]
+    fn test_extend() {
+        // Test small extend (less than 64 bits)
+        let mut builder = NullBufferBuilder::new(0);
+        builder.extend([true, false, true, true].iter().copied());
+        // bits: 0=true, 1=false, 2=true, 3=true -> 0b1101 = 13
+        assert_eq!(builder.as_slice().unwrap(), &[0b1101_u8]);
+
+        // Test extend with exactly 64 bits
+        let mut builder = NullBufferBuilder::new(0);
+        let pattern: Vec<bool> = (0..64).map(|i| i % 2 == 0).collect();
+        builder.extend(pattern.iter().copied());
+        // Even positions are true: 0, 2, 4, ... -> bits 0, 2, 4, ...
+        // In little-endian: 0b01010101 repeated
+        assert_eq!(
+            builder.as_slice().unwrap(),
+            &[0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55]
+        );
+
+        // Test extend with more than 64 bits (tests chunking)
+        let mut builder = NullBufferBuilder::new(0);
+        let pattern: Vec<bool> = (0..100).map(|i| i % 3 == 0).collect();
+        builder.extend(pattern.iter().copied());
+        assert_eq!(builder.len(), 100);
+        // Verify a few specific bits
+        let buf = builder.finish().unwrap();
+        assert!(buf.is_valid(0)); // 0 % 3 == 0
+        assert!(!buf.is_valid(1)); // 1 % 3 != 0
+        assert!(!buf.is_valid(2)); // 2 % 3 != 0
+        assert!(buf.is_valid(3)); // 3 % 3 == 0
+        assert!(buf.is_valid(99)); // 99 % 3 == 0
+
+        // Test extend with non-aligned start (tests bit-by-bit path)
+        let mut builder = NullBufferBuilder::new(0);
+        builder.append_non_null(); // Start at bit 1 (non-aligned)
+        builder.extend([false, true, false, true].iter().copied());

Review Comment:
   I think we should probably test non aligned writes with more than 64 bits as well (this only copies 4 bits)



##########
arrow-select/src/coalesce/primitive.rs:
##########
@@ -92,6 +93,145 @@ impl<T: ArrowPrimitiveType + Debug> InProgressArray for InProgressPrimitiveArray
         Ok(())
     }
 
+    /// Copy rows using a predicate
+    fn copy_rows_by_filter(&mut self, filter: &FilterPredicate) -> Result<(), ArrowError> {
+        self.ensure_capacity();
+
+        let s = self
+            .source
+            .as_ref()
+            .ok_or_else(|| {
+                ArrowError::InvalidArgumentError(
+                    "Internal Error: InProgressPrimitiveArray: source not set".to_string(),
+                )
+            })?
+            .as_primitive::<T>();
+
+        let values = s.values();
+        let count = filter.count();
+
+        // Use the predicate's strategy for optimal iteration
+        match filter.strategy() {
+            IterationStrategy::SlicesIterator => {
+                // Copy values, nulls using slices
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    for (start, end) in SlicesIterator::new(filter.filter_array()) {
+                        // SAFETY: slices are derived from filter predicate
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                        let slice = nulls.slice(start, end - start);
+                        self.nulls.append_buffer(&slice);
+                    }
+                } else {
+                    for (start, end) in SlicesIterator::new(filter.filter_array()) {
+                        // SAFETY: SlicesIterator produces valid ranges derived from filter
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                    }
+                    self.nulls.append_n_non_nulls(count);
+                }
+            }
+            IterationStrategy::Slices(slices) => {

Review Comment:
   I think this function needs some tests
   
   I ran code coverage like this
   
   ```shell
   cargo llvm-cov test --html -p arrow-buffer  -p arrow-select
   ```
   
   And there appears to be no coverage
   <img width="1097" height="949" alt="Image" src="https://github.com/user-attachments/assets/0e644657-d723-4f5f-925a-3c4ce86a78bc" />
   
   



##########
arrow-buffer/src/builder/null.rs:
##########
@@ -193,6 +193,85 @@ impl NullBufferBuilder {
         }
     }
 
+    /// Extends this builder with validity values.
+    ///
+    ///
+    /// # Example
+    /// ```
+    /// # use arrow_buffer::NullBufferBuilder;
+    /// let mut builder = NullBufferBuilder::new(8);
+    /// let validities = [true, false, true, true];
+    /// builder.extend(validities.iter().copied());
+    /// assert_eq!(builder.len(), 4);
+    /// ```
+    pub fn extend<I: Iterator<Item = bool>>(&mut self, iter: I) {
+        let (lower, upper) = iter.size_hint();
+        let len = upper.expect("Iterator must have exact size_hint");
+        debug_assert_eq!(lower, len, "Iterator must have exact size_hint");
+
+        if len == 0 {
+            return;
+        }
+
+        // Materialize since we're about to append bits
+        self.materialize_if_needed();
+
+        let buf = self.bitmap_builder.as_mut().unwrap();
+        let start_len = buf.len();
+        // Advance to allocate space, initializing new bits to 0
+        buf.advance(len);
+
+        let slice = buf.as_slice_mut();
+        let mut bit_idx = start_len;
+        let end_bit = start_len + len;
+
+        // Process in chunks of 64 bits when byte-aligned for better performance
+        if start_len % 8 == 0 {
+            let start_byte = start_len / 8;
+            let mut iter = iter.peekable();
+
+            // Process full u64 chunks (64 bits at a time)
+            while bit_idx + 64 <= end_bit && iter.peek().is_some() {
+                let mut chunk: u64 = 0;
+                for i in 0..64 {
+                    if let Some(valid) = iter.next() {
+                        if valid {
+                            chunk |= 1u64 << i;
+                        }
+                    } else {
+                        break;
+                    }
+                }
+                let byte_idx = (bit_idx - start_len) / 8 + start_byte;
+                // Write the u64 chunk as 8 bytes
+                slice[byte_idx..byte_idx + 8].copy_from_slice(&chunk.to_le_bytes());

Review Comment:
   could try unsafe here too as you ensured the right length above



##########
arrow-buffer/src/builder/null.rs:
##########
@@ -193,6 +193,85 @@ impl NullBufferBuilder {
         }
     }
 
+    /// Extends this builder with validity values.
+    ///
+    ///
+    /// # Example
+    /// ```
+    /// # use arrow_buffer::NullBufferBuilder;
+    /// let mut builder = NullBufferBuilder::new(8);
+    /// let validities = [true, false, true, true];
+    /// builder.extend(validities.iter().copied());
+    /// assert_eq!(builder.len(), 4);
+    /// ```
+    pub fn extend<I: Iterator<Item = bool>>(&mut self, iter: I) {
+        let (lower, upper) = iter.size_hint();

Review Comment:
   I think this method would be more generally when appending to any BooleanBuffer rather than just NullBufferBuilder
   
   As part of the goal to consolidate mutable boolean operations in `BooleanBufferBuilder` so it is easier to find (and optimize) them, would you be willing to move this code to `BooleanBufferBuilder`  so that the code in NullBufferBuilder looks like something like this (which is what most other methods in NullBufferBuilder look like)?
   
   ```rust
       pub fn extend<I: Iterator<Item = bool>>(&mut self, iter: I) {
           // Materialize since we're about to append bits
           self.materialize_if_needed();
           self.bitmap_builder.as_mut().unwrap().extend(iter)
       }
   ```
   
   
   
   
   



##########
arrow-buffer/src/builder/null.rs:
##########
@@ -193,6 +193,85 @@ impl NullBufferBuilder {
         }
     }
 
+    /// Extends this builder with validity values.
+    ///
+    ///
+    /// # Example
+    /// ```
+    /// # use arrow_buffer::NullBufferBuilder;
+    /// let mut builder = NullBufferBuilder::new(8);
+    /// let validities = [true, false, true, true];
+    /// builder.extend(validities.iter().copied());
+    /// assert_eq!(builder.len(), 4);
+    /// ```
+    pub fn extend<I: Iterator<Item = bool>>(&mut self, iter: I) {
+        let (lower, upper) = iter.size_hint();
+        let len = upper.expect("Iterator must have exact size_hint");
+        debug_assert_eq!(lower, len, "Iterator must have exact size_hint");
+
+        if len == 0 {
+            return;
+        }
+
+        // Materialize since we're about to append bits
+        self.materialize_if_needed();
+
+        let buf = self.bitmap_builder.as_mut().unwrap();
+        let start_len = buf.len();
+        // Advance to allocate space, initializing new bits to 0
+        buf.advance(len);
+
+        let slice = buf.as_slice_mut();
+        let mut bit_idx = start_len;
+        let end_bit = start_len + len;
+
+        // Process in chunks of 64 bits when byte-aligned for better performance
+        if start_len % 8 == 0 {
+            let start_byte = start_len / 8;
+            let mut iter = iter.peekable();
+
+            // Process full u64 chunks (64 bits at a time)
+            while bit_idx + 64 <= end_bit && iter.peek().is_some() {

Review Comment:
   As a follow on PR, it might be worth aligning first on 64 bit boundaries (so the underlying code doesn't have to handle aligning) -- aka handle bits 0..63 (until 64 bit alignment) specially and then use the u64 path



##########
arrow-select/src/coalesce.rs:
##########
@@ -237,10 +243,109 @@ impl BatchCoalescer {
         batch: RecordBatch,
         filter: &BooleanArray,
     ) -> Result<(), ArrowError> {
-        // TODO: optimize this to avoid materializing (copying the results

Review Comment:
   🎉 



##########
arrow-select/src/filter.rs:
##########
@@ -276,15 +289,22 @@ impl FilterBuilder {
 }
 
 /// The iteration strategy used to evaluate [`FilterPredicate`]
-#[derive(Debug)]
-enum IterationStrategy {
-    /// A lazily evaluated iterator of ranges
+///
+/// This determines how the filter will iterate over the selected rows.
+/// The strategy is chosen based on the selectivity of the filter.
+#[derive(Debug, Clone)]
+pub enum IterationStrategy {
+    /// A lazily evaluated iterator of ranges (slices)
+    ///
+    /// Best for low selectivity filters

Review Comment:
   I think the term "low selectivity" filters is inconsistently used. I recommend making this explicit
   
   ```suggestion
       /// Best for low selectivity filters (which select a relatively large number of rows)
   ```



##########
arrow-select/src/filter.rs:
##########
@@ -276,15 +289,22 @@ impl FilterBuilder {
 }
 
 /// The iteration strategy used to evaluate [`FilterPredicate`]
-#[derive(Debug)]
-enum IterationStrategy {
-    /// A lazily evaluated iterator of ranges
+///
+/// This determines how the filter will iterate over the selected rows.
+/// The strategy is chosen based on the selectivity of the filter.
+#[derive(Debug, Clone)]
+pub enum IterationStrategy {
+    /// A lazily evaluated iterator of ranges (slices)
+    ///
+    /// Best for low selectivity filters
     SlicesIterator,
     /// A lazily evaluated iterator of indices
+    ///
+    /// Best for high selectivity filters

Review Comment:
   ```suggestion
       /// Best for high selectivity filters (which select a relatively low number of rows)
   ```



##########
arrow-select/src/coalesce.rs:
##########
@@ -571,6 +788,15 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
     /// Return an error if the source array is not set
     fn copy_rows(&mut self, offset: usize, len: usize) -> Result<(), ArrowError>;
 
+    /// Copy rows at the given indices from the current source array into the in-progress array
+    fn copy_rows_by_filter(&mut self, filter: &FilterPredicate) -> Result<(), ArrowError> {
+        // Default implementation: iterate over indices from the filter

Review Comment:
   It seems like as a follow on we should implement something similar for the byte array filter types? If that is true I'll file a ticket



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2616173620


##########
arrow-buffer/src/builder/null.rs:
##########
@@ -193,6 +193,85 @@ impl NullBufferBuilder {
         }
     }
 
+    /// Extends this builder with validity values.
+    ///
+    ///
+    /// # Example
+    /// ```
+    /// # use arrow_buffer::NullBufferBuilder;
+    /// let mut builder = NullBufferBuilder::new(8);
+    /// let validities = [true, false, true, true];
+    /// builder.extend(validities.iter().copied());
+    /// assert_eq!(builder.len(), 4);
+    /// ```
+    pub fn extend<I: Iterator<Item = bool>>(&mut self, iter: I) {
+        let (lower, upper) = iter.size_hint();

Review Comment:
   Sounds good, I'll do that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2618385313


##########
arrow-buffer/src/builder/null.rs:
##########
@@ -193,6 +193,85 @@ impl NullBufferBuilder {
         }
     }
 
+    /// Extends this builder with validity values.
+    ///
+    ///
+    /// # Example
+    /// ```
+    /// # use arrow_buffer::NullBufferBuilder;
+    /// let mut builder = NullBufferBuilder::new(8);
+    /// let validities = [true, false, true, true];
+    /// builder.extend(validities.iter().copied());
+    /// assert_eq!(builder.len(), 4);
+    /// ```
+    pub fn extend<I: Iterator<Item = bool>>(&mut self, iter: I) {
+        let (lower, upper) = iter.size_hint();
+        let len = upper.expect("Iterator must have exact size_hint");
+        debug_assert_eq!(lower, len, "Iterator must have exact size_hint");
+
+        if len == 0 {
+            return;
+        }
+
+        // Materialize since we're about to append bits
+        self.materialize_if_needed();
+
+        let buf = self.bitmap_builder.as_mut().unwrap();
+        let start_len = buf.len();
+        // Advance to allocate space, initializing new bits to 0
+        buf.advance(len);
+
+        let slice = buf.as_slice_mut();
+        let mut bit_idx = start_len;
+        let end_bit = start_len + len;
+
+        // Process in chunks of 64 bits when byte-aligned for better performance

Review Comment:
   I just checked - this seems an additional ~30% improvement for null handling:
   
   ```
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1
                           time:   [2.4060 ms 2.4096 ms 2.4133 ms]
                           change: [−33.920% −32.902% −32.274%] (p = 0.00 < 0.05)
                           Performance has improved.
   Found 1 outliers among 100 measurements (1.00%)
     1 (1.00%) high mild
   
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8
                           time:   [2.1610 ms 2.1666 ms 2.1728 ms]
                           change: [−29.488% −28.499% −27.767%] (p = 0.00 < 0.05)
                           Performance has improved.
   Found 7 outliers among 100 measurements (7.00%)
     4 (4.00%) high mild
     3 (3.00%) high severe
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3654463272

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (b23524363f6297f2f4444fce3bd901e0b0ae6149) to ed9efe78e4cc958cc96707557818e754419debb0 [diff](https://github.com/apache/arrow-rs/compare/ed9efe78e4cc958cc96707557818e754419debb0..b23524363f6297f2f4444fce3bd901e0b0ae6149)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763798580

   > I am about out of time today, but here is what I was thinking both for testing and to support multi-column batches (it doesn't pass tests yet)
   > 
   > * [Implement fallback copy_with_filter Dandandan/arrow-rs#13](https://github.com/Dandandan/arrow-rs/pull/13)
   
   Thanks, I'll take a look <3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2702243507


##########
arrow-select/src/coalesce/byte_view.rs:
##########
@@ -279,18 +280,34 @@ impl<B: ByteViewType> InProgressByteViewArray<B> {
 }
 
 impl<B: ByteViewType> InProgressArray for InProgressByteViewArray<B> {
-    fn set_source(&mut self, source: Option<ArrayRef>) {
+    fn set_source(&mut self, source: Option<ArrayRef>, selectivity: Option<f64>) {
         self.source = source.map(|array| {
             let s = array.as_byte_view::<B>();
 
             let (need_gc, ideal_buffer_size) = if s.data_buffers().is_empty() {
                 (false, 0)
             } else {
+                // If selectivity is very low, it's almost certain that we want to GC
+                // because we are only taking a tiny fraction of the data.
+                // In this case, we avoid the expensive O(N) total_buffer_bytes_used() call.
+                //
+                // We use a threshold of 0.1 (10% selectivity).
+                if let Some(sel) = selectivity {
+                    if sel < 0.1 {

Review Comment:
   Probably  use 0.5 to make it sort of equivalent to the heuristic below?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763268667

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (9773da741e264ec4ed26afaab856959e4cf0f0db) to 1db1a8869cceb179aa885ed58da9f0b49c03eafe [diff](https://github.com/apache/arrow-rs/compare/1db1a8869cceb179aa885ed58da9f0b49c03eafe..9773da741e264ec4ed26afaab856959e4cf0f0db)
   BENCH_NAME=filter_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench filter_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763247485

   BTW I am working on tests and a more thorough review of this PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763394227

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                         coalesce_batches_filter                main
   -----                                                                         -----------------------                ----
   filter context decimal128 (kept 1/2)                                          1.00     45.0±4.74µs        ? ?/sec    1.02     45.9±6.40µs        ? ?/sec
   filter context decimal128 high selectivity (kept 1023/1024)                   1.04     51.2±1.19µs        ? ?/sec    1.00     49.3±1.35µs        ? ?/sec
   filter context decimal128 low selectivity (kept 1/1024)                       1.01    232.5±2.94ns        ? ?/sec    1.00    229.6±5.50ns        ? ?/sec
   filter context f32 (kept 1/2)                                                 1.00     77.9±1.18µs        ? ?/sec    1.12     87.3±0.36µs        ? ?/sec
   filter context f32 high selectivity (kept 1023/1024)                          1.00     10.4±0.58µs        ? ?/sec    1.00     10.5±0.54µs        ? ?/sec
   filter context f32 low selectivity (kept 1/1024)                              1.06    442.8±8.24ns        ? ?/sec    1.00    419.0±4.99ns        ? ?/sec
   filter context fsb with value length 20 (kept 1/2)                            1.00     60.7±0.88µs        ? ?/sec    1.16     70.7±0.88µs        ? ?/sec
   filter context fsb with value length 20 high selectivity (kept 1023/1024)     1.00     60.6±0.72µs        ? ?/sec    1.16     70.6±0.58µs        ? ?/sec
   filter context fsb with value length 20 low selectivity (kept 1/1024)         1.00     60.6±0.61µs        ? ?/sec    1.17     70.8±1.16µs        ? ?/sec
   filter context fsb with value length 5 (kept 1/2)                             1.00     60.6±0.42µs        ? ?/sec    1.17     70.7±0.67µs        ? ?/sec
   filter context fsb with value length 5 high selectivity (kept 1023/1024)      1.00     60.8±0.77µs        ? ?/sec    1.16     70.5±0.38µs        ? ?/sec
   filter context fsb with value length 5 low selectivity (kept 1/1024)          1.00     60.6±0.24µs        ? ?/sec    1.17     70.8±1.47µs        ? ?/sec
   filter context fsb with value length 50 (kept 1/2)                            1.00     60.7±0.48µs        ? ?/sec    1.16     70.5±0.26µs        ? ?/sec
   filter context fsb with value length 50 high selectivity (kept 1023/1024)     1.00     60.5±0.16µs        ? ?/sec    1.17     70.6±0.28µs        ? ?/sec
   filter context fsb with value length 50 low selectivity (kept 1/1024)         1.00     60.7±0.55µs        ? ?/sec    1.17     71.0±2.85µs        ? ?/sec
   filter context i32 (kept 1/2)                                                 1.01     16.5±0.10µs        ? ?/sec    1.00     16.4±0.10µs        ? ?/sec
   filter context i32 high selectivity (kept 1023/1024)                          1.00      6.6±0.57µs        ? ?/sec    1.02      6.8±0.51µs        ? ?/sec
   filter context i32 low selectivity (kept 1/1024)                              1.00    219.7±2.40ns        ? ?/sec    1.00    219.3±1.44ns        ? ?/sec
   filter context i32 w NULLs (kept 1/2)                                         1.00     77.6±0.46µs        ? ?/sec    1.13     87.4±0.40µs        ? ?/sec
   filter context i32 w NULLs high selectivity (kept 1023/1024)                  1.00     10.4±0.53µs        ? ?/sec    1.04     10.8±0.54µs        ? ?/sec
   filter context i32 w NULLs low selectivity (kept 1/1024)                      1.04    442.0±7.30ns        ? ?/sec    1.00   423.2±19.31ns        ? ?/sec
   filter context mixed string view (kept 1/2)                                   1.00    106.1±6.85µs        ? ?/sec    1.16    122.9±5.98µs        ? ?/sec
   filter context mixed string view high selectivity (kept 1023/1024)            1.01     55.8±0.87µs        ? ?/sec    1.00     55.1±1.88µs        ? ?/sec
   filter context mixed string view low selectivity (kept 1/1024)                1.06   641.8±11.18ns        ? ?/sec    1.00    603.2±2.19ns        ? ?/sec
   filter context short string view (kept 1/2)                                   1.00    109.3±6.75µs        ? ?/sec    1.07    117.4±7.72µs        ? ?/sec
   filter context short string view high selectivity (kept 1023/1024)            1.00     54.0±1.61µs        ? ?/sec    1.01     54.7±1.33µs        ? ?/sec
   filter context short string view low selectivity (kept 1/1024)                1.05    500.4±5.49ns        ? ?/sec    1.00   478.2±14.02ns        ? ?/sec
   filter context string (kept 1/2)                                              1.02   584.5±18.08µs        ? ?/sec    1.00   575.0±16.63µs        ? ?/sec
   filter context string dictionary (kept 1/2)                                   1.02     17.1±0.20µs        ? ?/sec    1.00     16.8±0.08µs        ? ?/sec
   filter context string dictionary high selectivity (kept 1023/1024)            1.00      7.0±0.56µs        ? ?/sec    1.06      7.4±0.48µs        ? ?/sec
   filter context string dictionary low selectivity (kept 1/1024)                1.00    631.9±9.12ns        ? ?/sec    1.00    629.2±4.30ns        ? ?/sec
   filter context string dictionary w NULLs (kept 1/2)                           1.00     78.7±3.84µs        ? ?/sec    1.12     88.1±0.88µs        ? ?/sec
   filter context string dictionary w NULLs high selectivity (kept 1023/1024)    1.00     10.9±0.57µs        ? ?/sec    1.01     11.1±0.54µs        ? ?/sec
   filter context string dictionary w NULLs low selectivity (kept 1/1024)        1.02   859.6±11.70ns        ? ?/sec    1.00    846.9±3.94ns        ? ?/sec
   filter context string high selectivity (kept 1023/1024)                       1.00   634.4±11.19µs        ? ?/sec    1.00   636.4±19.16µs        ? ?/sec
   filter context string low selectivity (kept 1/1024)                           1.10   916.4±18.75ns        ? ?/sec    1.00    835.7±8.71ns        ? ?/sec
   filter context u8 (kept 1/2)                                                  1.00     15.0±0.12µs        ? ?/sec    1.00     15.0±0.17µs        ? ?/sec
   filter context u8 high selectivity (kept 1023/1024)                           1.00  1797.8±12.44ns        ? ?/sec    1.11      2.0±0.01µs        ? ?/sec
   filter context u8 low selectivity (kept 1/1024)                               1.01    213.7±1.66ns        ? ?/sec    1.00    211.0±1.34ns        ? ?/sec
   filter context u8 w NULLs (kept 1/2)                                          1.00     75.9±0.27µs        ? ?/sec    1.13     85.9±0.39µs        ? ?/sec
   filter context u8 w NULLs high selectivity (kept 1023/1024)                   1.02      5.3±0.06µs        ? ?/sec    1.00      5.2±0.03µs        ? ?/sec
   filter context u8 w NULLs low selectivity (kept 1/1024)                       1.05    436.0±2.97ns        ? ?/sec    1.00    414.7±2.28ns        ? ?/sec
   filter decimal128 (kept 1/2)                                                  1.00     52.9±5.19µs        ? ?/sec    1.05     55.4±4.40µs        ? ?/sec
   filter decimal128 high selectivity (kept 1023/1024)                           1.00     53.0±1.73µs        ? ?/sec    1.00     53.1±1.24µs        ? ?/sec
   filter decimal128 low selectivity (kept 1/1024)                               1.08      3.1±0.03µs        ? ?/sec    1.00      2.9±0.05µs        ? ?/sec
   filter f32 (kept 1/2)                                                         1.07    148.9±2.01µs        ? ?/sec    1.00    139.2±2.72µs        ? ?/sec
   filter fsb with value length 20 (kept 1/2)                                    1.09    154.6±3.60µs        ? ?/sec    1.00    142.2±1.74µs        ? ?/sec
   filter fsb with value length 20 high selectivity (kept 1023/1024)             1.00     72.3±2.49µs        ? ?/sec    1.00     72.5±2.18µs        ? ?/sec
   filter fsb with value length 20 low selectivity (kept 1/1024)                 1.22      3.2±0.03µs        ? ?/sec    1.00      2.7±0.01µs        ? ?/sec
   filter fsb with value length 5 (kept 1/2)                                     1.13    159.9±1.50µs        ? ?/sec    1.00    141.9±3.20µs        ? ?/sec
   filter fsb with value length 5 high selectivity (kept 1023/1024)              1.00     11.9±0.58µs        ? ?/sec    1.00     12.0±0.78µs        ? ?/sec
   filter fsb with value length 5 low selectivity (kept 1/1024)                  1.22      3.2±0.04µs        ? ?/sec    1.00      2.6±0.05µs        ? ?/sec
   filter fsb with value length 50 (kept 1/2)                                    1.02   174.8±13.09µs        ? ?/sec    1.00   171.6±16.73µs        ? ?/sec
   filter fsb with value length 50 high selectivity (kept 1023/1024)             1.02    216.9±7.61µs        ? ?/sec    1.00    213.2±8.51µs        ? ?/sec
   filter fsb with value length 50 low selectivity (kept 1/1024)                 1.20      3.2±0.03µs        ? ?/sec    1.00      2.7±0.08µs        ? ?/sec
   filter i32 (kept 1/2)                                                         1.05     45.3±0.26µs        ? ?/sec    1.00     43.0±0.21µs        ? ?/sec
   filter i32 high selectivity (kept 1023/1024)                                  1.00      9.3±0.52µs        ? ?/sec    1.01      9.3±0.60µs        ? ?/sec
   filter i32 low selectivity (kept 1/1024)                                      1.23      2.9±0.04µs        ? ?/sec    1.00      2.4±0.04µs        ? ?/sec
   filter optimize (kept 1/2)                                                    1.00     54.4±0.40µs        ? ?/sec    1.00     54.4±0.18µs        ? ?/sec
   filter optimize high selectivity (kept 1023/1024)                             1.01      3.0±0.03µs        ? ?/sec    1.00      3.0±0.02µs        ? ?/sec
   filter optimize low selectivity (kept 1/1024)                                 1.12      3.1±0.01µs        ? ?/sec    1.00      2.8±0.04µs        ? ?/sec
   filter run array (kept 1/2)                                                   1.00    372.9±3.94µs        ? ?/sec    1.00    371.1±2.17µs        ? ?/sec
   filter run array high selectivity (kept 1023/1024)                            1.00    398.3±4.80µs        ? ?/sec    1.00    397.1±1.32µs        ? ?/sec
   filter run array low selectivity (kept 1/1024)                                1.00    283.4±1.48µs        ? ?/sec    1.00    282.7±4.14µs        ? ?/sec
   filter single record batch                                                    1.21     53.5±0.28µs        ? ?/sec    1.00     44.4±0.28µs        ? ?/sec
   filter u8 (kept 1/2)                                                          1.01     45.8±0.28µs        ? ?/sec    1.00     45.5±0.13µs        ? ?/sec
   filter u8 high selectivity (kept 1023/1024)                                   1.00      4.0±0.04µs        ? ?/sec    1.00      4.0±0.04µs        ? ?/sec
   filter u8 low selectivity (kept 1/1024)                                       1.12      3.3±0.03µs        ? ?/sec    1.00      3.0±0.07µs        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763386077

   > run benchmark filter_kernels
   
   I expect the filter kernels should not be changed, except for the 80-90% threshold for using the sliced kernel.
   It might make sense to add some ticket benchmarking the threshold, I think it's likely the optimal threshold is even a bit higher.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2702698934


##########
arrow-select/src/coalesce/primitive.rs:
##########
@@ -95,6 +96,151 @@ impl<T: ArrowPrimitiveType + Debug> InProgressArray for InProgressPrimitiveArray
         Ok(())
     }
 
+    /// Copy rows using a predicate
+    fn copy_rows_by_filter(
+        &mut self,
+        filter: &FilterPredicate,
+        offset: usize,
+        len: usize,
+    ) -> Result<(), ArrowError> {
+        self.ensure_capacity();
+
+        let s = self
+            .source
+            .as_ref()
+            .ok_or_else(|| {
+                ArrowError::InvalidArgumentError(
+                    "Internal Error: InProgressPrimitiveArray: source not set".to_string(),
+                )
+            })?
+            .slice(offset, len);
+        let s = s.as_primitive::<T>();
+
+        let values = s.values();
+        let count = filter.count();
+
+        // Use the predicate's strategy for optimal iteration
+        match filter.strategy() {
+            IterationStrategy::SlicesIterator => {
+                // Copy values, nulls using slices
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    for (start, end) in SlicesIterator::new(filter.filter_array()) {
+                        // SAFETY: slices are derived from filter predicate
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                        let slice = nulls.slice(start, end - start);
+                        self.nulls.append_buffer(&slice);
+                    }
+                } else {
+                    for (start, end) in SlicesIterator::new(filter.filter_array()) {
+                        // SAFETY: SlicesIterator produces valid ranges derived from filter
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                    }
+                    self.nulls.append_n_non_nulls(count);
+                }
+            }
+            IterationStrategy::Slices(slices) => {
+                // Copy values and nulls using precomputed slices - single iteration
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    for &(start, end) in slices {
+                        // SAFETY: slices are derived from filter predicate
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                        let slice = nulls.slice(start, end - start);
+                        self.nulls.append_buffer(&slice);
+                    }
+                } else {
+                    for &(start, end) in slices {
+                        // SAFETY: slices are derived from filter predicate
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                    }
+                    self.nulls.append_n_non_nulls(count);
+                }
+            }
+            IterationStrategy::IndexIterator => {
+                // Copy values and nulls for each index
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    let null_buffer = nulls.inner();
+                    let null_ptr = null_buffer.values().as_ptr();
+                    let null_offset = null_buffer.offset();
+
+                    // Collect indices for reuse (values + nulls)
+                    let indices = IndexIterator::new(filter.filter_array(), count);
+
+                    // Efficiently extend null buffer
+                    // SAFETY: indices iterator reports correct length
+                    unsafe {
+                        self.nulls.extend_trusted_len(
+                            indices.map(|idx| bit_util::get_bit_raw(null_ptr, idx + null_offset)),
+                        );
+                    }
+                    let indices = IndexIterator::new(filter.filter_array(), count);
+
+                    // Copy values
+                    // SAFETY: indices are derived from filter predicate
+                    self.current
+                        .extend(indices.map(|idx| unsafe { *values.get_unchecked(idx) }));
+                } else {
+                    self.nulls.append_n_non_nulls(count);
+                    let indices = IndexIterator::new(filter.filter_array(), count);
+                    // SAFETY: indices are derived from filter predicate
+                    self.current
+                        .extend(indices.map(|idx: usize| unsafe { *values.get_unchecked(idx) }));
+                }
+            }
+            IterationStrategy::Indices(indices) => {
+                // Copy values and nulls using precomputed indices
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    let null_buffer = nulls.inner();
+                    let null_ptr = null_buffer.values().as_ptr();
+                    let null_offset = null_buffer.offset();
+
+                    // Efficiently extend null buffer
+                    // SAFETY: indices iterator reports correct length
+                    unsafe {
+                        self.nulls.extend_trusted_len(
+                            indices
+                                .iter()
+                                .map(|&idx| bit_util::get_bit_raw(null_ptr, idx + null_offset)),
+                        );
+                    }
+
+                    // Copy values
+                    // SAFETY: indices are derived from filter predicate
+                    self.current.extend(
+                        indices
+                            .iter()
+                            .map(|&idx| unsafe { *values.get_unchecked(idx) }),
+                    );
+                } else {
+                    self.nulls.append_n_non_nulls(count);
+                    // SAFETY: indices are derived from filter predicate
+                    self.current.extend(
+                        indices
+                            .iter()
+                            .map(|&idx| unsafe { *values.get_unchecked(idx) }),
+                    )
+                };
+            }
+            IterationStrategy::All => {
+                // Copy all values

Review Comment:
   While trying to write a test to cover this code, I am pretty sure this is unreachable (as if the entire batch is selected, then the fast path above like this is executed:
   
   ```rust
           // Fast path: if all rows selected, just push the batch
           if selected_count == num_rows {
               return self.push_batch(batch);
           }
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2702393701


##########
arrow-select/src/coalesce/primitive.rs:
##########
@@ -95,6 +96,151 @@ impl<T: ArrowPrimitiveType + Debug> InProgressArray for InProgressPrimitiveArray
         Ok(())
     }
 
+    /// Copy rows using a predicate
+    fn copy_rows_by_filter(
+        &mut self,
+        filter: &FilterPredicate,
+        offset: usize,
+        len: usize,
+    ) -> Result<(), ArrowError> {
+        self.ensure_capacity();
+
+        let s = self
+            .source
+            .as_ref()
+            .ok_or_else(|| {
+                ArrowError::InvalidArgumentError(
+                    "Internal Error: InProgressPrimitiveArray: source not set".to_string(),
+                )
+            })?
+            .slice(offset, len);
+        let s = s.as_primitive::<T>();
+
+        let values = s.values();
+        let count = filter.count();
+
+        // Use the predicate's strategy for optimal iteration
+        match filter.strategy() {
+            IterationStrategy::SlicesIterator => {
+                // Copy values, nulls using slices
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    for (start, end) in SlicesIterator::new(filter.filter_array()) {
+                        // SAFETY: slices are derived from filter predicate
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                        let slice = nulls.slice(start, end - start);
+                        self.nulls.append_buffer(&slice);
+                    }
+                } else {
+                    for (start, end) in SlicesIterator::new(filter.filter_array()) {
+                        // SAFETY: SlicesIterator produces valid ranges derived from filter
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                    }
+                    self.nulls.append_n_non_nulls(count);
+                }
+            }
+            IterationStrategy::Slices(slices) => {
+                // Copy values and nulls using precomputed slices - single iteration
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    for &(start, end) in slices {
+                        // SAFETY: slices are derived from filter predicate
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                        let slice = nulls.slice(start, end - start);
+                        self.nulls.append_buffer(&slice);
+                    }
+                } else {
+                    for &(start, end) in slices {
+                        // SAFETY: slices are derived from filter predicate
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                    }
+                    self.nulls.append_n_non_nulls(count);
+                }
+            }
+            IterationStrategy::IndexIterator => {
+                // Copy values and nulls for each index
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    let null_buffer = nulls.inner();
+                    let null_ptr = null_buffer.values().as_ptr();
+                    let null_offset = null_buffer.offset();
+
+                    // Collect indices for reuse (values + nulls)
+                    let indices = IndexIterator::new(filter.filter_array(), count);
+
+                    // Efficiently extend null buffer
+                    // SAFETY: indices iterator reports correct length
+                    unsafe {
+                        self.nulls.extend_trusted_len(
+                            indices.map(|idx| bit_util::get_bit_raw(null_ptr, idx + null_offset)),
+                        );
+                    }
+                    let indices = IndexIterator::new(filter.filter_array(), count);
+
+                    // Copy values
+                    // SAFETY: indices are derived from filter predicate
+                    self.current
+                        .extend(indices.map(|idx| unsafe { *values.get_unchecked(idx) }));
+                } else {
+                    self.nulls.append_n_non_nulls(count);
+                    let indices = IndexIterator::new(filter.filter_array(), count);
+                    // SAFETY: indices are derived from filter predicate
+                    self.current
+                        .extend(indices.map(|idx: usize| unsafe { *values.get_unchecked(idx) }));
+                }
+            }
+            IterationStrategy::Indices(indices) => {
+                // Copy values and nulls using precomputed indices
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    let null_buffer = nulls.inner();
+                    let null_ptr = null_buffer.values().as_ptr();
+                    let null_offset = null_buffer.offset();
+
+                    // Efficiently extend null buffer
+                    // SAFETY: indices iterator reports correct length
+                    unsafe {
+                        self.nulls.extend_trusted_len(
+                            indices
+                                .iter()
+                                .map(|&idx| bit_util::get_bit_raw(null_ptr, idx + null_offset)),
+                        );
+                    }
+
+                    // Copy values
+                    // SAFETY: indices are derived from filter predicate
+                    self.current.extend(
+                        indices
+                            .iter()
+                            .map(|&idx| unsafe { *values.get_unchecked(idx) }),
+                    );

Review Comment:
   This can be extracted out of the if for the nulls and then you don't need to have the same thing in the `else`.
   
   
   Same as the rest



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3765683295

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                 main
   -----                                                                                -----------------------                 ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    248.7±2.68ms        ? ?/sec     1.00    249.8±3.75ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      7.7±0.09ms        ? ?/sec     1.04      8.1±0.23ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      3.9±0.09ms        ? ?/sec     1.01      3.9±0.13ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.1±0.02ms        ? ?/sec     1.08      3.4±0.04ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.01    236.5±3.09ms        ? ?/sec     1.00    233.6±2.75ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.8±0.11ms        ? ?/sec     1.01      8.9±0.09ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.00      4.2±0.07ms        ? ?/sec     1.06      4.5±0.10ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.5±0.07ms        ? ?/sec     1.28      4.4±0.07ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     54.0±0.99ms        ? ?/sec     1.05     56.6±0.84ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     10.9±0.22ms        ? ?/sec     1.02     11.2±0.26ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.1±0.31ms        ? ?/sec     1.00      9.1±0.38ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.7±0.16ms        ? ?/sec     1.28      9.9±0.16ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.02     66.9±0.42ms        ? ?/sec     1.00     65.9±0.58ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.3±0.12ms        ? ?/sec     1.00     12.3±0.16ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.1±0.22ms        ? ?/sec     1.04      9.4±0.38ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      7.8±0.18ms        ? ?/sec     1.20      9.3±0.18ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     44.6±0.39ms        ? ?/sec     1.00     44.8±0.51ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      5.3±0.17ms        ? ?/sec     1.05      5.5±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.00      4.2±0.21ms        ? ?/sec     1.05      4.4±0.22ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00  1063.8±12.19µs        ? ?/sec     2.64      2.8±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.09     59.4±0.58ms        ? ?/sec     1.00     54.5±0.70ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.05      7.8±0.05ms        ? ?/sec     1.00      7.5±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.00      4.9±0.16ms        ? ?/sec     1.06      5.2±0.20ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00  1972.9±118.58µs        ? ?/sec    1.83      3.6±0.11ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     39.2±0.53ms        ? ?/sec     1.03     40.2±0.28ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      4.1±0.04ms        ? ?/sec     1.05      4.3±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.05      2.2±0.23ms        ? ?/sec     1.00      2.1±0.13ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1049.6±18.68µs        ? ?/sec     1.24  1306.4±13.90µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.11     55.2±0.28ms        ? ?/sec     1.00     49.7±0.33ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.05      6.8±0.04ms        ? ?/sec     1.00      6.5±0.06ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.00      3.2±0.17ms        ? ?/sec     1.09      3.5±0.21ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00  1948.4±31.63µs        ? ?/sec     1.87      3.6±0.13ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     72.2±0.63ms        ? ?/sec     1.25     90.1±0.95ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.9±0.13ms        ? ?/sec     1.23      8.5±0.04ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.2±0.30ms        ? ?/sec     1.02      3.3±0.20ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1784.8±14.16µs        ? ?/sec     1.58      2.8±0.03ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    109.8±1.43ms        ? ?/sec     1.08    118.2±1.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     13.0±0.16ms        ? ?/sec     1.09     14.2±0.36ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      5.6±0.37ms        ? ?/sec     1.21      6.8±0.36ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      3.5±0.07ms        ? ?/sec     2.42      8.6±0.07ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.21     77.3±0.64ms        ? ?/sec     1.00     63.8±1.53ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.13      8.1±0.07ms        ? ?/sec     1.00      7.2±0.03ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.03      4.0±0.33ms        ? ?/sec     1.00      3.9±0.25ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.10  1447.1±31.40µs        ? ?/sec     1.00  1310.9±11.39µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.35    110.3±2.24ms        ? ?/sec     1.00     81.7±0.39ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.16     12.8±0.41ms        ? ?/sec     1.00     11.0±0.13ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.00      5.2±0.22ms        ? ?/sec     1.03      5.4±0.34ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      2.9±0.05ms        ? ?/sec     1.29      3.7±0.09ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771682719

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771691840

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771674676

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771700003

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771694168

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771700429

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771792926

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3774480941

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    254.9±2.59ms        ? ?/sec    1.02    259.0±3.61ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      8.0±0.13ms        ? ?/sec    1.04      8.3±0.13ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      4.0±0.09ms        ? ?/sec    1.01      4.0±0.06ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.2±0.05ms        ? ?/sec    1.10      3.5±0.08ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.02    242.7±2.85ms        ? ?/sec    1.00    237.6±3.16ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.9±0.07ms        ? ?/sec    1.02      9.1±0.26ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.03      4.5±0.17ms        ? ?/sec    1.00      4.4±0.14ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.6±0.04ms        ? ?/sec    1.28      4.6±0.08ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     53.3±1.27ms        ? ?/sec    1.03     54.9±0.75ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     11.1±0.32ms        ? ?/sec    1.03     11.3±0.20ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.3±0.15ms        ? ?/sec    1.00      9.3±0.23ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.8±0.33ms        ? ?/sec    1.44     11.2±0.89ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.01     66.3±1.66ms        ? ?/sec    1.00     65.8±1.89ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.5±0.15ms        ? ?/sec    1.00     12.4±0.29ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.7±0.35ms        ? ?/sec    1.03      9.9±0.34ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      8.7±0.53ms        ? ?/sec    1.10      9.5±0.16ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     28.3±0.28ms        ? ?/sec    1.52     43.0±0.62ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.6±0.05ms        ? ?/sec    1.20      5.6±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.15      5.2±0.15ms        ? ?/sec    1.00      4.5±0.17ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00      2.9±0.13ms        ? ?/sec    1.00      2.9±0.07ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     46.8±0.24ms        ? ?/sec    1.14     53.3±0.46ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      7.3±0.14ms        ? ?/sec    1.04      7.6±0.17ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.14      6.1±0.14ms        ? ?/sec    1.00      5.4±0.17ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.0±0.03ms        ? ?/sec    1.85      3.7±0.02ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     27.1±0.25ms        ? ?/sec    1.42     38.6±0.74ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.7±0.03ms        ? ?/sec    1.14      4.2±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.19      2.6±0.06ms        ? ?/sec    1.00      2.2±0.23ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1002.1±18.04µs        ? ?/sec    1.32  1318.2±16.16µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     44.7±0.26ms        ? ?/sec    1.07     47.9±0.35ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.4±0.07ms        ? ?/sec    1.01      6.4±0.08ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.15      3.9±0.12ms        ? ?/sec    1.00      3.4±0.15ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.2±0.02ms        ? ?/sec    1.73      3.8±0.05ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     68.8±1.94ms        ? ?/sec    1.24     85.1±0.58ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.9±0.05ms        ? ?/sec    1.22      8.4±0.10ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.04      3.6±0.25ms        ? ?/sec    1.00      3.4±0.37ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1515.7±17.60µs        ? ?/sec    1.84      2.8±0.03ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    106.0±1.07ms        ? ?/sec    1.07    113.4±1.28ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     13.0±0.11ms        ? ?/sec    1.07     13.9±0.27ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.7±0.39ms        ? ?/sec    1.01      6.7±0.34ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.4±0.05ms        ? ?/sec    2.05      8.9±0.07ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     52.6±0.31ms        ? ?/sec    1.21     63.6±0.88ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.02      7.6±0.07ms        ? ?/sec    1.00      7.5±0.06ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.54      6.1±0.26ms        ? ?/sec    1.00      4.0±0.36ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.57      2.1±0.06ms        ? ?/sec    1.00  1307.5±19.67µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.00     75.1±0.93ms        ? ?/sec    1.10     82.2±1.02ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     10.4±0.15ms        ? ?/sec    1.07     11.1±0.15ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.27      6.7±0.11ms        ? ?/sec    1.00      5.3±0.30ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      3.1±0.08ms        ? ?/sec    1.27      3.9±0.05ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3769147974

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    240.9±2.58ms        ? ?/sec    1.06    256.2±3.90ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      7.8±0.05ms        ? ?/sec    1.04      8.1±0.10ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      4.0±0.12ms        ? ?/sec    1.00      4.1±0.09ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.2±0.07ms        ? ?/sec    1.07      3.4±0.06ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    227.3±2.91ms        ? ?/sec    1.06    240.1±3.73ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.9±0.10ms        ? ?/sec    1.00      8.9±0.11ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.00      4.3±0.12ms        ? ?/sec    1.01      4.4±0.15ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.4±0.04ms        ? ?/sec    1.31      4.4±0.03ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     53.1±0.78ms        ? ?/sec    1.07     56.7±0.78ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     10.8±0.16ms        ? ?/sec    1.02     11.1±0.11ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.2±0.31ms        ? ?/sec    1.01      9.3±0.25ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.7±0.20ms        ? ?/sec    1.36     10.5±0.29ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     65.4±0.41ms        ? ?/sec    1.01     66.0±0.42ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.01     12.5±0.10ms        ? ?/sec    1.00     12.3±0.35ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.3±0.27ms        ? ?/sec    1.06      9.8±0.29ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      7.9±0.41ms        ? ?/sec    1.21      9.6±0.19ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     28.2±0.39ms        ? ?/sec    1.59     45.0±0.45ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.5±0.04ms        ? ?/sec    1.26      5.6±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.02      4.7±0.25ms        ? ?/sec    1.00      4.6±0.20ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00  1064.7±32.60µs        ? ?/sec    2.68      2.9±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     47.2±0.99ms        ? ?/sec    1.17     55.4±0.80ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      6.9±0.07ms        ? ?/sec    1.08      7.5±0.06ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.00      5.2±0.19ms        ? ?/sec    1.03      5.4±0.16ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.0±0.04ms        ? ?/sec    1.82      3.6±0.08ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     27.2±0.23ms        ? ?/sec    1.48     40.3±0.70ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.5±0.02ms        ? ?/sec    1.23      4.4±0.11ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.05      2.2±0.15ms        ? ?/sec    1.00      2.1±0.09ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1272.9±26.14µs        ? ?/sec    1.05  1334.0±29.38µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     45.4±0.18ms        ? ?/sec    1.10     50.1±0.47ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.2±0.05ms        ? ?/sec    1.06      6.6±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.00      3.3±0.20ms        ? ?/sec    1.06      3.5±0.21ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.2±0.03ms        ? ?/sec    1.67      3.6±0.07ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     69.2±0.71ms        ? ?/sec    1.31     90.6±1.22ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.9±0.02ms        ? ?/sec    1.25      8.6±0.27ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.4±0.31ms        ? ?/sec    1.06      3.6±0.38ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1729.8±23.12µs        ? ?/sec    1.63      2.8±0.07ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    106.5±1.54ms        ? ?/sec    1.11    118.6±1.98ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     13.1±0.35ms        ? ?/sec    1.07     14.1±0.43ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      5.8±0.46ms        ? ?/sec    1.12      6.5±0.18ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      3.5±0.06ms        ? ?/sec    2.47      8.6±0.06ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     46.9±0.37ms        ? ?/sec    1.34     62.9±0.66ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.00      6.4±0.18ms        ? ?/sec    1.14      7.2±0.05ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.17      4.4±0.19ms        ? ?/sec    1.00      3.8±0.29ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            2.19      2.9±0.04ms        ? ?/sec    1.00  1318.3±21.42µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.07     86.9±1.29ms        ? ?/sec    1.00     80.9±0.59ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     11.0±0.10ms        ? ?/sec    1.01     11.1±0.17ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.17      6.0±0.19ms        ? ?/sec    1.00      5.1±0.20ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.19      4.5±0.05ms        ? ?/sec    1.00      3.7±0.09ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3769720377

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                 main
   -----                                                                                -----------------------                 ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    239.6±3.77ms        ? ?/sec     1.04    249.4±2.27ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      7.8±0.11ms        ? ?/sec     1.04      8.1±0.07ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      4.0±0.08ms        ? ?/sec     1.00      3.9±0.13ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.2±0.03ms        ? ?/sec     1.06      3.4±0.04ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    225.6±2.60ms        ? ?/sec     1.04    234.2±2.98ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.7±0.05ms        ? ?/sec     1.02      8.9±0.17ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.00      4.3±0.12ms        ? ?/sec     1.03      4.4±0.12ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.4±0.05ms        ? ?/sec     1.29      4.4±0.07ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     52.7±0.92ms        ? ?/sec     1.08     56.8±0.61ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     10.7±0.23ms        ? ?/sec     1.04     11.1±0.14ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.2±0.27ms        ? ?/sec     1.01      9.3±0.33ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.5±0.14ms        ? ?/sec     1.33     10.0±0.16ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     65.5±0.43ms        ? ?/sec     1.01     66.0±0.39ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.3±0.12ms        ? ?/sec     1.01     12.4±0.14ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.2±0.29ms        ? ?/sec     1.03      9.5±0.32ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      7.8±0.16ms        ? ?/sec     1.19      9.3±0.17ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     28.0±0.35ms        ? ?/sec     1.61     45.0±0.28ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.4±0.06ms        ? ?/sec     1.27      5.6±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.20      5.3±0.21ms        ? ?/sec     1.00      4.4±0.22ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00  1006.7±26.80µs        ? ?/sec     2.82      2.8±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     46.5±0.18ms        ? ?/sec     1.19     55.2±1.19ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      6.9±0.04ms        ? ?/sec     1.09      7.5±0.09ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.06      5.8±0.21ms        ? ?/sec     1.00      5.4±0.25ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00  1960.7±129.89µs        ? ?/sec    1.84      3.6±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     26.8±0.20ms        ? ?/sec     1.52     40.6±0.49ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.4±0.03ms        ? ?/sec     1.29      4.3±0.08ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.19      2.6±0.11ms        ? ?/sec     1.00      2.2±0.19ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1254.4±14.45µs        ? ?/sec     1.05  1312.0±14.77µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     45.2±0.60ms        ? ?/sec     1.10     49.9±0.89ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.1±0.13ms        ? ?/sec     1.07      6.5±0.15ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.04      3.6±0.15ms        ? ?/sec     1.00      3.5±0.18ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.1±0.02ms        ? ?/sec     1.70      3.6±0.08ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     69.4±0.60ms        ? ?/sec     1.31     91.0±0.67ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.7±0.06ms        ? ?/sec     1.30      8.7±0.18ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.4±0.33ms        ? ?/sec     1.08      3.6±0.39ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1786.2±23.72µs        ? ?/sec     1.63      2.9±0.03ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    106.2±0.79ms        ? ?/sec     1.12    118.4±1.51ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     12.9±0.15ms        ? ?/sec     1.10     14.2±0.08ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      5.4±0.26ms        ? ?/sec     1.22      6.6±0.28ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      3.5±0.05ms        ? ?/sec     2.44      8.6±0.10ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     44.5±0.51ms        ? ?/sec     1.42     63.2±0.26ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.00      5.4±0.03ms        ? ?/sec     1.32      7.2±0.06ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.00      4.0±0.14ms        ? ?/sec     1.02      4.0±0.39ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.95      2.6±0.03ms        ? ?/sec     1.00  1309.6±59.13µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.03     83.8±0.53ms        ? ?/sec     1.00     81.4±2.11ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     10.1±0.07ms        ? ?/sec     1.09     11.1±0.05ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.06      5.6±0.22ms        ? ?/sec     1.00      5.3±0.32ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.12      4.2±0.06ms        ? ?/sec     1.00      3.7±0.05ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771704286

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771704739

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771714110

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771720323

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771770533

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771783407

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771782848

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771957831

   
   run benchmark coalesce_kernels
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3732104575

   Still have to allocate time to finish this up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-4259670594

   Marking as draft as I think this PR is no longer waiting on feedback and I am trying to make it easier to find PRs in need of review. Please mark it as ready for review when it is ready for another look 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2694135915


##########
arrow-select/src/coalesce/primitive.rs:
##########
@@ -95,6 +96,145 @@ impl<T: ArrowPrimitiveType + Debug> InProgressArray for InProgressPrimitiveArray
         Ok(())
     }
 
+    /// Copy rows using a predicate

Review Comment:
   BTW I think this PR may be related
   - https://github.com/apache/arrow-rs/pull/7771
   
   That was my attempt to avoid having to copy the logic from filter



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3765542519

   run benchmark coalesce_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771684343

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771703847

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771689020

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771694648

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771679099

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771710008

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771708632

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771714980

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771715937

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771763466

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771757263

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771719893

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771772095

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771776377

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771784462

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771785073

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771779098

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771787201

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771788896

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771798547

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771787785

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771800393

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3773553071

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    247.7±3.30ms        ? ?/sec    1.01    250.1±2.54ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      7.8±0.13ms        ? ?/sec    1.04      8.1±0.11ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      3.9±0.12ms        ? ?/sec    1.01      4.0±0.10ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.1±0.03ms        ? ?/sec    1.10      3.4±0.04ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.01    235.7±2.80ms        ? ?/sec    1.00    234.4±3.30ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.7±0.09ms        ? ?/sec    1.01      8.8±0.09ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.00      4.4±0.11ms        ? ?/sec    1.00      4.4±0.13ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.5±0.05ms        ? ?/sec    1.25      4.4±0.06ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     52.5±1.43ms        ? ?/sec    1.09     57.4±0.91ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     10.8±0.14ms        ? ?/sec    1.03     11.1±0.12ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.01      8.9±0.38ms        ? ?/sec    1.00      8.8±0.29ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.4±0.12ms        ? ?/sec    1.37     10.1±0.46ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     65.1±0.79ms        ? ?/sec    1.04     67.7±1.28ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.02     12.6±0.21ms        ? ?/sec    1.00     12.4±0.39ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.02      9.4±0.37ms        ? ?/sec    1.00      9.2±0.30ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      7.9±0.14ms        ? ?/sec    1.17      9.2±0.13ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     28.2±0.49ms        ? ?/sec    1.63     45.8±0.31ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.6±0.05ms        ? ?/sec    1.22      5.6±0.07ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.17      5.1±0.19ms        ? ?/sec    1.00      4.3±0.25ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00      2.7±0.07ms        ? ?/sec    1.03      2.8±0.02ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     46.4±0.40ms        ? ?/sec    1.21     56.1±0.42ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      7.2±0.13ms        ? ?/sec    1.05      7.6±0.22ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.18      6.0±0.24ms        ? ?/sec    1.00      5.1±0.28ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.1±0.04ms        ? ?/sec    1.74      3.6±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     26.9±0.37ms        ? ?/sec    1.55     41.5±0.36ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.7±0.03ms        ? ?/sec    1.21      4.4±0.03ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.27      2.8±0.18ms        ? ?/sec    1.00      2.2±0.15ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00   998.6±17.38µs        ? ?/sec    1.32  1316.0±19.98µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     44.6±0.38ms        ? ?/sec    1.15     51.1±1.13ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.4±0.23ms        ? ?/sec    1.03      6.6±0.29ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.22      4.1±0.15ms        ? ?/sec    1.00      3.3±0.18ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.2±0.02ms        ? ?/sec    1.65      3.6±0.08ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     68.1±1.18ms        ? ?/sec    1.38     93.9±0.96ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.8±0.04ms        ? ?/sec    1.29      8.8±0.08ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.5±0.39ms        ? ?/sec    1.00      3.5±0.30ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1514.9±33.93µs        ? ?/sec    1.89      2.9±0.03ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    105.2±0.60ms        ? ?/sec    1.15    121.2±1.38ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     12.9±0.07ms        ? ?/sec    1.09     14.2±0.13ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.04      6.6±0.44ms        ? ?/sec    1.00      6.4±0.37ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.3±0.11ms        ? ?/sec    1.95      8.5±0.06ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     52.1±0.14ms        ? ?/sec    1.23     64.0±0.37ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.03      7.6±0.03ms        ? ?/sec    1.00      7.3±0.11ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.60      6.1±0.11ms        ? ?/sec    1.00      3.8±0.29ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.58      2.1±0.05ms        ? ?/sec    1.00  1301.1±23.53µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.00     74.4±1.68ms        ? ?/sec    1.09     81.5±0.58ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     10.2±0.19ms        ? ?/sec    1.09     11.2±0.28ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.28      7.2±0.29ms        ? ?/sec    1.00      5.6±0.36ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      3.1±0.05ms        ? ?/sec    1.18      3.7±0.04ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3617296926

   ```
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     54.1±1.62ms        ? ?/sec    1.80     97.2±0.21ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      5.9±0.03ms        ? ?/sec    1.57      9.3±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.2±0.09ms        ? ?/sec    1.17      3.7±0.05ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00      2.7±0.01ms        ? ?/sec    1.14      3.1±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00     60.9±0.09ms        ? ?/sec    2.06    125.4±0.26ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     10.9±0.04ms        ? ?/sec    1.38     15.1±0.06ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.16      8.6±0.20ms        ? ?/sec    1.00      7.4±0.36ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.62     14.7±0.04ms        ? ?/sec    1.00      9.1±0.04ms        ? ?/sec
   ```
   
   Pretty good I would say... probably have to look a bit more at the null-handling speed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3643825662

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (bb025cf4db6c36e1f2144ec87a29f8374bd1d6c5) to ed9efe78e4cc958cc96707557818e754419debb0 [diff](https://github.com/apache/arrow-rs/compare/ed9efe78e4cc958cc96707557818e754419debb0..bb025cf4db6c36e1f2144ec87a29f8374bd1d6c5)
   BENCH_NAME=filter_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench filter_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3644009066

   > Hi @Dandandan -- I am working through the arrow-rs review backlog. coalesce benchmarks look better. Filter kernels look potentially slower. I'll rerun and try to see if we can reproduce the results
   
   I don't think the filter kernels should have any impact other than the threshold, but that isn't covered by a benchmark.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3644088331

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                         coalesce_batches_filter                main
   -----                                                                         -----------------------                ----
   filter context decimal128 (kept 1/2)                                          1.06     47.0±7.08µs        ? ?/sec    1.00     44.3±1.94µs        ? ?/sec
   filter context decimal128 high selectivity (kept 1023/1024)                   1.02     51.2±1.84µs        ? ?/sec    1.00     50.0±2.98µs        ? ?/sec
   filter context decimal128 low selectivity (kept 1/1024)                       1.00    241.4±2.88ns        ? ?/sec    1.00    240.8±4.26ns        ? ?/sec
   filter context f32 (kept 1/2)                                                 1.00     77.9±0.90µs        ? ?/sec    1.00     77.7±0.95µs        ? ?/sec
   filter context f32 high selectivity (kept 1023/1024)                          1.02     10.5±0.43µs        ? ?/sec    1.00     10.3±0.44µs        ? ?/sec
   filter context f32 low selectivity (kept 1/1024)                              1.00    442.0±7.36ns        ? ?/sec    1.03    454.6±1.42ns        ? ?/sec
   filter context fsb with value length 20 (kept 1/2)                            1.00     60.6±0.67µs        ? ?/sec    1.00     60.9±0.62µs        ? ?/sec
   filter context fsb with value length 20 high selectivity (kept 1023/1024)     1.00     60.7±0.74µs        ? ?/sec    1.00     60.8±0.52µs        ? ?/sec
   filter context fsb with value length 20 low selectivity (kept 1/1024)         1.00     60.6±0.30µs        ? ?/sec    1.00     60.8±0.61µs        ? ?/sec
   filter context fsb with value length 5 (kept 1/2)                             1.00     60.7±0.52µs        ? ?/sec    1.01     61.1±0.80µs        ? ?/sec
   filter context fsb with value length 5 high selectivity (kept 1023/1024)      1.00     60.9±1.82µs        ? ?/sec    1.00     60.7±0.56µs        ? ?/sec
   filter context fsb with value length 5 low selectivity (kept 1/1024)          1.00     60.7±0.43µs        ? ?/sec    1.00     60.6±0.16µs        ? ?/sec
   filter context fsb with value length 50 (kept 1/2)                            1.00     60.8±0.32µs        ? ?/sec    1.02     61.9±5.83µs        ? ?/sec
   filter context fsb with value length 50 high selectivity (kept 1023/1024)     1.00     60.9±1.96µs        ? ?/sec    1.00     60.7±0.69µs        ? ?/sec
   filter context fsb with value length 50 low selectivity (kept 1/1024)         1.00     60.6±0.42µs        ? ?/sec    1.00     60.7±0.34µs        ? ?/sec
   filter context i32 (kept 1/2)                                                 1.02     17.0±0.11µs        ? ?/sec    1.00     16.6±0.09µs        ? ?/sec
   filter context i32 high selectivity (kept 1023/1024)                          1.00      6.5±0.45µs        ? ?/sec    1.00      6.6±0.38µs        ? ?/sec
   filter context i32 low selectivity (kept 1/1024)                              1.02    239.3±1.79ns        ? ?/sec    1.00    234.5±2.04ns        ? ?/sec
   filter context i32 w NULLs (kept 1/2)                                         1.01     78.0±0.70µs        ? ?/sec    1.00     77.5±0.39µs        ? ?/sec
   filter context i32 w NULLs high selectivity (kept 1023/1024)                  1.05     10.6±0.52µs        ? ?/sec    1.00     10.1±0.53µs        ? ?/sec
   filter context i32 w NULLs low selectivity (kept 1/1024)                      1.00   450.4±20.45ns        ? ?/sec    1.03    463.5±3.66ns        ? ?/sec
   filter context mixed string view (kept 1/2)                                   1.01    105.2±5.62µs        ? ?/sec    1.00    104.2±4.47µs        ? ?/sec
   filter context mixed string view high selectivity (kept 1023/1024)            1.00     54.9±1.34µs        ? ?/sec    1.01     55.5±1.30µs        ? ?/sec
   filter context mixed string view low selectivity (kept 1/1024)                1.00    663.5±1.51ns        ? ?/sec    1.02    678.2±2.96ns        ? ?/sec
   filter context short string view (kept 1/2)                                   1.00    103.8±4.66µs        ? ?/sec    1.01    104.5±5.17µs        ? ?/sec
   filter context short string view high selectivity (kept 1023/1024)            1.00     53.3±1.67µs        ? ?/sec    1.04     55.5±1.91µs        ? ?/sec
   filter context short string view low selectivity (kept 1/1024)                1.00    471.2±8.20ns        ? ?/sec    1.01    476.8±5.12ns        ? ?/sec
   filter context string (kept 1/2)                                              1.00   576.6±11.44µs        ? ?/sec    1.01   584.2±12.53µs        ? ?/sec
   filter context string dictionary (kept 1/2)                                   1.00     17.0±0.13µs        ? ?/sec    1.01     17.1±0.11µs        ? ?/sec
   filter context string dictionary high selectivity (kept 1023/1024)            1.02      7.6±0.29µs        ? ?/sec    1.00      7.4±0.58µs        ? ?/sec
   filter context string dictionary low selectivity (kept 1/1024)                1.03    829.0±9.39ns        ? ?/sec    1.00    806.5±4.65ns        ? ?/sec
   filter context string dictionary w NULLs (kept 1/2)                           1.00     78.8±0.44µs        ? ?/sec    1.00     78.4±1.77µs        ? ?/sec
   filter context string dictionary w NULLs high selectivity (kept 1023/1024)    1.06     11.5±0.49µs        ? ?/sec    1.00     10.9±0.44µs        ? ?/sec
   filter context string dictionary w NULLs low selectivity (kept 1/1024)        1.00  1053.4±19.67ns        ? ?/sec    1.00  1054.2±15.33ns        ? ?/sec
   filter context string high selectivity (kept 1023/1024)                       1.04   683.6±37.71µs        ? ?/sec    1.00   659.9±22.70µs        ? ?/sec
   filter context string low selectivity (kept 1/1024)                           1.00   1045.0±4.96ns        ? ?/sec    1.01   1059.2±5.93ns        ? ?/sec
   filter context u8 (kept 1/2)                                                  1.00     15.0±0.22µs        ? ?/sec    1.00     15.0±0.09µs        ? ?/sec
   filter context u8 high selectivity (kept 1023/1024)                           1.00  1817.8±13.65ns        ? ?/sec    1.00  1809.0±22.61ns        ? ?/sec
   filter context u8 low selectivity (kept 1/1024)                               1.02    233.1±8.29ns        ? ?/sec    1.00    227.5±3.72ns        ? ?/sec
   filter context u8 w NULLs (kept 1/2)                                          1.00     76.2±0.83µs        ? ?/sec    1.00     75.9±0.33µs        ? ?/sec
   filter context u8 w NULLs high selectivity (kept 1023/1024)                   1.00      5.3±0.09µs        ? ?/sec    1.00      5.3±0.09µs        ? ?/sec
   filter context u8 w NULLs low selectivity (kept 1/1024)                       1.00    436.9±3.27ns        ? ?/sec    1.04    455.9±5.89ns        ? ?/sec
   filter decimal128 (kept 1/2)                                                  1.00     50.7±3.62µs        ? ?/sec    1.14     57.8±3.52µs        ? ?/sec
   filter decimal128 high selectivity (kept 1023/1024)                           1.01     53.4±1.60µs        ? ?/sec    1.00     52.6±1.58µs        ? ?/sec
   filter decimal128 low selectivity (kept 1/1024)                               1.00      3.2±0.01µs        ? ?/sec    1.02      3.2±0.03µs        ? ?/sec
   filter f32 (kept 1/2)                                                         1.01    157.2±0.54µs        ? ?/sec    1.00    155.8±1.49µs        ? ?/sec
   filter fsb with value length 20 (kept 1/2)                                    1.05    131.9±2.38µs        ? ?/sec    1.00    125.3±1.78µs        ? ?/sec
   filter fsb with value length 20 high selectivity (kept 1023/1024)             1.01     74.1±2.24µs        ? ?/sec    1.00     73.6±2.14µs        ? ?/sec
   filter fsb with value length 20 low selectivity (kept 1/1024)                 1.00      3.1±0.03µs        ? ?/sec    1.10      3.4±0.02µs        ? ?/sec
   filter fsb with value length 5 (kept 1/2)                                     1.13    137.3±0.78µs        ? ?/sec    1.00    121.4±0.88µs        ? ?/sec
   filter fsb with value length 5 high selectivity (kept 1023/1024)              1.06     11.7±0.67µs        ? ?/sec    1.00     11.1±0.61µs        ? ?/sec
   filter fsb with value length 5 low selectivity (kept 1/1024)                  1.00      3.1±0.03µs        ? ?/sec    1.08      3.4±0.01µs        ? ?/sec
   filter fsb with value length 50 (kept 1/2)                                    1.02   165.8±14.63µs        ? ?/sec    1.00   163.2±12.91µs        ? ?/sec
   filter fsb with value length 50 high selectivity (kept 1023/1024)             1.02    214.8±7.61µs        ? ?/sec    1.00    210.2±6.72µs        ? ?/sec
   filter fsb with value length 50 low selectivity (kept 1/1024)                 1.00      3.1±0.03µs        ? ?/sec    1.07      3.3±0.05µs        ? ?/sec
   filter i32 (kept 1/2)                                                         1.07     46.6±0.57µs        ? ?/sec    1.00     43.4±0.77µs        ? ?/sec
   filter i32 high selectivity (kept 1023/1024)                                  1.00      8.7±0.38µs        ? ?/sec    1.00      8.8±0.41µs        ? ?/sec
   filter i32 low selectivity (kept 1/1024)                                      1.39      3.3±0.02µs        ? ?/sec    1.00      2.4±0.01µs        ? ?/sec
   filter optimize (kept 1/2)                                                    1.00     54.5±0.44µs        ? ?/sec    1.00     54.4±0.93µs        ? ?/sec
   filter optimize high selectivity (kept 1023/1024)                             1.00      2.6±0.04µs        ? ?/sec    1.00      2.6±0.03µs        ? ?/sec
   filter optimize low selectivity (kept 1/1024)                                 1.10      3.1±0.09µs        ? ?/sec    1.00      2.8±0.14µs        ? ?/sec
   filter run array (kept 1/2)                                                   1.00    425.0±1.68µs        ? ?/sec    1.00    423.0±7.01µs        ? ?/sec
   filter run array high selectivity (kept 1023/1024)                            1.00    449.7±6.02µs        ? ?/sec    1.00    449.4±4.66µs        ? ?/sec
   filter run array low selectivity (kept 1/1024)                                1.00    336.6±2.12µs        ? ?/sec    1.00    335.4±2.43µs        ? ?/sec
   filter single record batch                                                    1.03     45.6±0.15µs        ? ?/sec    1.00     44.2±0.14µs        ? ?/sec
   filter u8 (kept 1/2)                                                          1.00     45.2±0.87µs        ? ?/sec    1.01     45.7±0.74µs        ? ?/sec
   filter u8 high selectivity (kept 1023/1024)                                   1.02      3.8±0.02µs        ? ?/sec    1.00      3.8±0.05µs        ? ?/sec
   filter u8 low selectivity (kept 1/1024)                                       1.00      3.3±0.07µs        ? ?/sec    1.00      3.3±0.07µs        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2701042297


##########
arrow-select/src/coalesce.rs:
##########
@@ -238,10 +245,112 @@ impl BatchCoalescer {
         batch: RecordBatch,
         filter: &BooleanArray,
     ) -> Result<(), ArrowError> {
-        // TODO: optimize this to avoid materializing (copying the results
-        // of filter to a new batch)
-        let filtered_batch = filter_record_batch(&batch, filter)?;
-        self.push_batch(filtered_batch)
+        // We only support primitve now, fallback to filter_record_batch for other types
+        // Also, skip optimization when filter is not very selective
+        if batch
+            .schema()
+            .fields()
+            .iter()
+            .any(|field| !field.data_type().is_primitive())

Review Comment:
   Because of the all-or-nothing check it has no effect on tpch/clickbench performance yet. I am not sure if we can somehow use the conventional filter kernel for non-supported arrays 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2701038833


##########
arrow-select/src/coalesce.rs:
##########
@@ -238,10 +245,112 @@ impl BatchCoalescer {
         batch: RecordBatch,
         filter: &BooleanArray,
     ) -> Result<(), ArrowError> {
-        // TODO: optimize this to avoid materializing (copying the results
-        // of filter to a new batch)
-        let filtered_batch = filter_record_batch(&batch, filter)?;
-        self.push_batch(filtered_batch)
+        // We only support primitve now, fallback to filter_record_batch for other types
+        // Also, skip optimization when filter is not very selective
+        if batch
+            .schema()
+            .fields()
+            .iter()
+            .any(|field| !field.data_type().is_primitive())
+            || self

Review Comment:
   I think benchmarks show it is better to just use the kernel always.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2702355920


##########
arrow-select/src/coalesce.rs:
##########
@@ -238,10 +242,100 @@ impl BatchCoalescer {
         batch: RecordBatch,
         filter: &BooleanArray,
     ) -> Result<(), ArrowError> {
-        // TODO: optimize this to avoid materializing (copying the results
-        // of filter to a new batch)
-        let filtered_batch = filter_record_batch(&batch, filter)?;
-        self.push_batch(filtered_batch)
+        // We only support primitve now, fallback to filter_record_batch for other types
+        // Also, skip optimization when filter is not very selectivex§
+
+        // Build an optimized filter predicate that chooses the best iteration strategy
+        let is_optimize_beneficial = is_optimize_beneficial_record_batch(&batch);
+        let selected_count = filter.true_count();
+        let num_rows = batch.num_rows();
+
+        // Fast path: skip if no rows selected
+        if selected_count == 0 {
+            return Ok(());
+        }
+
+        // Fast path: if all rows selected, just push the batch
+        if selected_count == num_rows {
+            return self.push_batch(batch);
+        }
+
+        let selectivity = Some(selected_count as f64 / num_rows as f64);
+        let (_schema, arrays, _num_rows) = batch.into_parts();
+
+        // Setup input arrays as sources
+        assert_eq!(arrays.len(), self.in_progress_arrays.len());
+        self.in_progress_arrays
+            .iter_mut()
+            .zip(&arrays)
+            .for_each(|(in_progress, array)| {
+                in_progress.set_source(Some(Arc::clone(array)), selectivity);
+            });
+
+        // Choose iteration strategy based on the optimized predicate
+        self.copy_from_filter(filter, is_optimize_beneficial, selected_count)?;
+
+        // Clear sources to allow memory to be freed
+        for in_progress in self.in_progress_arrays.iter_mut() {
+            in_progress.set_source(None, None);
+        }
+
+        Ok(())
+    }
+
+    /// Helper to copy rows at the given indices, handling batch boundaries efficiently
+    ///
+    /// This method batches the index iteration to avoid per-row batch boundary checks.
+    fn copy_from_filter(
+        &mut self,
+        filter: &BooleanArray,

Review Comment:
   Why getting `BooleanArray` and not `BooleanBuffer`? does nulls have meaning here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2702401478


##########
arrow-buffer/src/builder/null.rs:
##########
@@ -412,4 +439,47 @@ mod tests {
 
         assert_eq!(builder.finish(), None);
     }
+
+    #[test]
+    fn test_extend() {

Review Comment:
   Please add another test that you extend all valids (the case where materialization is not really needed)
   
   and another test when you extends all nulls



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2702353763


##########
arrow-select/src/coalesce/primitive.rs:
##########
@@ -95,6 +96,151 @@ impl<T: ArrowPrimitiveType + Debug> InProgressArray for InProgressPrimitiveArray
         Ok(())
     }
 
+    /// Copy rows using a predicate
+    fn copy_rows_by_filter(
+        &mut self,
+        filter: &FilterPredicate,
+        offset: usize,
+        len: usize,
+    ) -> Result<(), ArrowError> {
+        self.ensure_capacity();
+
+        let s = self
+            .source
+            .as_ref()
+            .ok_or_else(|| {
+                ArrowError::InvalidArgumentError(
+                    "Internal Error: InProgressPrimitiveArray: source not set".to_string(),
+                )
+            })?
+            .slice(offset, len);
+        let s = s.as_primitive::<T>();
+
+        let values = s.values();
+        let count = filter.count();
+
+        // Use the predicate's strategy for optimal iteration
+        match filter.strategy() {
+            IterationStrategy::SlicesIterator => {
+                // Copy values, nulls using slices
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    for (start, end) in SlicesIterator::new(filter.filter_array()) {
+                        // SAFETY: slices are derived from filter predicate
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });

Review Comment:
   Note that this might not be vectorized, I would check in Compiler Explorer.
   
   as LLVM in some cases is not smart enough for this (that capacity suffice)
   
   (same for the rest)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3764349753

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (58190b8e4d584284467a2c37ea293361ecc5e895) to 1db1a8869cceb179aa885ed58da9f0b49c03eafe [diff](https://github.com/apache/arrow-rs/compare/1db1a8869cceb179aa885ed58da9f0b49c03eafe..58190b8e4d584284467a2c37ea293361ecc5e895)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2701081585


##########
arrow-select/src/filter.rs:
##########
@@ -321,7 +341,13 @@ enum IterationStrategy {
 impl IterationStrategy {
     /// The default [`IterationStrategy`] for a filter of length `filter_length`
     /// and selecting `filter_count` rows
-    fn default_strategy(filter_length: usize, filter_count: usize) -> Self {
+    ///
+    /// Returns:
+    /// - [`IterationStrategy::None`] if `filter_count` is 0
+    /// - [`IterationStrategy::All`] if `filter_count == filter_length`
+    /// - [`IterationStrategy::SlicesIterator`] if selectivity > 80%

Review Comment:
   Yes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-4795287633

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2616171354


##########
arrow-select/src/coalesce.rs:
##########
@@ -526,6 +631,118 @@ impl BatchCoalescer {
     }
 }
 
+/// Find the position after the n-th set bit in a boolean array starting from `start`.
+/// Returns the position after the n-th set bit, or the end of the array if fewer than n bits are set.

Review Comment:
   Good idea



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3754778386

   The null handling code using bititerator might be slightly slower than earlier (but removes some complex code and the earlier didn't take care of sliced arrays). I think we might recover perf later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3764349640

   
   run benchmark coalesce_kernels
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3764952182

   Looks almost on par now... now we can avoid `filter` altogether for views


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771765470

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771766048

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771773759

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771755403

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771761339

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771675575

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771676032

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671384

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771678621

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771796196

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3759934437

   checking this one out


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763372597

   > BTW I am working on tests and a more thorough review of this PR
   
   Thank you <3 I'll probably have a bit of time later today again to check / review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771555384

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (63443dafc8ec5470043fb637ef12c488a6e229ea) to 9c6065c451a1131393191ea828151639f031bf87 [diff](https://github.com/apache/arrow-rs/compare/9c6065c451a1131393191ea828151639f031bf87..63443dafc8ec5470043fb637ef12c488a6e229ea)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771704897

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    240.5±2.25ms        ? ?/sec    1.08    260.0±7.09ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      7.9±0.11ms        ? ?/sec    1.05      8.2±0.19ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      4.0±0.13ms        ? ?/sec    1.00      4.0±0.17ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.2±0.07ms        ? ?/sec    1.07      3.4±0.11ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    226.3±2.97ms        ? ?/sec    1.03    232.7±3.52ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.8±0.12ms        ? ?/sec    1.01      8.9±0.13ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.04      4.5±0.11ms        ? ?/sec    1.00      4.3±0.12ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.6±0.02ms        ? ?/sec    1.24      4.4±0.10ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     54.3±1.45ms        ? ?/sec    1.05     57.3±0.67ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     11.1±0.28ms        ? ?/sec    1.01     11.2±0.16ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.2±0.36ms        ? ?/sec    1.02      9.3±0.40ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.5±0.13ms        ? ?/sec    1.33      9.9±0.20ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.01     67.3±1.18ms        ? ?/sec    1.00     66.9±1.32ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.4±0.13ms        ? ?/sec    1.00     12.5±0.31ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.5±0.35ms        ? ?/sec    1.03      9.8±0.42ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      8.2±0.19ms        ? ?/sec    1.14      9.3±0.15ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     27.9±0.15ms        ? ?/sec    1.64     45.9±0.76ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.5±0.04ms        ? ?/sec    1.26      5.7±0.14ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.16      5.0±0.28ms        ? ?/sec    1.00      4.3±0.16ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00      2.8±0.04ms        ? ?/sec    1.01      2.8±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     46.5±0.79ms        ? ?/sec    1.20     55.8±0.61ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      7.1±0.16ms        ? ?/sec    1.06      7.6±0.06ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.15      6.0±0.20ms        ? ?/sec    1.00      5.3±0.19ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.2±0.03ms        ? ?/sec    1.59      3.6±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     26.8±0.27ms        ? ?/sec    1.54     41.2±0.98ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.5±0.04ms        ? ?/sec    1.25      4.4±0.15ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.23      2.7±0.11ms        ? ?/sec    1.00      2.2±0.19ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.01  1327.3±20.38µs        ? ?/sec    1.00  1318.1±28.29µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     44.7±0.64ms        ? ?/sec    1.13     50.6±1.24ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.2±0.05ms        ? ?/sec    1.05      6.5±0.19ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.21      4.0±0.15ms        ? ?/sec    1.00      3.3±0.18ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.5±0.03ms        ? ?/sec    1.44      3.6±0.10ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     69.3±0.30ms        ? ?/sec    1.33     92.2±0.49ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.9±0.07ms        ? ?/sec    1.27      8.8±0.19ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.5±0.38ms        ? ?/sec    1.01      3.5±0.32ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1738.0±24.44µs        ? ?/sec    1.66      2.9±0.09ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    107.0±1.71ms        ? ?/sec    1.12    120.4±1.80ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     13.1±0.08ms        ? ?/sec    1.08     14.1±0.31ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.07      7.1±0.40ms        ? ?/sec    1.00      6.6±0.41ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.6±0.05ms        ? ?/sec    1.84      8.4±0.11ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     52.0±0.52ms        ? ?/sec    1.22     63.5±1.96ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.01      7.4±0.04ms        ? ?/sec    1.00      7.3±0.07ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.64      6.2±0.26ms        ? ?/sec    1.00      3.8±0.27ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            2.37      3.1±0.05ms        ? ?/sec    1.00  1288.3±14.78µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.00     74.7±2.31ms        ? ?/sec    1.08     80.8±1.59ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     10.2±0.05ms        ? ?/sec    1.09     11.1±0.42ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.32      7.3±0.24ms        ? ?/sec    1.00      5.5±0.31ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.13      4.2±0.05ms        ? ?/sec    1.00      3.7±0.08ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771684736

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771698929

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771780287

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771782302

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771786649

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771793909

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771791803

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3769900074

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    248.9±3.31ms        ? ?/sec    1.00    249.7±2.87ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      7.7±0.08ms        ? ?/sec    1.06      8.2±0.10ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      3.8±0.09ms        ? ?/sec    1.03      4.0±0.13ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.1±0.05ms        ? ?/sec    1.09      3.4±0.08ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.01    236.2±4.21ms        ? ?/sec    1.00    234.4±4.03ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.7±0.13ms        ? ?/sec    1.02      9.0±0.29ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.00      4.3±0.09ms        ? ?/sec    1.02      4.4±0.11ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.5±0.02ms        ? ?/sec    1.26      4.4±0.02ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     52.0±1.90ms        ? ?/sec    1.09     56.5±0.41ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     10.7±0.18ms        ? ?/sec    1.06     11.3±0.15ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.04      9.4±0.30ms        ? ?/sec    1.00      9.0±0.19ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.4±0.15ms        ? ?/sec    1.35     10.0±0.15ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     64.8±0.42ms        ? ?/sec    1.01     65.7±0.42ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.4±0.15ms        ? ?/sec    1.00     12.4±0.23ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.4±0.28ms        ? ?/sec    1.01      9.5±0.27ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      8.0±0.12ms        ? ?/sec    1.18      9.4±0.22ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     27.2±0.40ms        ? ?/sec    1.64     44.6±0.28ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.4±0.03ms        ? ?/sec    1.26      5.6±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.09      5.1±0.22ms        ? ?/sec    1.00      4.7±0.22ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00  1022.7±15.27µs        ? ?/sec    2.80      2.9±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     45.6±0.21ms        ? ?/sec    1.20     54.9±1.24ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      6.9±0.04ms        ? ?/sec    1.10      7.6±0.25ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.08      6.0±0.18ms        ? ?/sec    1.00      5.6±0.23ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.2±0.02ms        ? ?/sec    1.63      3.6±0.06ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     25.9±0.14ms        ? ?/sec    1.56     40.3±1.22ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.4±0.04ms        ? ?/sec    1.28      4.3±0.03ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.23      2.7±0.09ms        ? ?/sec    1.00      2.2±0.20ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.01  1345.5±40.97µs        ? ?/sec    1.00  1331.3±30.66µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     44.5±1.12ms        ? ?/sec    1.12     49.9±0.77ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.0±0.11ms        ? ?/sec    1.08      6.5±0.08ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.18      4.0±0.13ms        ? ?/sec    1.00      3.4±0.17ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.5±0.04ms        ? ?/sec    1.44      3.6±0.03ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     67.6±2.07ms        ? ?/sec    1.33     89.9±0.46ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.7±0.02ms        ? ?/sec    1.28      8.5±0.26ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.1±0.12ms        ? ?/sec    1.06      3.3±0.20ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1681.6±16.09µs        ? ?/sec    1.69      2.8±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    105.0±1.91ms        ? ?/sec    1.13    118.3±2.59ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     12.8±0.26ms        ? ?/sec    1.10     14.1±0.07ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.6±0.43ms        ? ?/sec    1.08      7.1±0.33ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.5±0.15ms        ? ?/sec    1.90      8.6±0.05ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     46.4±0.47ms        ? ?/sec    1.35     62.8±0.76ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.00      5.6±0.02ms        ? ?/sec    1.28      7.2±0.18ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.16      4.3±0.16ms        ? ?/sec    1.00      3.7±0.17ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            2.01      2.6±0.09ms        ? ?/sec    1.00  1312.1±14.76µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.06     85.4±0.67ms        ? ?/sec    1.00     80.9±0.48ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     10.4±0.09ms        ? ?/sec    1.06     11.0±0.07ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.12      5.8±0.24ms        ? ?/sec    1.00      5.1±0.17ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.13      4.2±0.02ms        ? ?/sec    1.00      3.7±0.06ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 10x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3613278691

   Hm it seems iy contains a bug, probably makes the benchmark results off as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3627376800

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (d2b5d296471710319ecc8fa0d0662ea466f5ab80) to ed9efe78e4cc958cc96707557818e754419debb0 [diff](https://github.com/apache/arrow-rs/compare/ed9efe78e4cc958cc96707557818e754419debb0..d2b5d296471710319ecc8fa0d0662ea466f5ab80)
   BENCH_NAME=filter_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench filter_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 10x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3613213068

   🤖 Hi @alamb, you asked to view the benchmark queue (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3613212887).
   
   | Job | User | Benchmarks | Comment |
   | --- | --- | --- | --- |
   | `arrow-8933-3613162300.sh` | alamb | default | `https://github.com/apache/arrow-rs/pull/8933#issuecomment-3613162300` |
   | `arrow-8933-3613131981.sh` | alamb | filter_kernels | `https://github.com/apache/arrow-rs/pull/8933#issuecomment-3613131981` |
   | `arrow-8951-3613212415.sh` | alamb | filter_kernels | `https://github.com/apache/arrow-rs/pull/8951#issuecomment-3613212415` |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3644014310

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (bb025cf4db6c36e1f2144ec87a29f8374bd1d6c5) to ed9efe78e4cc958cc96707557818e754419debb0 [diff](https://github.com/apache/arrow-rs/compare/ed9efe78e4cc958cc96707557818e754419debb0..bb025cf4db6c36e1f2144ec87a29f8374bd1d6c5)
   BENCH_NAME=filter_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench filter_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3816006005

   The plan is now to finalize https://github.com/apache/arrow-rs/pulls to see it's effects on this kernel.
   (I believe non-zero offsets after slicing can have some impact).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2617124197


##########
arrow-buffer/src/builder/null.rs:
##########
@@ -193,6 +193,85 @@ impl NullBufferBuilder {
         }
     }
 
+    /// Extends this builder with validity values.
+    ///
+    ///
+    /// # Example
+    /// ```
+    /// # use arrow_buffer::NullBufferBuilder;
+    /// let mut builder = NullBufferBuilder::new(8);
+    /// let validities = [true, false, true, true];
+    /// builder.extend(validities.iter().copied());
+    /// assert_eq!(builder.len(), 4);
+    /// ```
+    pub fn extend<I: Iterator<Item = bool>>(&mut self, iter: I) {
+        let (lower, upper) = iter.size_hint();
+        let len = upper.expect("Iterator must have exact size_hint");
+        debug_assert_eq!(lower, len, "Iterator must have exact size_hint");
+
+        if len == 0 {
+            return;
+        }
+
+        // Materialize since we're about to append bits
+        self.materialize_if_needed();
+
+        let buf = self.bitmap_builder.as_mut().unwrap();
+        let start_len = buf.len();
+        // Advance to allocate space, initializing new bits to 0
+        buf.advance(len);
+
+        let slice = buf.as_slice_mut();
+        let mut bit_idx = start_len;
+        let end_bit = start_len + len;
+
+        // Process in chunks of 64 bits when byte-aligned for better performance

Review Comment:
   I'm a bit curious, why this don't have some part for unaligned an aligned handling, and 
   ```
   handle_unaligned() // handled start_len % 8 header
   handle_aligned() // handle inner payloads
   handle_unaligned() // handle_trailer
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3654541687

   ```
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.4±0.25ms        ? ?/sec    1.16      7.4±0.36ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.5±0.03ms        ? ?/sec    2.02      9.1±0.05ms        ? ?/sec
   ```
   
   nice


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3765272052

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (be6b7964915bdc7f20cf1b3bec735b719f1c6272) to 1db1a8869cceb179aa885ed58da9f0b49c03eafe [diff](https://github.com/apache/arrow-rs/compare/1db1a8869cceb179aa885ed58da9f0b49c03eafe..be6b7964915bdc7f20cf1b3bec735b719f1c6272)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3765561626

   > I am watching this PR -- please ping me / let me know what I can do to be helpful. Otherwise I'll plan to check in each day
   
   I have around 1 hour a day availavle. I am planning to / the following work could be done:
   
   * implement/benchmark the faster byteview implementation
   * improve filterpredicate slice to not allocate
   * address feedback
   * create PRs out of changes 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3768902876

   
   run benchmark coalesce_kernels
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3765906845

   Here is a PR to add some more test coverage (it targets main)
   - https://github.com/apache/arrow-rs/pull/9218


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3769818118

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (99772cf63542692ad07e3e9dd24565e74dd957f1) to 9c6065c451a1131393191ea828151639f031bf87 [diff](https://github.com/apache/arrow-rs/compare/9c6065c451a1131393191ea828151639f031bf87..99772cf63542692ad07e3e9dd24565e74dd957f1)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3768415195

   run benchmark coalesce_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771764446

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771753056

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771758933

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771760359

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771764969

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771790584

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771786132

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771769439

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771804040

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771673789

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771706566

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771706104

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771700829

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771710540

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771713668

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771707585

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771679510

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771693447

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771683332

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771698265

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 10x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3613212887

   show benchmark queue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3617181011

   
   > run benchmark coalesce_kernels
   
   I added this to the allowed benchmarks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3617254346

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.01    261.9±3.23ms        ? ?/sec    1.00    259.4±2.06ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      8.6±0.14ms        ? ?/sec    1.01      8.7±0.10ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      4.1±0.06ms        ? ?/sec    1.01      4.1±0.09ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.5±0.01ms        ? ?/sec    1.02      3.5±0.02ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    245.6±2.39ms        ? ?/sec    1.27    312.5±3.08ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.01      9.4±0.09ms        ? ?/sec    1.00      9.4±0.07ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.00      4.5±0.08ms        ? ?/sec    1.02      4.6±0.08ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      4.6±0.03ms        ? ?/sec    1.01      4.6±0.02ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.01     59.6±1.58ms        ? ?/sec    1.00     59.2±0.34ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     11.6±0.18ms        ? ?/sec    1.00     11.6±0.18ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.01      9.3±0.18ms        ? ?/sec    1.00      9.2±0.09ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      8.2±0.22ms        ? ?/sec    1.28     10.4±0.24ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.01     70.3±0.25ms        ? ?/sec    1.00     69.9±0.25ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.01     12.9±0.14ms        ? ?/sec    1.00     12.8±0.06ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.8±0.05ms        ? ?/sec    1.06     10.4±0.16ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00     10.0±0.25ms        ? ?/sec    1.02     10.1±0.20ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.05     50.7±0.30ms        ? ?/sec    1.00     48.1±0.17ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.03      6.2±0.06ms        ? ?/sec    1.00      6.0±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.00      4.5±0.11ms        ? ?/sec    1.00      4.5±0.15ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.02      3.1±0.03ms        ? ?/sec    1.00      3.0±0.02ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.04     60.3±0.24ms        ? ?/sec    1.00     58.1±0.25ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.03      8.2±0.03ms        ? ?/sec    1.00      7.9±0.03ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.00      5.6±0.13ms        ? ?/sec    1.07      6.0±0.11ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      3.9±0.02ms        ? ?/sec    1.01      3.9±0.01ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.03     43.5±0.56ms        ? ?/sec    1.00     42.5±0.09ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.05      4.9±0.22ms        ? ?/sec    1.00      4.7±0.01ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.03      2.4±0.05ms        ? ?/sec    1.00      2.3±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1466.4±10.31µs        ? ?/sec    1.05   1537.3±9.34µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.02     53.3±0.16ms        ? ?/sec    1.00     52.1±0.13ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.01      7.2±0.03ms        ? ?/sec    1.00      7.1±0.02ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.00      3.7±0.03ms        ? ?/sec    1.06      3.9±0.07ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.01      3.9±0.02ms        ? ?/sec    1.00      3.9±0.01ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     54.1±1.62ms        ? ?/sec    1.80     97.2±0.21ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      5.9±0.03ms        ? ?/sec    1.57      9.3±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.2±0.09ms        ? ?/sec    1.17      3.7±0.05ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00      2.7±0.01ms        ? ?/sec    1.14      3.1±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00     60.9±0.09ms        ? ?/sec    2.06    125.4±0.26ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     10.9±0.04ms        ? ?/sec    1.38     15.1±0.06ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.16      8.6±0.20ms        ? ?/sec    1.00      7.4±0.36ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.62     14.7±0.04ms        ? ?/sec    1.00      9.1±0.04ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.04     68.2±0.48ms        ? ?/sec    1.00     65.7±1.26ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.09      7.9±0.04ms        ? ?/sec    1.00      7.3±0.02ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.00      3.6±0.17ms        ? ?/sec    1.08      3.9±0.21ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.00   1400.0±6.27µs        ? ?/sec    1.02   1421.7±6.47µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.10     92.0±0.23ms        ? ?/sec    1.00     83.6±0.13ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.06     11.6±0.05ms        ? ?/sec    1.00     11.0±0.05ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.00      5.1±0.08ms        ? ?/sec    1.10      5.7±0.33ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      3.8±0.01ms        ? ?/sec    1.01      3.8±0.01ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 10x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3613551385

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                         coalesce_batches_filter                main
   -----                                                                         -----------------------                ----
   filter context decimal128 (kept 1/2)                                          1.36     57.5±5.45µs        ? ?/sec    1.00     42.1±1.93µs        ? ?/sec
   filter context decimal128 high selectivity (kept 1023/1024)                   1.00     55.7±4.51µs        ? ?/sec    1.09     60.5±0.29µs        ? ?/sec
   filter context decimal128 low selectivity (kept 1/1024)                       1.00    242.4±0.35ns        ? ?/sec    1.06    256.0±1.60ns        ? ?/sec
   filter context f32 (kept 1/2)                                                 1.00     77.7±1.20µs        ? ?/sec    1.00     78.0±2.52µs        ? ?/sec
   filter context f32 high selectivity (kept 1023/1024)                          1.00      9.9±0.32µs        ? ?/sec    1.01     10.1±0.30µs        ? ?/sec
   filter context f32 low selectivity (kept 1/1024)                              1.00    444.2±7.59ns        ? ?/sec    1.06   469.4±13.36ns        ? ?/sec
   filter context fsb with value length 20 (kept 1/2)                            1.00     60.7±1.16µs        ? ?/sec    1.00     60.7±0.37µs        ? ?/sec
   filter context fsb with value length 20 high selectivity (kept 1023/1024)     1.00     60.7±0.36µs        ? ?/sec    1.00     60.7±0.56µs        ? ?/sec
   filter context fsb with value length 20 low selectivity (kept 1/1024)         1.00     60.6±0.26µs        ? ?/sec    1.00     60.8±1.05µs        ? ?/sec
   filter context fsb with value length 5 (kept 1/2)                             1.00     60.8±1.45µs        ? ?/sec    1.00     60.7±1.02µs        ? ?/sec
   filter context fsb with value length 5 high selectivity (kept 1023/1024)      1.00     60.7±0.71µs        ? ?/sec    1.00     60.8±1.22µs        ? ?/sec
   filter context fsb with value length 5 low selectivity (kept 1/1024)          1.01     61.2±3.05µs        ? ?/sec    1.00     60.8±0.90µs        ? ?/sec
   filter context fsb with value length 50 (kept 1/2)                            1.00     60.7±0.46µs        ? ?/sec    1.00     60.8±0.46µs        ? ?/sec
   filter context fsb with value length 50 high selectivity (kept 1023/1024)     1.00     61.0±2.06µs        ? ?/sec    1.00     60.7±0.55µs        ? ?/sec
   filter context fsb with value length 50 low selectivity (kept 1/1024)         1.00     60.8±1.25µs        ? ?/sec    1.00     60.8±1.00µs        ? ?/sec
   filter context i32 (kept 1/2)                                                 1.01     16.6±0.28µs        ? ?/sec    1.00     16.5±0.30µs        ? ?/sec
   filter context i32 high selectivity (kept 1023/1024)                          1.04      6.5±0.20µs        ? ?/sec    1.00      6.2±0.17µs        ? ?/sec
   filter context i32 low selectivity (kept 1/1024)                              1.00    236.0±5.78ns        ? ?/sec    1.05    246.9±1.45ns        ? ?/sec
   filter context i32 w NULLs (kept 1/2)                                         1.00     77.8±2.17µs        ? ?/sec    1.00     77.9±0.80µs        ? ?/sec
   filter context i32 w NULLs high selectivity (kept 1023/1024)                  1.00     10.1±0.52µs        ? ?/sec    1.04     10.5±0.18µs        ? ?/sec
   filter context i32 w NULLs low selectivity (kept 1/1024)                      1.00    446.9±4.94ns        ? ?/sec    1.06    471.6±6.49ns        ? ?/sec
   filter context mixed string view (kept 1/2)                                   1.00    109.0±3.21µs        ? ?/sec    1.11    120.7±3.20µs        ? ?/sec
   filter context mixed string view high selectivity (kept 1023/1024)            1.00     53.9±2.45µs        ? ?/sec    1.03     55.3±2.41µs        ? ?/sec
   filter context mixed string view low selectivity (kept 1/1024)                1.00   654.9±19.57ns        ? ?/sec    1.04   677.9±18.99ns        ? ?/sec
   filter context short string view (kept 1/2)                                   1.00    104.2±1.47µs        ? ?/sec    1.08    112.2±3.44µs        ? ?/sec
   filter context short string view high selectivity (kept 1023/1024)            1.02     55.5±1.25µs        ? ?/sec    1.00     54.5±0.23µs        ? ?/sec
   filter context short string view low selectivity (kept 1/1024)                1.00    464.2±2.70ns        ? ?/sec    1.06    491.4±7.75ns        ? ?/sec
   filter context string (kept 1/2)                                              1.03   599.4±17.30µs        ? ?/sec    1.00    582.1±5.14µs        ? ?/sec
   filter context string dictionary (kept 1/2)                                   1.00     17.0±0.13µs        ? ?/sec    1.02     17.3±0.27µs        ? ?/sec
   filter context string dictionary high selectivity (kept 1023/1024)            1.00      7.0±0.34µs        ? ?/sec    1.02      7.2±0.27µs        ? ?/sec
   filter context string dictionary low selectivity (kept 1/1024)                1.02    847.3±9.58ns        ? ?/sec    1.00    829.8±3.84ns        ? ?/sec
   filter context string dictionary w NULLs (kept 1/2)                           1.00     78.8±1.05µs        ? ?/sec    1.00     78.9±2.34µs        ? ?/sec
   filter context string dictionary w NULLs high selectivity (kept 1023/1024)    1.00     10.7±0.41µs        ? ?/sec    1.01     10.8±0.35µs        ? ?/sec
   filter context string dictionary w NULLs low selectivity (kept 1/1024)        1.01  1076.9±14.42ns        ? ?/sec    1.00  1067.4±30.14ns        ? ?/sec
   filter context string high selectivity (kept 1023/1024)                       1.00   703.0±13.80µs        ? ?/sec    1.00   703.8±19.93µs        ? ?/sec
   filter context string low selectivity (kept 1/1024)                           1.00  1016.7±52.17ns        ? ?/sec    1.02  1036.2±34.58ns        ? ?/sec
   filter context u8 (kept 1/2)                                                  1.00     14.9±0.05µs        ? ?/sec    1.00     15.0±0.14µs        ? ?/sec
   filter context u8 high selectivity (kept 1023/1024)                           1.00  1829.3±23.69ns        ? ?/sec    1.11      2.0±0.01µs        ? ?/sec
   filter context u8 low selectivity (kept 1/1024)                               1.00    231.0±5.30ns        ? ?/sec    1.03    238.8±0.83ns        ? ?/sec
   filter context u8 w NULLs (kept 1/2)                                          1.00     75.9±0.20µs        ? ?/sec    1.00     76.1±0.78µs        ? ?/sec
   filter context u8 w NULLs high selectivity (kept 1023/1024)                   1.00      5.1±0.08µs        ? ?/sec    1.05      5.4±0.06µs        ? ?/sec
   filter context u8 w NULLs low selectivity (kept 1/1024)                       1.00   441.3±12.39ns        ? ?/sec    1.06    467.4±2.32ns        ? ?/sec
   filter decimal128 (kept 1/2)                                                  1.00     49.5±0.83µs        ? ?/sec    1.18     58.6±2.81µs        ? ?/sec
   filter decimal128 high selectivity (kept 1023/1024)                           1.17     61.3±2.70µs        ? ?/sec    1.00     52.6±1.25µs        ? ?/sec
   filter decimal128 low selectivity (kept 1/1024)                               1.00      2.9±0.09µs        ? ?/sec    1.13      3.2±0.08µs        ? ?/sec
   filter f32 (kept 1/2)                                                         1.07    166.6±7.99µs        ? ?/sec    1.00    156.4±2.84µs        ? ?/sec
   filter fsb with value length 20 (kept 1/2)                                    1.12    141.6±1.19µs        ? ?/sec    1.00    126.0±3.73µs        ? ?/sec
   filter fsb with value length 20 high selectivity (kept 1023/1024)             1.11     76.6±1.07µs        ? ?/sec    1.00     68.7±1.04µs        ? ?/sec
   filter fsb with value length 20 low selectivity (kept 1/1024)                 1.00      2.7±0.09µs        ? ?/sec    1.29      3.5±0.10µs        ? ?/sec
   filter fsb with value length 5 (kept 1/2)                                     1.17    141.8±2.30µs        ? ?/sec    1.00    121.1±0.87µs        ? ?/sec
   filter fsb with value length 5 high selectivity (kept 1023/1024)              1.00     10.8±0.16µs        ? ?/sec    1.05     11.3±0.33µs        ? ?/sec
   filter fsb with value length 5 low selectivity (kept 1/1024)                  1.00      2.6±0.08µs        ? ?/sec    1.28      3.3±0.02µs        ? ?/sec
   filter fsb with value length 50 (kept 1/2)                                    1.05    189.3±7.05µs        ? ?/sec    1.00    181.1±9.22µs        ? ?/sec
   filter fsb with value length 50 high selectivity (kept 1023/1024)             1.00    255.5±8.77µs        ? ?/sec    1.03    264.3±6.26µs        ? ?/sec
   filter fsb with value length 50 low selectivity (kept 1/1024)                 1.00      2.6±0.03µs        ? ?/sec    1.27      3.3±0.10µs        ? ?/sec
   filter i32 (kept 1/2)                                                         1.25     53.8±0.68µs        ? ?/sec    1.00     43.2±0.31µs        ? ?/sec
   filter i32 high selectivity (kept 1023/1024)                                  1.05      8.9±0.48µs        ? ?/sec    1.00      8.4±0.32µs        ? ?/sec
   filter i32 low selectivity (kept 1/1024)                                      1.24      2.9±0.06µs        ? ?/sec    1.00      2.4±0.03µs        ? ?/sec
   filter optimize (kept 1/2)                                                    1.01     54.8±2.99µs        ? ?/sec    1.00     54.5±1.51µs        ? ?/sec
   filter optimize high selectivity (kept 1023/1024)                             1.18      3.1±0.14µs        ? ?/sec    1.00      2.6±0.01µs        ? ?/sec
   filter optimize low selectivity (kept 1/1024)                                 1.00      2.7±0.00µs        ? ?/sec    1.00      2.7±0.02µs        ? ?/sec
   filter run array (kept 1/2)                                                   1.03   436.4±17.42µs        ? ?/sec    1.00    422.5±4.27µs        ? ?/sec
   filter run array high selectivity (kept 1023/1024)                            1.01    452.6±7.45µs        ? ?/sec    1.00   449.3±12.94µs        ? ?/sec
   filter run array low selectivity (kept 1/1024)                                1.01   336.4±10.57µs        ? ?/sec    1.00    334.5±2.82µs        ? ?/sec
   filter single record batch                                                    1.23     54.3±2.92µs        ? ?/sec    1.00     44.2±0.07µs        ? ?/sec
   filter u8 (kept 1/2)                                                          1.00     45.5±0.99µs        ? ?/sec    1.00     45.7±0.44µs        ? ?/sec
   filter u8 high selectivity (kept 1023/1024)                                   1.05      4.0±0.11µs        ? ?/sec    1.00      3.8±0.04µs        ? ?/sec
   filter u8 low selectivity (kept 1/1024)                                       1.00      3.0±0.05µs        ? ?/sec    1.12      3.3±0.11µs        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3614262486

   @alamb I think it's ok now - I called AI (Opus 4.5) for some help on the `find_nth_set_bit_position` function.
   
   Mainly needs some polish and seeing if we can improve the `filter: primitive, 8192, nulls: 0.1, selectivity: 0.8` case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3617337060

   > Pretty good I would say... probably have to look a bit more at the null-handling speed
   
   I feel there is a bunch of null handling performance to be had via work in
   - #8561 
   
   I'll try and review this PR more carefully later today


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 10x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3612958124

   @alamb you are probably interested in this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3625209650

   run benchmark coalesce_kernels filter_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3643665704

   Hi @Dandandan  -- I am working through the arrow-rs review backlog. coalesce benchmarks look better. Filter kernels look potentially slower. I'll rerun and try to see if we can see that


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771675110

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771679929

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771681203

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771686743

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3772062429

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (0a5a575772c3ba3a7096997b66af8b910b4ad7b7) to 9c6065c451a1131393191ea828151639f031bf87 [diff](https://github.com/apache/arrow-rs/compare/9c6065c451a1131393191ea828151639f031bf87..0a5a575772c3ba3a7096997b66af8b910b4ad7b7)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771802518

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771796750

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771800940

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771948634

   Hmm that's weird, my local results are so different.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3772158942

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    251.5±2.74ms        ? ?/sec    1.01    253.5±2.74ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      7.9±0.08ms        ? ?/sec    1.05      8.3±0.19ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.01      4.0±0.12ms        ? ?/sec    1.00      4.0±0.11ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.1±0.03ms        ? ?/sec    1.08      3.4±0.03ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    238.0±2.71ms        ? ?/sec    1.00    238.1±3.54ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.9±0.11ms        ? ?/sec    1.01      9.0±0.12ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.02      4.4±0.09ms        ? ?/sec    1.00      4.3±0.12ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.6±0.03ms        ? ?/sec    1.24      4.4±0.03ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     53.0±1.15ms        ? ?/sec    1.08     57.2±0.57ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     10.9±0.14ms        ? ?/sec    1.03     11.3±0.15ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.2±0.41ms        ? ?/sec    1.00      9.2±0.38ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.5±0.13ms        ? ?/sec    1.35     10.1±0.20ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     65.6±0.89ms        ? ?/sec    1.02     67.1±0.95ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.5±0.13ms        ? ?/sec    1.01     12.6±0.17ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.01      9.6±0.28ms        ? ?/sec    1.00      9.5±0.32ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      8.1±0.14ms        ? ?/sec    1.16      9.4±0.15ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     28.8±0.13ms        ? ?/sec    1.58     45.5±0.25ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.7±0.25ms        ? ?/sec    1.21      5.7±0.07ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.20      5.3±0.24ms        ? ?/sec    1.00      4.5±0.23ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.01      2.9±0.03ms        ? ?/sec    1.00      2.9±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     47.7±0.56ms        ? ?/sec    1.17     56.0±0.40ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      7.4±0.04ms        ? ?/sec    1.02      7.5±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.14      6.1±0.17ms        ? ?/sec    1.00      5.4±0.24ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.1±0.05ms        ? ?/sec    1.70      3.5±0.02ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     27.6±0.12ms        ? ?/sec    1.49     41.1±0.18ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.7±0.04ms        ? ?/sec    1.19      4.4±0.02ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.31      2.8±0.22ms        ? ?/sec    1.00      2.2±0.16ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1003.8±66.37µs        ? ?/sec    1.31  1318.7±26.64µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     45.8±0.42ms        ? ?/sec    1.11     50.8±0.30ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.5±0.22ms        ? ?/sec    1.00      6.5±0.13ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.20      4.0±0.23ms        ? ?/sec    1.00      3.4±0.22ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.2±0.02ms        ? ?/sec    1.61      3.6±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     69.7±1.11ms        ? ?/sec    1.32     92.2±1.66ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.9±0.15ms        ? ?/sec    1.27      8.8±0.12ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.04      3.5±0.40ms        ? ?/sec    1.00      3.4±0.22ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00   1610.6±7.85µs        ? ?/sec    1.73      2.8±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    107.1±0.85ms        ? ?/sec    1.13    120.7±1.35ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     13.1±0.07ms        ? ?/sec    1.08     14.1±0.55ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.7±0.37ms        ? ?/sec    1.01      6.8±0.35ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.4±0.11ms        ? ?/sec    1.89      8.4±0.17ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     53.5±1.44ms        ? ?/sec    1.18     63.4±0.28ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.05      7.7±0.16ms        ? ?/sec    1.00      7.3±0.19ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.67      6.1±0.24ms        ? ?/sec    1.00      3.7±0.17ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.63      2.1±0.07ms        ? ?/sec    1.00   1277.3±8.95µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.00     75.7±0.33ms        ? ?/sec    1.07     80.9±0.95ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     10.5±0.05ms        ? ?/sec    1.06     11.1±0.23ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.42      7.1±0.27ms        ? ?/sec    1.00      5.0±0.12ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      3.2±0.03ms        ? ?/sec    1.16      3.7±0.03ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771701647

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771702058

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771709100

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771715457

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771696526

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771699554

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771703427

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771774769

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771779742

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771785599

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771771577

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771778516

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771768942

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771772682

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3774398699

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (d3cf7cb75dfb3dc48cfbea69c63178c78f04c056) to ebace17225a1ca3946fad19c54ad53ec76dc5b68 [diff](https://github.com/apache/arrow-rs/compare/ebace17225a1ca3946fad19c54ad53ec76dc5b68..d3cf7cb75dfb3dc48cfbea69c63178c78f04c056)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3619747417

   run benchmark coalesce_kernels filter_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3619751276

   run benchmark coalesce_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3627439628

   > Andrew Lamb Weekly-ish Open Source plan - 2025-12-08 apache/datafusion#19210
   
   Nice, so this confirms the higher threshold also is a speed up for filter kernels.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2616173444


##########
arrow-buffer/src/builder/null.rs:
##########
@@ -193,6 +193,85 @@ impl NullBufferBuilder {
         }
     }
 
+    /// Extends this builder with validity values.
+    ///
+    ///
+    /// # Example
+    /// ```
+    /// # use arrow_buffer::NullBufferBuilder;
+    /// let mut builder = NullBufferBuilder::new(8);
+    /// let validities = [true, false, true, true];
+    /// builder.extend(validities.iter().copied());
+    /// assert_eq!(builder.len(), 4);
+    /// ```
+    pub fn extend<I: Iterator<Item = bool>>(&mut self, iter: I) {
+        let (lower, upper) = iter.size_hint();
+        let len = upper.expect("Iterator must have exact size_hint");
+        debug_assert_eq!(lower, len, "Iterator must have exact size_hint");
+
+        if len == 0 {
+            return;
+        }
+
+        // Materialize since we're about to append bits
+        self.materialize_if_needed();
+
+        let buf = self.bitmap_builder.as_mut().unwrap();
+        let start_len = buf.len();
+        // Advance to allocate space, initializing new bits to 0
+        buf.advance(len);
+
+        let slice = buf.as_slice_mut();
+        let mut bit_idx = start_len;
+        let end_bit = start_len + len;
+
+        // Process in chunks of 64 bits when byte-aligned for better performance
+        if start_len % 8 == 0 {
+            let start_byte = start_len / 8;
+            let mut iter = iter.peekable();
+
+            // Process full u64 chunks (64 bits at a time)
+            while bit_idx + 64 <= end_bit && iter.peek().is_some() {

Review Comment:
   :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3634205957

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.01    277.1±1.94ms        ? ?/sec    1.00    273.9±2.22ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.04      9.6±0.29ms        ? ?/sec    1.00      9.2±0.32ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.02      4.4±0.13ms        ? ?/sec    1.00      4.3±0.10ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.5±0.05ms        ? ?/sec    1.06      3.7±0.05ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    261.6±1.88ms        ? ?/sec    1.28    333.6±2.60ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00     10.0±0.36ms        ? ?/sec    1.00     10.1±0.43ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.05      4.9±0.05ms        ? ?/sec    1.00      4.6±0.09ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.8±0.04ms        ? ?/sec    1.25      4.8±0.06ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.04     66.0±1.43ms        ? ?/sec    1.00     63.2±1.47ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     11.9±0.17ms        ? ?/sec    1.01     12.1±0.21ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.01     10.6±0.47ms        ? ?/sec    1.00     10.5±0.26ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      9.9±0.21ms        ? ?/sec    1.34     13.2±0.38ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.01     73.2±1.24ms        ? ?/sec    1.00     72.3±0.90ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.03     13.5±0.19ms        ? ?/sec    1.00     13.1±0.16ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.06     11.2±0.40ms        ? ?/sec    1.00     10.6±0.32ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00     10.2±0.27ms        ? ?/sec    1.14     11.6±0.46ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.01     49.7±0.34ms        ? ?/sec    1.00     49.0±1.09ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.01      6.2±0.06ms        ? ?/sec    1.00      6.1±0.20ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.04      5.0±0.20ms        ? ?/sec    1.00      4.8±0.13ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00      3.1±0.11ms        ? ?/sec    1.15      3.6±0.13ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.03     60.2±1.30ms        ? ?/sec    1.00     58.6±0.81ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      8.2±0.11ms        ? ?/sec    1.00      8.2±0.09ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.10      6.2±0.16ms        ? ?/sec    1.00      5.6±0.13ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.3±0.04ms        ? ?/sec    1.74      4.0±0.06ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.02     43.6±0.62ms        ? ?/sec    1.00     42.9±0.57ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.01      4.8±0.05ms        ? ?/sec    1.00      4.8±0.10ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.09      2.6±0.20ms        ? ?/sec    1.00      2.4±0.09ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1181.2±18.79µs        ? ?/sec    1.32  1563.3±17.05µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.02     53.2±0.69ms        ? ?/sec    1.00     52.1±0.63ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.01      7.2±0.31ms        ? ?/sec    1.00      7.1±0.05ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.08      3.9±0.10ms        ? ?/sec    1.00      3.7±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.4±0.02ms        ? ?/sec    1.71      4.0±0.04ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     54.1±0.43ms        ? ?/sec    1.80     97.2±0.29ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      5.9±0.06ms        ? ?/sec    1.59      9.3±0.19ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.9±0.42ms        ? ?/sec    1.00      3.9±0.12ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1730.1±35.92µs        ? ?/sec    1.81      3.1±0.03ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00     59.1±0.67ms        ? ?/sec    2.13    126.1±1.67ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00      8.2±0.06ms        ? ?/sec    1.84     15.0±0.23ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.5±0.30ms        ? ?/sec    1.08      7.0±0.11ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.9±0.07ms        ? ?/sec    1.85      9.1±0.08ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.03     68.0±0.33ms        ? ?/sec    1.00     66.1±0.38ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.04      7.7±0.05ms        ? ?/sec    1.00      7.4±0.24ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.03      4.1±0.32ms        ? ?/sec    1.00      3.9±0.10ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.00  1366.2±15.03µs        ? ?/sec    1.06  1442.0±19.64µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.06     89.9±0.59ms        ? ?/sec    1.00     84.7±1.17ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.04     11.6±0.09ms        ? ?/sec    1.00     11.2±0.07ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.00      5.2±0.31ms        ? ?/sec    1.03      5.4±0.22ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      2.7±0.07ms        ? ?/sec    1.45      4.0±0.04ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3643908996

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                         coalesce_batches_filter                main
   -----                                                                         -----------------------                ----
   filter context decimal128 (kept 1/2)                                          1.00     45.9±6.33µs        ? ?/sec    1.01     46.4±6.05µs        ? ?/sec
   filter context decimal128 high selectivity (kept 1023/1024)                   1.03     51.1±2.22µs        ? ?/sec    1.00     49.4±1.18µs        ? ?/sec
   filter context decimal128 low selectivity (kept 1/1024)                       1.00    241.1±2.07ns        ? ?/sec    1.01    244.2±1.03ns        ? ?/sec
   filter context f32 (kept 1/2)                                                 1.00     78.0±1.94µs        ? ?/sec    1.00     77.9±0.71µs        ? ?/sec
   filter context f32 high selectivity (kept 1023/1024)                          1.00     10.4±0.49µs        ? ?/sec    1.00     10.4±0.31µs        ? ?/sec
   filter context f32 low selectivity (kept 1/1024)                              1.00    451.1±1.68ns        ? ?/sec    1.02    458.4±7.82ns        ? ?/sec
   filter context fsb with value length 20 (kept 1/2)                            1.00     60.6±0.28µs        ? ?/sec    1.00     60.6±0.24µs        ? ?/sec
   filter context fsb with value length 20 high selectivity (kept 1023/1024)     1.00     60.8±0.96µs        ? ?/sec    1.00     60.7±0.90µs        ? ?/sec
   filter context fsb with value length 20 low selectivity (kept 1/1024)         1.00     60.5±0.24µs        ? ?/sec    1.00     60.8±0.81µs        ? ?/sec
   filter context fsb with value length 5 (kept 1/2)                             1.00     61.1±1.47µs        ? ?/sec    1.00     60.8±0.80µs        ? ?/sec
   filter context fsb with value length 5 high selectivity (kept 1023/1024)      1.00     60.8±0.80µs        ? ?/sec    1.00     60.7±0.76µs        ? ?/sec
   filter context fsb with value length 5 low selectivity (kept 1/1024)          1.00     60.7±0.66µs        ? ?/sec    1.00     61.0±1.31µs        ? ?/sec
   filter context fsb with value length 50 (kept 1/2)                            1.00     60.6±0.93µs        ? ?/sec    1.00     60.8±0.33µs        ? ?/sec
   filter context fsb with value length 50 high selectivity (kept 1023/1024)     1.00     60.9±0.62µs        ? ?/sec    1.00     60.7±0.25µs        ? ?/sec
   filter context fsb with value length 50 low selectivity (kept 1/1024)         1.00     60.6±0.46µs        ? ?/sec    1.00     60.7±0.60µs        ? ?/sec
   filter context i32 (kept 1/2)                                                 1.01     16.6±0.09µs        ? ?/sec    1.00     16.5±0.17µs        ? ?/sec
   filter context i32 high selectivity (kept 1023/1024)                          1.00      6.4±0.40µs        ? ?/sec    1.02      6.6±0.40µs        ? ?/sec
   filter context i32 low selectivity (kept 1/1024)                              1.01    238.3±1.05ns        ? ?/sec    1.00    235.9±3.08ns        ? ?/sec
   filter context i32 w NULLs (kept 1/2)                                         1.00     77.7±0.54µs        ? ?/sec    1.00     77.7±1.59µs        ? ?/sec
   filter context i32 w NULLs high selectivity (kept 1023/1024)                  1.00     10.2±0.45µs        ? ?/sec    1.05     10.8±0.41µs        ? ?/sec
   filter context i32 w NULLs low selectivity (kept 1/1024)                      1.00    454.0±5.90ns        ? ?/sec    1.02    462.1±7.68ns        ? ?/sec
   filter context mixed string view (kept 1/2)                                   1.00    104.5±4.74µs        ? ?/sec    1.02    106.5±4.35µs        ? ?/sec
   filter context mixed string view high selectivity (kept 1023/1024)            1.00     53.9±1.70µs        ? ?/sec    1.04     56.2±1.61µs        ? ?/sec
   filter context mixed string view low selectivity (kept 1/1024)                1.00    675.7±7.20ns        ? ?/sec    1.01    684.0±2.67ns        ? ?/sec
   filter context short string view (kept 1/2)                                   1.00    104.8±5.30µs        ? ?/sec    1.02    107.0±3.28µs        ? ?/sec
   filter context short string view high selectivity (kept 1023/1024)            1.01     55.5±1.57µs        ? ?/sec    1.00     54.9±1.38µs        ? ?/sec
   filter context short string view low selectivity (kept 1/1024)                1.01    482.5±4.17ns        ? ?/sec    1.00    477.0±1.44ns        ? ?/sec
   filter context string (kept 1/2)                                              1.00   585.3±12.49µs        ? ?/sec    1.03   603.2±17.73µs        ? ?/sec
   filter context string dictionary (kept 1/2)                                   1.01     17.4±0.13µs        ? ?/sec    1.00     17.2±0.16µs        ? ?/sec
   filter context string dictionary high selectivity (kept 1023/1024)            1.00      7.2±0.50µs        ? ?/sec    1.09      7.9±0.19µs        ? ?/sec
   filter context string dictionary low selectivity (kept 1/1024)                1.00    832.1±8.58ns        ? ?/sec    1.01   842.1±14.13ns        ? ?/sec
   filter context string dictionary w NULLs (kept 1/2)                           1.01     79.1±1.46µs        ? ?/sec    1.00     78.6±0.86µs        ? ?/sec
   filter context string dictionary w NULLs high selectivity (kept 1023/1024)    1.00     11.1±0.46µs        ? ?/sec    1.01     11.2±0.26µs        ? ?/sec
   filter context string dictionary w NULLs low selectivity (kept 1/1024)        1.00  1064.7±16.00ns        ? ?/sec    1.02   1086.4±7.18ns        ? ?/sec
   filter context string high selectivity (kept 1023/1024)                       1.00   670.3±23.59µs        ? ?/sec    1.37  917.2±161.37µs        ? ?/sec
   filter context string low selectivity (kept 1/1024)                           1.02   1037.6±5.15ns        ? ?/sec    1.00  1015.4±17.73ns        ? ?/sec
   filter context u8 (kept 1/2)                                                  1.00     15.0±0.23µs        ? ?/sec    1.00     15.0±0.08µs        ? ?/sec
   filter context u8 high selectivity (kept 1023/1024)                           1.00  1820.1±12.84ns        ? ?/sec    1.00  1826.1±31.22ns        ? ?/sec
   filter context u8 low selectivity (kept 1/1024)                               1.00    230.7±1.42ns        ? ?/sec    1.00    231.1±2.77ns        ? ?/sec
   filter context u8 w NULLs (kept 1/2)                                          1.00     76.0±0.66µs        ? ?/sec    1.00     76.2±0.61µs        ? ?/sec
   filter context u8 w NULLs high selectivity (kept 1023/1024)                   1.00      5.4±0.12µs        ? ?/sec    1.00      5.4±0.03µs        ? ?/sec
   filter context u8 w NULLs low selectivity (kept 1/1024)                       1.00    446.4±8.90ns        ? ?/sec    1.02    457.0±1.87ns        ? ?/sec
   filter decimal128 (kept 1/2)                                                  1.00     51.9±4.04µs        ? ?/sec    1.10     57.0±3.08µs        ? ?/sec
   filter decimal128 high selectivity (kept 1023/1024)                           1.00     52.6±1.80µs        ? ?/sec    1.00     52.5±1.20µs        ? ?/sec
   filter decimal128 low selectivity (kept 1/1024)                               1.00      3.2±0.10µs        ? ?/sec    1.02      3.2±0.04µs        ? ?/sec
   filter f32 (kept 1/2)                                                         1.01    157.2±1.78µs        ? ?/sec    1.00    156.4±1.54µs        ? ?/sec
   filter fsb with value length 20 (kept 1/2)                                    1.06    132.2±1.80µs        ? ?/sec    1.00    124.8±0.92µs        ? ?/sec
   filter fsb with value length 20 high selectivity (kept 1023/1024)             1.00     71.8±2.78µs        ? ?/sec    1.00     71.6±2.42µs        ? ?/sec
   filter fsb with value length 20 low selectivity (kept 1/1024)                 1.00      3.2±0.02µs        ? ?/sec    1.07      3.4±0.03µs        ? ?/sec
   filter fsb with value length 5 (kept 1/2)                                     1.13    137.3±0.79µs        ? ?/sec    1.00    121.2±0.60µs        ? ?/sec
   filter fsb with value length 5 high selectivity (kept 1023/1024)              1.00     11.3±0.65µs        ? ?/sec    1.01     11.4±0.63µs        ? ?/sec
   filter fsb with value length 5 low selectivity (kept 1/1024)                  1.00      3.1±0.01µs        ? ?/sec    1.09      3.4±0.06µs        ? ?/sec
   filter fsb with value length 50 (kept 1/2)                                    1.00   164.1±13.09µs        ? ?/sec    1.04   170.8±17.51µs        ? ?/sec
   filter fsb with value length 50 high selectivity (kept 1023/1024)             1.00    204.1±6.86µs        ? ?/sec    1.01    206.4±8.06µs        ? ?/sec
   filter fsb with value length 50 low selectivity (kept 1/1024)                 1.00      3.2±0.03µs        ? ?/sec    1.06      3.4±0.02µs        ? ?/sec
   filter i32 (kept 1/2)                                                         1.07     46.4±0.12µs        ? ?/sec    1.00     43.3±0.16µs        ? ?/sec
   filter i32 high selectivity (kept 1023/1024)                                  1.00      8.5±0.36µs        ? ?/sec    1.05      9.0±0.39µs        ? ?/sec
   filter i32 low selectivity (kept 1/1024)                                      1.39      3.3±0.06µs        ? ?/sec    1.00      2.4±0.01µs        ? ?/sec
   filter optimize (kept 1/2)                                                    1.01     54.5±0.54µs        ? ?/sec    1.00     54.1±0.37µs        ? ?/sec
   filter optimize high selectivity (kept 1023/1024)                             1.00      2.6±0.01µs        ? ?/sec    1.01      2.6±0.01µs        ? ?/sec
   filter optimize low selectivity (kept 1/1024)                                 1.11      3.0±0.02µs        ? ?/sec    1.00      2.8±0.02µs        ? ?/sec
   filter run array (kept 1/2)                                                   1.00    426.0±5.10µs        ? ?/sec    1.00   424.0±11.28µs        ? ?/sec
   filter run array high selectivity (kept 1023/1024)                            1.00    450.1±4.75µs        ? ?/sec    1.00    450.9±5.01µs        ? ?/sec
   filter run array low selectivity (kept 1/1024)                                1.01    337.5±5.50µs        ? ?/sec    1.00    335.4±3.55µs        ? ?/sec
   filter single record batch                                                    1.03     45.6±0.24µs        ? ?/sec    1.00     44.3±0.42µs        ? ?/sec
   filter u8 (kept 1/2)                                                          1.00     45.1±0.11µs        ? ?/sec    1.01     45.6±0.20µs        ? ?/sec
   filter u8 high selectivity (kept 1023/1024)                                   1.02      3.8±0.03µs        ? ?/sec    1.00      3.7±0.03µs        ? ?/sec
   filter u8 low selectivity (kept 1/1024)                                       1.00      3.3±0.05µs        ? ?/sec    1.00      3.3±0.02µs        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3754467270

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                                coalesce_batches_filter                main
   -----                                                                                -----------------------                ----
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    252.8±3.00ms        ? ?/sec    1.04    261.7±5.39ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      8.4±0.27ms        ? ?/sec    1.00      8.4±0.19ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.06      4.2±0.12ms        ? ?/sec    1.00      3.9±0.13ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.2±0.06ms        ? ?/sec    1.09      3.4±0.03ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    237.7±2.36ms        ? ?/sec    1.31    312.0±5.94ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.02      9.3±0.08ms        ? ?/sec    1.00      9.1±0.24ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.00      4.5±0.13ms        ? ?/sec    1.01      4.5±0.08ms        ? ?/sec
   filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.5±0.07ms        ? ?/sec    1.28      4.5±0.10ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.01     58.6±1.71ms        ? ?/sec    1.00     57.8±0.68ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     11.1±0.24ms        ? ?/sec    1.00     11.1±0.26ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.4±0.42ms        ? ?/sec    1.03      9.8±0.86ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.04      8.4±0.32ms        ? ?/sec    1.00      8.1±0.22ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.03     69.0±0.97ms        ? ?/sec    1.00     67.2±1.60ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.5±0.41ms        ? ?/sec    1.03     12.9±0.76ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.7±0.13ms        ? ?/sec    1.01      9.8±0.31ms        ? ?/sec
   filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      8.0±0.20ms        ? ?/sec    1.17      9.4±0.18ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.03     47.8±0.65ms        ? ?/sec    1.00     46.5±0.79ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      5.6±0.03ms        ? ?/sec    1.00      5.6±0.15ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.01      4.1±0.18ms        ? ?/sec    1.00      4.1±0.10ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00      2.5±0.11ms        ? ?/sec    1.14      2.8±0.04ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.05     58.1±0.32ms        ? ?/sec    1.00     55.6±0.37ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      7.6±0.08ms        ? ?/sec    1.01      7.7±0.23ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.00      5.3±0.14ms        ? ?/sec    1.01      5.4±0.30ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00  1978.6±56.95µs        ? ?/sec    1.85      3.7±0.02ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.03     42.9±0.53ms        ? ?/sec    1.00     41.6±0.59ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.01      4.4±0.07ms        ? ?/sec    1.00      4.4±0.06ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.03      2.2±0.15ms        ? ?/sec    1.00      2.2±0.07ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00   983.7±16.21µs        ? ?/sec    1.39  1365.0±23.97µs        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.04     52.6±0.25ms        ? ?/sec    1.00     50.4±0.19ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.6±0.08ms        ? ?/sec    1.00      6.6±0.07ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.01      3.4±0.18ms        ? ?/sec    1.00      3.4±0.11ms        ? ?/sec
   filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00  1943.9±63.55µs        ? ?/sec    1.88      3.7±0.04ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     53.4±1.46ms        ? ?/sec    1.81     97.0±0.43ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      5.4±0.03ms        ? ?/sec    1.66      8.9±0.03ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      2.9±0.04ms        ? ?/sec    1.20      3.5±0.16ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1659.6±14.46µs        ? ?/sec    1.70      2.8±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00     60.5±1.04ms        ? ?/sec    2.07    125.0±2.53ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00      9.4±0.05ms        ? ?/sec    1.56     14.6±0.14ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.5±0.39ms        ? ?/sec    1.00      6.5±0.14ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.5±0.06ms        ? ?/sec    1.93      8.6±0.04ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.04     65.7±0.28ms        ? ?/sec    1.00     63.1±0.36ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.03      7.4±0.09ms        ? ?/sec    1.00      7.2±0.07ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.00      3.7±0.32ms        ? ?/sec    1.04      3.8±0.24ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.00  1249.9±42.99µs        ? ?/sec    1.08   1355.0±9.21µs        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.02     88.8±1.25ms        ? ?/sec    1.00     87.3±2.41ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.04     11.1±0.25ms        ? ?/sec    1.00     10.7±0.08ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.11      5.5±0.23ms        ? ?/sec    1.00      4.9±0.09ms        ? ?/sec
   filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      2.6±0.02ms        ? ?/sec    1.48      3.8±0.02ms        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3754760980

   ```
   filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     53.4±1.46ms        ? ?/sec    1.81     97.0±0.43ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      5.4±0.03ms        ? ?/sec    1.66      8.9±0.03ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      2.9±0.04ms        ? ?/sec    1.20      3.5±0.16ms        ? ?/sec
   filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1659.6±14.46µs        ? ?/sec    1.70      2.8±0.02ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00     60.5±1.04ms        ? ?/sec    2.07    125.0±2.53ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00      9.4±0.05ms        ? ?/sec    1.56     14.6±0.14ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.5±0.39ms        ? ?/sec    1.00      6.5±0.14ms        ? ?/sec
   filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.5±0.06ms        ? ?/sec    1.93      8.6±0.04ms        ? ?/sec
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3765658265

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (6de89a3fe5185469a135f8f45c6479868b7eee4b) to 9c6065c451a1131393191ea828151639f031bf87 [diff](https://github.com/apache/arrow-rs/compare/9c6065c451a1131393191ea828151639f031bf87..6de89a3fe5185469a135f8f45c6479868b7eee4b)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2702404229


##########
arrow-select/src/filter.rs:
##########
@@ -37,9 +37,9 @@ use arrow_schema::*;
 /// [`SlicesIterator`] to copy ranges of values. Otherwise iterate
 /// over individual rows using [`IndexIterator`]
 ///
-/// Threshold of 0.8 chosen based on <https://dl.acm.org/doi/abs/10.1145/3465998.3466009>
+/// Threshold of 0.9 chosen based on benchmarking results
 ///
-const FILTER_SLICES_SELECTIVITY_THRESHOLD: f64 = 0.8;
+const FILTER_SLICES_SELECTIVITY_THRESHOLD: f64 = 0.9;

Review Comment:
   Did you run with the same machines as the paper run with? (Intel Xeon Platinum 8124M CPU @3.00GHz with AVX512.)
   
   because you might improve on machine a but harm machine b.
   
   and if the machine you are running on is not bare metal, you will also have larger noise



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2701038833


##########
arrow-select/src/coalesce.rs:
##########
@@ -238,10 +245,112 @@ impl BatchCoalescer {
         batch: RecordBatch,
         filter: &BooleanArray,
     ) -> Result<(), ArrowError> {
-        // TODO: optimize this to avoid materializing (copying the results
-        // of filter to a new batch)
-        let filtered_batch = filter_record_batch(&batch, filter)?;
-        self.push_batch(filtered_batch)
+        // We only support primitve now, fallback to filter_record_batch for other types
+        // Also, skip optimization when filter is not very selective
+        if batch
+            .schema()
+            .fields()
+            .iter()
+            .any(|field| !field.data_type().is_primitive())
+            || self

Review Comment:
   I think benchmarks show it is better to just use the kernel always (so I think this condition needs to go).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763452027

   > I expect the filter kernels should not be changed, except for the 80-90% threshold for using the sliced kernel.
   
   Yeah, that is why I was running the benchmark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763734164

   The reason the existing test coverage for filter batch doesn't cover the new code is that it uses a multi-column record batch
   
   
   Another thing I noticed around while trying to test is that the filter predicate passed to `copy_rows_by_filter` is for a sliced filter array, but the underlying InProgressArrays seem to treat it as though it started at the start of the array
   
   I think that may be a bug, but I am looking more carefully now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 10x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3614203673

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3614203469).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: arrow_reader, concatenate_kernels, filter_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`
   Unsupported benchmarks: coalesce_kernels.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3629135486

   on my list


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3627509541

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                         coalesce_batches_filter                main
   -----                                                                         -----------------------                ----
   filter context decimal128 (kept 1/2)                                          1.12     48.0±6.74µs        ? ?/sec    1.00     42.7±2.30µs        ? ?/sec
   filter context decimal128 high selectivity (kept 1023/1024)                   1.04     51.8±1.43µs        ? ?/sec    1.00     49.8±1.98µs        ? ?/sec
   filter context decimal128 low selectivity (kept 1/1024)                       1.00    246.6±2.17ns        ? ?/sec    1.02    250.7±8.70ns        ? ?/sec
   filter context f32 (kept 1/2)                                                 1.00     77.6±0.34µs        ? ?/sec    1.00     77.7±1.04µs        ? ?/sec
   filter context f32 high selectivity (kept 1023/1024)                          1.00      9.6±0.30µs        ? ?/sec    1.04     10.0±0.35µs        ? ?/sec
   filter context f32 low selectivity (kept 1/1024)                              1.00    453.9±8.60ns        ? ?/sec    1.03    465.9±1.78ns        ? ?/sec
   filter context fsb with value length 20 (kept 1/2)                            1.00     60.7±0.44µs        ? ?/sec    1.00     60.6±0.23µs        ? ?/sec
   filter context fsb with value length 20 high selectivity (kept 1023/1024)     1.00     60.8±1.16µs        ? ?/sec    1.00     60.7±0.47µs        ? ?/sec
   filter context fsb with value length 20 low selectivity (kept 1/1024)         1.00     60.6±0.25µs        ? ?/sec    1.00     60.6±0.22µs        ? ?/sec
   filter context fsb with value length 5 (kept 1/2)                             1.00     60.9±2.04µs        ? ?/sec    1.00     61.0±1.50µs        ? ?/sec
   filter context fsb with value length 5 high selectivity (kept 1023/1024)      1.00     60.8±1.38µs        ? ?/sec    1.00     60.6±0.39µs        ? ?/sec
   filter context fsb with value length 5 low selectivity (kept 1/1024)          1.00     60.5±0.17µs        ? ?/sec    1.00     60.7±0.37µs        ? ?/sec
   filter context fsb with value length 50 (kept 1/2)                            1.00     60.8±1.05µs        ? ?/sec    1.00     60.6±0.41µs        ? ?/sec
   filter context fsb with value length 50 high selectivity (kept 1023/1024)     1.00     60.6±0.30µs        ? ?/sec    1.00     60.7±0.78µs        ? ?/sec
   filter context fsb with value length 50 low selectivity (kept 1/1024)         1.00     60.7±0.54µs        ? ?/sec    1.00     60.7±0.50µs        ? ?/sec
   filter context i32 (kept 1/2)                                                 1.00     16.6±0.15µs        ? ?/sec    1.00     16.5±0.11µs        ? ?/sec
   filter context i32 high selectivity (kept 1023/1024)                          1.03      6.4±0.26µs        ? ?/sec    1.00      6.2±0.27µs        ? ?/sec
   filter context i32 low selectivity (kept 1/1024)                              1.00    236.9±0.69ns        ? ?/sec    1.02    240.5±1.53ns        ? ?/sec
   filter context i32 w NULLs (kept 1/2)                                         1.00     77.8±1.01µs        ? ?/sec    1.00     77.5±0.35µs        ? ?/sec
   filter context i32 w NULLs high selectivity (kept 1023/1024)                  1.01     10.4±0.28µs        ? ?/sec    1.00     10.3±0.50µs        ? ?/sec
   filter context i32 w NULLs low selectivity (kept 1/1024)                      1.00    445.7±1.90ns        ? ?/sec    1.06    470.6±0.97ns        ? ?/sec
   filter context mixed string view (kept 1/2)                                   1.02    103.3±2.61µs        ? ?/sec    1.00    101.6±1.87µs        ? ?/sec
   filter context mixed string view high selectivity (kept 1023/1024)            1.01     55.4±1.79µs        ? ?/sec    1.00     55.0±1.21µs        ? ?/sec
   filter context mixed string view low selectivity (kept 1/1024)                1.01    673.2±3.53ns        ? ?/sec    1.00    668.5±7.34ns        ? ?/sec
   filter context short string view (kept 1/2)                                   1.00    102.9±3.52µs        ? ?/sec    1.02    104.8±4.15µs        ? ?/sec
   filter context short string view high selectivity (kept 1023/1024)            1.00     53.4±1.23µs        ? ?/sec    1.01     54.0±1.40µs        ? ?/sec
   filter context short string view low selectivity (kept 1/1024)                1.00   470.4±10.21ns        ? ?/sec    1.02    479.2±3.80ns        ? ?/sec
   filter context string (kept 1/2)                                              1.00   571.6±11.64µs        ? ?/sec    1.01    575.3±6.77µs        ? ?/sec
   filter context string dictionary (kept 1/2)                                   1.00     17.3±0.11µs        ? ?/sec    1.01     17.5±0.15µs        ? ?/sec
   filter context string dictionary high selectivity (kept 1023/1024)            1.09      7.6±0.16µs        ? ?/sec    1.00      7.0±0.27µs        ? ?/sec
   filter context string dictionary low selectivity (kept 1/1024)                1.00    819.0±3.24ns        ? ?/sec    1.01    823.3±4.54ns        ? ?/sec
   filter context string dictionary w NULLs (kept 1/2)                           1.00     78.5±1.16µs        ? ?/sec    1.00     78.5±0.39µs        ? ?/sec
   filter context string dictionary w NULLs high selectivity (kept 1023/1024)    1.00     10.6±0.28µs        ? ?/sec    1.03     11.0±0.45µs        ? ?/sec
   filter context string dictionary w NULLs low selectivity (kept 1/1024)        1.00   1045.6±4.56ns        ? ?/sec    1.02  1061.4±14.24ns        ? ?/sec
   filter context string high selectivity (kept 1023/1024)                       1.02   696.1±23.21µs        ? ?/sec    1.00   681.8±17.88µs        ? ?/sec
   filter context string low selectivity (kept 1/1024)                           1.00    897.1±7.30ns        ? ?/sec    1.15  1033.3±27.38ns        ? ?/sec
   filter context u8 (kept 1/2)                                                  1.00     15.0±0.16µs        ? ?/sec    1.00     15.0±0.13µs        ? ?/sec
   filter context u8 high selectivity (kept 1023/1024)                           1.00  1828.5±36.36ns        ? ?/sec    1.00  1822.6±15.03ns        ? ?/sec
   filter context u8 low selectivity (kept 1/1024)                               1.00    236.0±2.00ns        ? ?/sec    1.00    235.0±0.85ns        ? ?/sec
   filter context u8 w NULLs (kept 1/2)                                          1.00     75.9±0.45µs        ? ?/sec    1.00     76.1±0.99µs        ? ?/sec
   filter context u8 w NULLs high selectivity (kept 1023/1024)                   1.00      5.1±0.04µs        ? ?/sec    1.05      5.4±0.09µs        ? ?/sec
   filter context u8 w NULLs low selectivity (kept 1/1024)                       1.00    443.0±2.34ns        ? ?/sec    1.05    466.5±5.04ns        ? ?/sec
   filter decimal128 (kept 1/2)                                                  1.03     57.0±3.41µs        ? ?/sec    1.00     55.1±0.45µs        ? ?/sec
   filter decimal128 high selectivity (kept 1023/1024)                           1.04     55.4±1.84µs        ? ?/sec    1.00     53.5±1.06µs        ? ?/sec
   filter decimal128 low selectivity (kept 1/1024)                               1.00      2.8±0.02µs        ? ?/sec    1.14      3.2±0.02µs        ? ?/sec
   filter f32 (kept 1/2)                                                         1.06    165.2±2.92µs        ? ?/sec    1.00    155.9±1.54µs        ? ?/sec
   filter fsb with value length 20 (kept 1/2)                                    1.13    142.0±0.91µs        ? ?/sec    1.00    125.8±0.54µs        ? ?/sec
   filter fsb with value length 20 high selectivity (kept 1023/1024)             1.00     72.0±3.10µs        ? ?/sec    1.02     73.7±0.94µs        ? ?/sec
   filter fsb with value length 20 low selectivity (kept 1/1024)                 1.00      2.7±0.02µs        ? ?/sec    1.26      3.4±0.04µs        ? ?/sec
   filter fsb with value length 5 (kept 1/2)                                     1.17    141.5±0.77µs        ? ?/sec    1.00    121.0±0.55µs        ? ?/sec
   filter fsb with value length 5 high selectivity (kept 1023/1024)              1.05     11.3±0.48µs        ? ?/sec    1.00     10.8±0.63µs        ? ?/sec
   filter fsb with value length 5 low selectivity (kept 1/1024)                  1.00      2.6±0.01µs        ? ?/sec    1.28      3.3±0.01µs        ? ?/sec
   filter fsb with value length 50 (kept 1/2)                                    1.00    187.6±9.25µs        ? ?/sec    1.02   190.4±10.86µs        ? ?/sec
   filter fsb with value length 50 high selectivity (kept 1023/1024)             1.05    231.0±4.47µs        ? ?/sec    1.00    219.9±7.16µs        ? ?/sec
   filter fsb with value length 50 low selectivity (kept 1/1024)                 1.00      2.6±0.03µs        ? ?/sec    1.30      3.4±0.01µs        ? ?/sec
   filter i32 (kept 1/2)                                                         1.24     53.6±0.72µs        ? ?/sec    1.00     43.2±0.27µs        ? ?/sec
   filter i32 high selectivity (kept 1023/1024)                                  1.06      9.1±0.53µs        ? ?/sec    1.00      8.6±0.41µs        ? ?/sec
   filter i32 low selectivity (kept 1/1024)                                      1.23      2.9±0.03µs        ? ?/sec    1.00      2.4±0.12µs        ? ?/sec
   filter optimize (kept 1/2)                                                    1.00     54.3±1.34µs        ? ?/sec    1.00     54.2±0.52µs        ? ?/sec
   filter optimize high selectivity (kept 1023/1024)                             1.14      3.0±0.02µs        ? ?/sec    1.00      2.6±0.02µs        ? ?/sec
   filter optimize low selectivity (kept 1/1024)                                 1.00      2.7±0.01µs        ? ?/sec    1.00      2.7±0.01µs        ? ?/sec
   filter run array (kept 1/2)                                                   1.02    431.2±2.74µs        ? ?/sec    1.00    421.5±1.60µs        ? ?/sec
   filter run array high selectivity (kept 1023/1024)                            1.00    449.1±1.88µs        ? ?/sec    1.00    449.9±6.45µs        ? ?/sec
   filter run array low selectivity (kept 1/1024)                                1.02    339.7±7.42µs        ? ?/sec    1.00    334.6±2.84µs        ? ?/sec
   filter single record batch                                                    1.21     53.7±0.17µs        ? ?/sec    1.00     44.3±0.13µs        ? ?/sec
   filter u8 (kept 1/2)                                                          1.00     45.3±0.10µs        ? ?/sec    1.01     45.7±0.47µs        ? ?/sec
   filter u8 high selectivity (kept 1023/1024)                                   1.07      4.0±0.02µs        ? ?/sec    1.00      3.8±0.03µs        ? ?/sec
   filter u8 low selectivity (kept 1/1024)                                       1.00      3.0±0.07µs        ? ?/sec    1.11      3.3±0.07µs        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2618331428


##########
arrow-buffer/src/builder/null.rs:
##########
@@ -193,6 +193,85 @@ impl NullBufferBuilder {
         }
     }
 
+    /// Extends this builder with validity values.
+    ///
+    ///
+    /// # Example
+    /// ```
+    /// # use arrow_buffer::NullBufferBuilder;
+    /// let mut builder = NullBufferBuilder::new(8);
+    /// let validities = [true, false, true, true];
+    /// builder.extend(validities.iter().copied());
+    /// assert_eq!(builder.len(), 4);
+    /// ```
+    pub fn extend<I: Iterator<Item = bool>>(&mut self, iter: I) {
+        let (lower, upper) = iter.size_hint();
+        let len = upper.expect("Iterator must have exact size_hint");
+        debug_assert_eq!(lower, len, "Iterator must have exact size_hint");
+
+        if len == 0 {
+            return;
+        }
+
+        // Materialize since we're about to append bits
+        self.materialize_if_needed();
+
+        let buf = self.bitmap_builder.as_mut().unwrap();
+        let start_len = buf.len();
+        // Advance to allocate space, initializing new bits to 0
+        buf.advance(len);
+
+        let slice = buf.as_slice_mut();
+        let mut bit_idx = start_len;
+        let end_bit = start_len + len;
+
+        // Process in chunks of 64 bits when byte-aligned for better performance

Review Comment:
   I think this is the same comment as @alamb has?
   
   Yeah perhaps this can improve performance (will see guided by benchmarks).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3754395618

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (a1d40979049ca963cd9324b25295d3c11b2b84b6) to c70804a728287f38fc51b04c8dbb1e7b72bceb9b [diff](https://github.com/apache/arrow-rs/compare/c70804a728287f38fc51b04c8dbb1e7b72bceb9b..a1d40979049ca963cd9324b25295d3c11b2b84b6)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3765392982

   Merged up to get clippy fix


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3765562581

   I am working on some additional testing 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3763857144

   I found the bug -- fix incoming (then I really have to go)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2701042297


##########
arrow-select/src/coalesce.rs:
##########
@@ -238,10 +245,112 @@ impl BatchCoalescer {
         batch: RecordBatch,
         filter: &BooleanArray,
     ) -> Result<(), ArrowError> {
-        // TODO: optimize this to avoid materializing (copying the results
-        // of filter to a new batch)
-        let filtered_batch = filter_record_batch(&batch, filter)?;
-        self.push_batch(filtered_batch)
+        // We only support primitve now, fallback to filter_record_batch for other types
+        // Also, skip optimization when filter is not very selective
+        if batch
+            .schema()
+            .fields()
+            .iter()
+            .any(|field| !field.data_type().is_primitive())

Review Comment:
   Because of the all-or-nothing check it has no effect on tpch/clickbench performance yet. I am not sure if we can somehow use the conventional filter for non-supported arrays 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2701157664


##########
arrow-select/src/coalesce/primitive.rs:
##########
@@ -95,6 +96,145 @@ impl<T: ArrowPrimitiveType + Debug> InProgressArray for InProgressPrimitiveArray
         Ok(())
     }
 
+    /// Copy rows using a predicate
+    fn copy_rows_by_filter(&mut self, filter: &FilterPredicate) -> Result<(), ArrowError> {
+        self.ensure_capacity();
+
+        let s = self
+            .source
+            .as_ref()
+            .ok_or_else(|| {
+                ArrowError::InvalidArgumentError(
+                    "Internal Error: InProgressPrimitiveArray: source not set".to_string(),
+                )
+            })?
+            .as_primitive::<T>();
+
+        let values = s.values();
+        let count = filter.count();
+
+        // Use the predicate's strategy for optimal iteration
+        match filter.strategy() {
+            IterationStrategy::SlicesIterator => {
+                // Copy values, nulls using slices
+                if let Some(nulls) = s.nulls().filter(|n| n.null_count() > 0) {
+                    for (start, end) in SlicesIterator::new(filter.filter_array()) {
+                        // SAFETY: slices are derived from filter predicate
+                        self.current
+                            .extend_from_slice(unsafe { values.get_unchecked(start..end) });
+                        let slice = nulls.slice(start, end - start);

Review Comment:
   Yes that would probably be faster



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2701178041


##########
arrow-select/src/coalesce/byte_view.rs:
##########
@@ -347,6 +348,33 @@ impl<B: ByteViewType> InProgressArray for InProgressByteViewArray<B> {
         Ok(())
     }
 
+    fn copy_rows_by_filter(
+        &mut self,
+        filter: &FilterPredicate,
+        offset: usize,
+        len: usize,
+    ) -> Result<(), ArrowError> {
+        // Temporarily swap out the source with a filtered slice
+        let Some(source) = self.source.as_mut() else {
+            return Err(ArrowError::InvalidArgumentError(
+                "Internal Error: InProgressByteViewArray: source not set".to_string(),
+            ));
+        };
+        // TODO special case offset 0 to skip this nonsense
+        let source_slice = source.array.slice(offset, len);
+        let filtered = filter.filter(&source_slice)?;
+        let old_source = std::mem::replace(&mut source.array, filtered);

Review Comment:
   I think we have to call `set_source` to calculate `need_gc` etc. for the filtered array



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771712789

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771707073

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771717334

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771714576

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#discussion_r2704873894


##########
arrow-select/src/filter.rs:
##########
@@ -37,9 +37,9 @@ use arrow_schema::*;
 /// [`SlicesIterator`] to copy ranges of values. Otherwise iterate
 /// over individual rows using [`IndexIterator`]
 ///
-/// Threshold of 0.8 chosen based on <https://dl.acm.org/doi/abs/10.1145/3465998.3466009>
+/// Threshold of 0.9 chosen based on benchmarking results
 ///
-const FILTER_SLICES_SELECTIVITY_THRESHOLD: f64 = 0.8;
+const FILTER_SLICES_SELECTIVITY_THRESHOLD: f64 = 0.9;

Review Comment:
   We don't run with the same machine though I think @alamb is a Intel CPU and my local CPU.
   
   The details of our kernels could be very different from the paper, so I think we mainly could go with benchmarking (and improve the implementation of our code to better generate the instructions).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771797467

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771776905

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671956

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771676418

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771676845

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771682108

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771685746

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771705180

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771671037).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771754011

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771754935

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771751172

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771752105

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757

   
   run benchmarks coalesce_kernels
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771769944

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771774293

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771762878

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771754476

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771762195

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771766534

   🤖 Hi @Dandandan, thanks for the request (https://github.com/apache/arrow-rs/pull/8951#issuecomment-3771750757).
   
   [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
   - Standard: (none)
   - Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels
   
   Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3654384848

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing coalesce_batches_filter (f718f2ec4fc5c069b8ed2d5a1892487e6340dc20) to ed9efe78e4cc958cc96707557818e754419debb0 [diff](https://github.com/apache/arrow-rs/compare/ed9efe78e4cc958cc96707557818e754419debb0..f718f2ec4fc5c069b8ed2d5a1892487e6340dc20)
   BENCH_NAME=coalesce_kernels
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=coalesce_batches_filter
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make `push_batch_with_filter` up to 3x faster for primitive types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8951:
URL: https://github.com/apache/arrow-rs/pull/8951#issuecomment-3643666446

   run benchmark filter_kernels


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org