[GitHub] [arrow-rs] viirya commented on pull request #3198: Use SlicesIterator for ArrayData Equality

[GitBox <gi...@apache.org>]

viirya commented on PR #3198:
URL: https://github.com/apache/arrow-rs/pull/3198#issuecomment-1328138638

   > Perhaps this could be simplified by using try_for_each_valid_idx?
   
   Tried with `try_for_each_valid_idx`, but the benchmarks look not good:
   
   ```
   equal_nulls_512         time:   [1.8361 µs 1.8471 µs 1.8602 µs]
                           change: [+171.23% +172.28% +173.48%] (p = 0.00 < 0.05)
                           Performance has regressed.
   Found 23 outliers among 100 measurements (23.00%)
     1 (1.00%) low severe
     1 (1.00%) low mild
     3 (3.00%) high mild
     18 (18.00%) high severe
   
   equal_51200             time:   [303.39 µs 303.51 µs 303.67 µs]
                           change: [+191.52% +191.71% +191.91%] (p = 0.00 < 0.05)
                           Performance has regressed.
   Found 4 outliers among 100 measurements (4.00%)
   ```
   
   `try_for_each_valid_idx` comes to compare each element so it doesn't help on the long array with few nulls (i.e., `equal_51200`), it also brings quite regression on many null case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org