[GitHub] [arrow] zfoobar commented on issue #34845: Performance of explicit array comparison faster vs. Arrow compute

["zfoobar (via GitHub)" <gi...@apache.org>]

zfoobar commented on issue #34845:
URL: https://github.com/apache/arrow/issues/34845#issuecomment-1500916907

   he indeed made a good point. Here is my test code, eager to hear your thoughts:
   https://gist.github.com/zfoobar/f61c25271f20d54ea72b065b8e3c6a4c
   
   Here are some sample runs with order of magnitude increases in input size. You can see where the performance superiority of Arrow kicks in (marked in green below)
   
   (pyarrow-dev) Zacharys-MacBook-Air:arrow_scratch zacharyfierstadt$ ./my_example 10000
   Generating 10000 random values..
   Array explicitly compared
   Time: 0.216 ms
   Arrays compared using a compute function
   Time: 9.757 ms
   Table created
   Writing CSV file
   
   (pyarrow-dev) Zacharys-MacBook-Air:arrow_scratch zacharyfierstadt$ ./my_example 100000
   Generating 100000 random values..
   Array explicitly compared
   Time: 1.148 ms
   Arrays compared using a compute function
   Time: 13.329 ms
   Table created
   Writing CSV file
   
   (pyarrow-dev) Zacharys-MacBook-Air:arrow_scratch zacharyfierstadt$ ./my_example 1000000
   Generating 1000000 random values..
   Array explicitly compared
   Time: 12.184 ms
   Arrays compared using a compute function
   **Time: 9.332 ms**
   Table created
   Writing CSV file
   
   (pyarrow-dev) Zacharys-MacBook-Air:arrow_scratch zacharyfierstadt$ ./my_example 10000000
   Generating 10000000 random values..
   Array explicitly compared
   Time: 87.849 ms
   Arrays compared using a compute function
   **Time: 39.409 ms**
   Table created
   Writing CSV file
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org