[GitHub] [arrow-rs] ritchie46 commented on pull request #348: Reduce memory usage of concat (large)utf8

[GitBox <gi...@apache.org>]

ritchie46 commented on pull request #348:
URL: https://github.com/apache/arrow-rs/pull/348#issuecomment-848502242


   > @ritchie46 do we have some (micro) benchmark results, like:
   
   ```
   Gnuplot not found, using plotters backend
   concat str 10k          time:   [162.54 us 162.73 us 163.00 us]                           
                           change: [-3.7750% -3.6483% -3.5084%] (p = 0.00 < 0.05)
                           Performance has improved.
   Found 2 outliers among 100 measurements (2.00%)
     1 (1.00%) high mild
     1 (1.00%) high severe
   
   concat str 50% nulls 10k                                                                            
                           time:   [619.71 us 621.16 us 623.28 us]
                           change: [-1.1095% -0.9350% -0.7141%] (p = 0.00 < 0.05)
                           Change within noise threshold.
   Found 10 outliers among 100 measurements (10.00%)
     1 (1.00%) low mild
     2 (2.00%) high mild
     7 (7.00%) high severe
   
   concat str 5% nulls 10k time:   [391.61 us 391.71 us 391.81 us]                                    
                           change: [-1.3337% -1.2446% -1.1878%] (p = 0.00 < 0.05)
                           Performance has improved.
   Found 9 outliers among 100 measurements (9.00%)
     6 (6.00%) high mild
     3 (3.00%) high severe
   ```
   
   Performance wise it doesn't matter/ or hurt that much. So it mostly is more memory efficient. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org