You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/03/28 14:38:30 UTC

[GitHub] [arrow] Dandandan opened a new pull request #9826: ARROW-12123: [Rust][DataFusion] Use smallvec for indices for better join performance

Dandandan opened a new pull request #9826:
URL: https://github.com/apache/arrow/pull/9826


   This is better for the common case where (almost) every value in the build side will be unique, and thus will be store once per this list, avoiding heap allocations and extra indirections (not saving the index inline but in a different  `Vec`).
   
   this is a > 10% improvement for TCP-H query 5 (SF=1, 16 partitions/threads, loaded in memory).
   
   master:
   `Query 5 avg time: 169.63 ms`
   
   PR:
   `Query 5 avg time: 151.07 ms` 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #9826: ARROW-12123: [Rust][DataFusion] Use smallvec for indices for better join performance

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9826:
URL: https://github.com/apache/arrow/pull/9826#issuecomment-808905830


   https://issues.apache.org/jira/browse/ARROW-12123


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] alamb closed pull request #9826: ARROW-12123: [Rust][DataFusion] Use smallvec for indices for better join performance

Posted by GitBox <gi...@apache.org>.
alamb closed pull request #9826:
URL: https://github.com/apache/arrow/pull/9826


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] Dandandan edited a comment on pull request #9826: ARROW-12123: [Rust][DataFusion] Use smallvec for indices for better join performance

Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on pull request #9826:
URL: https://github.com/apache/arrow/pull/9826#issuecomment-810362925


   Merged with master
   
   A small change with some nice performance benefits for common joins, without (for now at least) creating a custom hashmap datastructure.
   `smallvec` is already used by transitive dependencies, and is small, so doesn't seem a big dependency addition. Also it is suggested to be used in Arrow before (for the array data structure).
   
   FYI @alamb @jorgecarleitao @andygrove


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] Dandandan commented on pull request #9826: ARROW-12123: [Rust][DataFusion] Use smallvec for indices for better join performance

Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #9826:
URL: https://github.com/apache/arrow/pull/9826#issuecomment-810362925


   Rebased against master
   
   A small change with some nice performance benefits for common joins, without (for now at least) creating a custom hashmap datastructure.
   `smallvec` is already used by transitive dependencies, and is small, so doesn't seem a big dependency addition. Also it is suggested to be used in Arrow before (for the array data structure).
   
   FYI @alamb @jorgecarleitao @andygrove


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] alamb commented on pull request #9826: ARROW-12123: [Rust][DataFusion] Use smallvec for indices for better join performance

Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9826:
URL: https://github.com/apache/arrow/pull/9826#issuecomment-811024944


   The integration test failure https://github.com/apache/arrow/pull/9826/checks?check_run_id=2229013566 seems to be related to out of disk space and not the changes in this PR
   
   ```
   
   npm WARN tar ENOSPC: no space left on device, write
   npm WARN tar ENOSPC: no space left on device, write
   npm ERR! code ENOSPC
   npm ERR! syscall write
   npm ERR! errno -28
   npm ERR! nospc ENOSPC: no space left on device, write
   npm ERR! nospc There appears to be insufficient space on your system to finish.
   npm ERR! nospc Clear up some disk space and try again.
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org