You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/07/11 14:52:29 UTC

[GitHub] [arrow-datafusion] andygrove opened a new issue #707: Ballista: Avoid pointless shuffle write of Parquet partitions

andygrove opened a new issue #707:
URL: https://github.com/apache/arrow-datafusion/issues/707


   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   When running TPC-H query 12 I see query plans like this being executed:
   
   ```
   ShuffleWriterExec: None
     ParquetExec: ...
   ```
   
   This means we are reading a Parquet partition and immediately streaming it to disk in IPC format so that it can be read again later. This is introducing a redundant write and read of this data.
   
   **Describe the solution you'd like**
   TBD
   
   **Describe alternatives you've considered**
   TBD
   
   **Additional context**
   None
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove commented on issue #707: Ballista: Avoid pointless shuffle write of Parquet partitions

Posted by GitBox <gi...@apache.org>.
andygrove commented on issue #707:
URL: https://github.com/apache/arrow-datafusion/issues/707#issuecomment-877826983


   It turns out there is a fundamental bug here. I am working on fixing this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove closed issue #707: Ballista: Finish implementing shuffle mechanism

Posted by GitBox <gi...@apache.org>.
andygrove closed issue #707:
URL: https://github.com/apache/arrow-datafusion/issues/707


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org