You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/16 15:50:14 UTC

[GitHub] [arrow-datafusion] andygrove opened a new issue, #4925: Implement optimizer rule to remove redundant repartitioning

andygrove opened a new issue, #4925:
URL: https://github.com/apache/arrow-datafusion/issues/4925

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   When running benchmark q2 I see this in the plan:
   
   ```
   RepartitionExec: partitioning=Hash([Column { name: "r_regionkey", index: 0 }], 48), metrics=[fetch_time=69.072358ms, send_time=48ns, repart_time=48ns]
     RepartitionExec: partitioning=RoundRobinBatch(48), metrics=[fetch_time=273.515µs, send_time=1ns, repart_time=1ns]
   ```
   
   We introduce the `RoundRobinBatch` partitioning to increase parallelism but then we insert the hash partitioning later for the join. It is redundant to repartition twice here, and it would be better to just preparation on the join key right away.
   
   **Describe the solution you'd like**
   Implement new rule to push hash partitioning down and replace the round robin partitioning.
   
   **Describe alternatives you've considered**
   None
   
   **Additional context**
   None
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove closed issue #4925: Implement optimizer rule to remove redundant repartitioning

Posted by GitBox <gi...@apache.org>.
andygrove closed issue #4925: Implement optimizer rule to remove redundant repartitioning
URL: https://github.com/apache/arrow-datafusion/issues/4925


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove commented on issue #4925: Implement optimizer rule to remove redundant repartitioning

Posted by GitBox <gi...@apache.org>.
andygrove commented on issue #4925:
URL: https://github.com/apache/arrow-datafusion/issues/4925#issuecomment-1384670490

   Ballista already removes these round-robin repartitions so closing this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org