You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2024/03/13 20:10:05 UTC

[I] Support build right with HashJoin in DataFusion [arrow-datafusion]

viirya opened a new issue, #9603:
URL: https://github.com/apache/arrow-datafusion/issues/9603

   ### Is your feature request related to a problem or challenge?
   
   See the discussion at Comet: https://github.com/apache/arrow-datafusion-comet/pull/194#discussion_r1521912155
   
   > Yea, in DataFusion, only left side could be the build side. But in Spark, the HashJoin operator has a build side parameter to indicate which side is build side. The operator will do right thing accordingly internally. So currently we cannot just create a DataFusion HashJoin operator with right side as build side.
   
   > It can be swapped between left and right side, only if we also swap outputs and also column binding in joining keys and joining filter. I'd like to relax the build side constraint in DataFusion instead of doing the swap in Comet.
   
   ### Describe the solution you'd like
   
   HashJoin supports build right option.
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support build right with HashJoin in DataFusion [arrow-datafusion]

Posted by "metesynnada (via GitHub)" <gi...@apache.org>.
metesynnada commented on issue #9603:
URL: https://github.com/apache/arrow-datafusion/issues/9603#issuecomment-1999490390

   Got it, I was figuring out if this could be a blocker.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support build right with HashJoin in DataFusion [datafusion]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #9603: Support build right with HashJoin in DataFusion
URL: https://github.com/apache/datafusion/issues/9603


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support build right with HashJoin in DataFusion [datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #9603:
URL: https://github.com/apache/datafusion/issues/9603#issuecomment-2111007724

   I will work on this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support build right with HashJoin in DataFusion [arrow-datafusion]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #9603:
URL: https://github.com/apache/arrow-datafusion/issues/9603#issuecomment-2000029281

   > No, I think technically we can do the swapping on joining keys, join filter, join output, etc. in Comet. I think it would be good if we can make DataFusion HashJoin more flexible to remove this constraint.
   
   That sounds cool, and such initiative will help DF to support its own join order hints instead of hardcoded join order


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support build right with HashJoin in DataFusion [arrow-datafusion]

Posted by "metesynnada (via GitHub)" <gi...@apache.org>.
metesynnada commented on issue #9603:
URL: https://github.com/apache/arrow-datafusion/issues/9603#issuecomment-1996775831

   Is there a technical problem with swapping join sides in Comet? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support build right with HashJoin in DataFusion [datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #9603:
URL: https://github.com/apache/datafusion/issues/9603#issuecomment-2111007415

   I think this is not related to SQL parser. The SQL query looks the same. But in Spark, it can decide hash side based on relation statistics or the hint given by users.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support build right with HashJoin in DataFusion [arrow-datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #9603:
URL: https://github.com/apache/arrow-datafusion/issues/9603#issuecomment-1997961026

   No, I think technically we can do the swapping on joining keys, join filter, join output, etc. in Comet. I think it would be good if we can make DataFusion HashJoin more flexible to remove this constraint.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support build right with HashJoin in DataFusion [arrow-datafusion]

Posted by "edmondop (via GitHub)" <gi...@apache.org>.
edmondop commented on issue #9603:
URL: https://github.com/apache/arrow-datafusion/issues/9603#issuecomment-2059956637

   @comphead how would this work? do you think we should extend the SQL parser to support hints? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org