You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2024/01/25 08:10:55 UTC

[PR] Relax join keys constraint from Column to any physical expression [arrow-datafusion]

viirya opened a new pull request, #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   Currently the join keys of join operators like `SortMergeJoin` are restricted to be `Column`. But it is commonly we use expressions (e.g., `l_col + 1 = r_col + 2`) other than simply columns as join keys. From the query plan, DataFusion seems to add additional `Project` under join operator which projects the expressions into columns. So the above join operators take join keys as columns.
   
   However, in other query engines, e.g., Spark, its query plan doesn't have the additional projection but its join operators directly take general expressions as join keys.
   
   That means if we cannot delegate such join operators to DataFusion physical join operators which require join keys must be columns.
   
   This patch tries to relax this join keys constraint of physical join operators. So we can construct DataFusion physical join operator using general expressions as join keys.
   
   This patch doesn't change how DataFusion plans the join operators. I.e., DataFusion still plans a join operation using non-column join keys into projection + join operator with columns.
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1912741825

   Thank you @alamb
   
   Note that "investigate removing projection before join input" is not a necessary step for join support in Comet, because Comet doesn't depend on DataFusion's query planning. But definitely we have other steps to work on DataFusion Join.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1915390225

   > While doing so, I wonder if comet could use a custom PhysicalOptimzer pass to add a ProjectionExec 🤔
   
   As I mentioned in the description, adding an additional Projection under Join (e.g., SortMergeJoin) doesn't make a lot sense for Spark due to its distribution nature. An additional Projection means we materialize extra columns (the join key expressions) earlier and which could lead to additional data in shuffle/sort.
   
   So what this does is to relax join key constraints in DataFusion, for example now physical SortMergeJoin can take expressions other than Column as join keys. In Spark/Comet side, now we can translate Spark SortMergeJoin directly to DataFusion SortMergeJoin. We don't need to change Spark physical query structure.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1914596727

   I merged this PR up from main to pick up the clippy fix


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "mustafasrepo (via GitHub)" <gi...@apache.org>.
mustafasrepo commented on code in PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#discussion_r1469162694


##########
datafusion/physical-plan/src/joins/utils.rs:
##########
@@ -405,12 +406,47 @@ pub fn check_join_is_valid(left: &Schema, right: &Schema, on: JoinOnRef) -> Resu
 fn check_join_set_is_valid(
     left: &HashSet<Column>,
     right: &HashSet<Column>,
-    on: &[(Column, Column)],
+    on: &[(PhysicalExprRef, PhysicalExprRef)],
 ) -> Result<()> {
-    let on_left = &on.iter().map(|on| on.0.clone()).collect::<HashSet<_>>();
+    let on_left = &on
+        .iter()
+        .flat_map(|on| {
+            let left = on.0.clone();
+
+            let mut columns = vec![];
+            left.apply(&mut |expr| {
+                Ok({
+                    if let Some(column) = expr.as_any().downcast_ref::<Column>() {
+                        columns.push(column.clone());
+                    }
+                    VisitRecursion::Continue
+                })
+            })
+            .unwrap();
+            columns
+        })
+        .collect::<HashSet<_>>();
     let left_missing = on_left.difference(left).collect::<HashSet<_>>();
 
-    let on_right = &on.iter().map(|on| on.1.clone()).collect::<HashSet<_>>();
+    let on_right = &on
+        .iter()
+        .flat_map(|on| {
+            let right = on.1.clone();
+
+            let mut columns = vec![];
+            right
+                .apply(&mut |expr| {
+                    Ok({
+                        if let Some(column) = expr.as_any().downcast_ref::<Column>() {
+                            columns.push(column.clone());
+                        }
+                        VisitRecursion::Continue
+                    })
+                })
+                .unwrap();
+            columns

Review Comment:
   Similar to comment above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1915602228

   > wouldn't it actually make more sense to compute the expressions prior to the networked shuffle so only 2 columns of data (`lcol_1 + lcol_2` and `rcol_1 + rcol_2`) need to be sent, rather than the 4 original columns 🤔
   
   Hmm, except for joining keys, I think you still can list other columns (e.g., the original 4 columns) into selection list? So they are not always able to be removed from shuffle, I think?
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1915379206

   I wrote up https://github.com/apache/arrow-datafusion/issues/9056#issue-2106223131
   
   While doing so, I wonder if comet could use a custom PhysicalOptimzer pass to add a ProjectionExec 🤔 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1915670650

   > Hmm, except for joining keys, I think you still can list other columns (e.g., the original 4 columns) into selection list? So they are not always able to be removed from shuffle, I think?
   
   Sure, if the columns are used elsewhere in the plan, they can't be removed. I was thinking of the case when they aren't used anywhere else
   
   However, I am not sure how often that happens in the real world
   
   Maybe a range join 🤔 
   
   ```SQL
   SELECT ...
   FROM stocks JOIN splits ON (
     stocks.symbol = splits.symbol AND 
     stocks.ts < splits.max_time AND stocks.ts > splits.min_time
   )
   ```
   
   But I suppose the expressions are't equijoin predicates anyways 🤔 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#discussion_r1468161433


##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -1529,15 +1521,8 @@ mod tests {
     ) -> Result<(Vec<String>, Vec<RecordBatch>)> {
         let partition_count = 4;
 
-        let (left_expr, right_expr) = on
-            .iter()
-            .map(|(l, r)| {
-                (
-                    Arc::new(l.clone()) as Arc<dyn PhysicalExpr>,
-                    Arc::new(r.clone()) as Arc<dyn PhysicalExpr>,
-                )
-            })
-            .unzip();
+        let (left_expr, right_expr) =
+            on.iter().map(|(l, r)| (l.clone(), r.clone())).unzip();

Review Comment:
   In many ways this change makes the code cleaner as it doesn't have to special case Column so much



##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -278,7 +277,7 @@ pub struct HashJoinExec {
     /// right (probe) side which are filtered by the hash table
     pub right: Arc<dyn ExecutionPlan>,
     /// Set of equijoin columns from the relations: `(left_col, right_col)`
-    pub on: Vec<(Column, Column)>,
+    pub on: Vec<(PhysicalExprRef, PhysicalExprRef)>,

Review Comment:
   👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "metesynnada (via GitHub)" <gi...@apache.org>.
metesynnada commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1914078936

   I will review this as soon as possible, particularly the effects on SHJ.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "mustafasrepo (via GitHub)" <gi...@apache.org>.
mustafasrepo commented on code in PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#discussion_r1469162445


##########
datafusion/physical-plan/src/joins/utils.rs:
##########
@@ -405,12 +406,47 @@ pub fn check_join_is_valid(left: &Schema, right: &Schema, on: JoinOnRef) -> Resu
 fn check_join_set_is_valid(
     left: &HashSet<Column>,
     right: &HashSet<Column>,
-    on: &[(Column, Column)],
+    on: &[(PhysicalExprRef, PhysicalExprRef)],
 ) -> Result<()> {
-    let on_left = &on.iter().map(|on| on.0.clone()).collect::<HashSet<_>>();
+    let on_left = &on
+        .iter()
+        .flat_map(|on| {
+            let left = on.0.clone();
+
+            let mut columns = vec![];
+            left.apply(&mut |expr| {
+                Ok({
+                    if let Some(column) = expr.as_any().downcast_ref::<Column>() {
+                        columns.push(column.clone());
+                    }
+                    VisitRecursion::Continue
+                })
+            })
+            .unwrap();

Review Comment:
   For this section, I think you can use `collect_columns` util. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1915509262

   > As I mentioned in the description, adding an additional Projection under Join (e.g., SortMergeJoin) doesn't make a lot sense for Spark due to its distribution nature.
   
   🤔 
   
   > (note that by adding additional projection before join in Spark it means more data to be shuffled/sorted which can be bad for performance)
   
   I don't understand this
   
   If the join is on `lcol_1 + lcol_2` = `rcol_1 + rcol_2` the plan will have to materialize four columns `lcol_1`, `lcol_2`, `rcol_1` and `rcol_2`, and would have to feed all 4 of those columns to the join
   
   wouldn't it actually make more sense to compute the expressions prior to the networked shuffle so only 2 columns of data (`lcol_1 + lcol_2` and `rcol_1 + rcol_2`) need to be sent, rather than the 4 original columns 🤔 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1913244756

   > Note that "investigate removing projection before join input" is not a necessary step for join support in Comet, because Comet doesn't depend on DataFusion's query planning. But definitely we have other steps to work on DataFusion Join.
   
   That makes sense. I think having DataFusion planned joins and Spark planned joins going through different codepaths not only potentially duplicates code it makes it more lkely there are discrepancies that may lead to bugs (e.g. it may be hard to test the spark path as part of the datafusion tests)
   
   For that reason I suggest we make a ticket listing the overall project where we can collect other follow on steps as we come on tehem  (not that you or Comet people will actually do the work, other contributors are likely to be able to help too I suspect). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1915694340

   So in Spark the physical query plan doesn't have additional Projection under Join operator for the join key expressions, in generally we would like to keep it as it without changing the query plan structure to avoid introducing unknown issues or performance regressions. As this PR relaxes the join key constraints for DataFusion Join operators, we don't need to add such Projection during translating Spark query plan in Comet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1916225662

   I think use `expr` instead of `column` in `condition` is ok for me.
   
   Some databases materialize `expressions` into `columns` in projects mainly to facilitate the calculation of data distributions. So it should be noted that we can correctly calculate the `data distribution` of expressions in `Join Condition`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1913739384

   > This makes sense and in fact just figuring out all the steps to do is typically a large undertaking itself
   > 
   > I have had reasonable luck with partially formed Epic that we incrementally and collaboratively fill out (e.g. #8916 and #3148 )
   > 
   > If you don't object, I will file such a ticket and we can see how useful / not useful it is.
   
   No, I don't object it at all, I just feel I have no too much steps to fill there. 😄  Hope it could be useful. 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1915327141

   Thank you @alamb @mustafasrepo 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1913619978

   BTW the reason I am pushing on the epic ticket is twofold: 
   1. I want to support comet as I think it will drive DataFusion significantly forward and 
   2. I want to be better about making things that are important to committers more visible as I think there is demand across the larger community to help when there are well understood tasks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "mustafasrepo (via GitHub)" <gi...@apache.org>.
mustafasrepo commented on code in PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#discussion_r1469171957


##########
datafusion/core/src/physical_optimizer/enforce_distribution.rs:
##########
@@ -1886,8 +1887,8 @@ pub(crate) mod tests {
 
         // Join on (a == b1)
         let join_on = vec![(
-            Column::new_with_schema("a", &schema()).unwrap(),
-            Column::new_with_schema("b1", &right.schema()).unwrap(),
+            Arc::new(Column::new_with_schema("a", &schema()).unwrap()) as _,

Review Comment:
   I think we can use `col` function (which is in the  file`datafusion/physical_expr/src/expressions/column.rs`) here
   as `col("a", &schema()).unwrap()` instead of `Arc::new(Column::new_with_schema("a", &schema()).unwrap()) as _`. However, there are lots of changes like this. And it is purely stylistic. This change is not important.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1913490767

   For the creation of an EPIC style ticket, I am hesitate about it because for now I don't have all steps known to do. I may have next one step/work to do as I saw it during experimenting the integration of Spark and DataFusion with Comet on SortMergeJoin. But this is basically new attempt and I am not sure what issues could be encountered in next. Currently I'm working on it in an incremental style to solve each issue happened/seen during the integration.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1915280958

   Thanks again @viirya 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1915696558

   > As this PR relaxes the join key constraints for DataFusion Join operators, we don't need to add such Projection during translating Spark query plan in Comet.
   
   Yeah, but presumably you will have to update the Join operators to take Expressions (rather than `Column`s) and evaluate them on the inputs 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1915736893

   > Yeah, but presumably you will have to update the Join operators in DataFusion to take Expressions (rather than `Column`s) and evaluate them on the inputs
   
   I think this is already done by this PR. The Join operators in DataFusion actually already evaluate join key expressions (previously they are `Column`s). They are not simply taking the values of columns from input arrays but invoking `evaluate` physical expression API. So once we relax the join key types to general expression, it does what we need, i.e., evaluating the provided join key expressions. It saves me some works to update the operators as you said.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Relax join keys constraint from Column to any physical expression for physical join operators [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8991:
URL: https://github.com/apache/arrow-datafusion/pull/8991#issuecomment-1913619430

   > For the creation of an EPIC style ticket, I am hesitate about it because for now I don't have all steps known to do
   
   This makes sense and in fact just figuring out all the steps to do is typically a large undertaking itself
   
   I have had reasonable luck with partially formed Epic that we incrementally and collaboratively fill out (e.g. https://github.com/apache/arrow-datafusion/issues/8916 and #3148 )
   
   If you don't object, I will file such a ticket and we can see how useful / not useful it is.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org