You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/27 03:47:40 UTC
[GitHub] [spark] wangyum opened a new pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for all children of NullIntolerant expression
wangyum opened a new pull request #28642:
URL: https://github.com/apache/spark/pull/28642
### What changes were proposed in this pull request?
We should infer `IsNotNull` for all children of `NullIntolerant` expressions. For example:
```sql
CREATE TABLE t1(c1 string, c2 string);
CREATE TABLE t2(c1 string, c2 string);
EXPLAIN SELECT t1.* FROM t1 JOIN t2 ON coalesce(t1.c1, t1.c2)=t2.c1;
```
After this pr the query plan is:
```
== Physical Plan ==
*(5) Project [c1#23, c2#24]
+- *(5) SortMergeJoin [coalesce(c1#23, c2#24)], [c1#25], Inner
:- *(2) Sort [coalesce(c1#23, c2#24) ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(coalesce(c1#23, c2#24), 200), true, [id=#95]
: +- *(1) Filter isnotnull(coalesce(c1#23, c2#24))
: +- Scan hive default.t1 [c1#23, c2#24], HiveTableRelation `default`.`t1`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#23, c2#24], Statistics(sizeInBytes=8.0 EiB)
+- *(4) Sort [c1#25 ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(c1#25, 200), true, [id=#103]
+- *(3) Filter isnotnull(c1#25)
+- Scan hive default.t2 [c1#25], HiveTableRelation `default`.`t2`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#25, c2#26], Statistics(sizeInBytes=8.0 EiB)
```
### Why are the changes needed?
Improve query performance and Hive support this optimization.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test and benchmark test.
Before this pr:
After this pr:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r435725708
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/InferFiltersFromConstraintsSuite.scala
##########
@@ -316,4 +316,19 @@ class InferFiltersFromConstraintsSuite extends PlanTest {
condition)
}
}
+
+ test("Infer IsNotNull for non null-intolerant child of null intolerant join condition") {
+ testConstraintsAfterJoin(
+ testRelation.subquery('left),
+ testRelation.subquery('right),
+ testRelation.where(IsNotNull(Coalesce(Seq('a, 'b)))).subquery('left),
+ testRelation.where(IsNotNull('c)).subquery('right),
+ Inner,
+ Some(Coalesce(Seq("left.a".attr, "left.b".attr)) === "right.c".attr))
+ }
+
+ test("Should not infer IsNotNull for non null-intolerant child from same table") {
+ comparePlans(Optimize.execute(testRelation.where(Coalesce(Seq('a, 'b)) === 'c).analyze),
+ testRelation.where(Coalesce(Seq('a, 'b)) === 'c && IsNotNull('c)).analyze)
Review comment:
```
hive> EXPLAIN SELECT t1.* FROM t1 WHERE coalesce(t1.a, t1.b)=t1.c;
OK
STAGE DEPENDENCIES:
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: t1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (COALESCE(a,b) = c) (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: a (type: string), b (type: string), c (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
ListSink
Time taken: 4.026 seconds, Fetched: 20 row(s)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for all children of NullIntolerant expression
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-633929364
**[Test build #123119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123119/testReport)** for PR 28642 at commit [`d657299`](https://github.com/apache/spark/commit/d65729944c966cdd8a6077bac69d5073af264f96).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for all children of NullIntolerant expression
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-633929945
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643914470
**[Test build #124032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124032/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896219608
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142276/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-899275771
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46988/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-895866583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953088064
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144659/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953358721
**[Test build #144661 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144661/testReport)** for PR 28642 at commit [`7796e5c`](https://github.com/apache/spark/commit/7796e5ceed08209f7ea6e4da15659418913a84a1).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-952971582
**[Test build #144661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144661/testReport)** for PR 28642 at commit [`7796e5c`](https://github.com/apache/spark/commit/7796e5ceed08209f7ea6e4da15659418913a84a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643944315
**[Test build #124032 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124032/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953709676
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896485211
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142299/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639299066
**[Test build #123552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123552/testReport)** for PR 28642 at commit [`a5f52a8`](https://github.com/apache/spark/commit/a5f52a8c7449bc65c35deb06ddb2b9f4bd059104).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-895866583
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142271/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639294078
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639294078
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639407937
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r439940687
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
##########
@@ -1039,7 +1039,7 @@ class JoinSuite extends QueryTest with SharedSparkSession with AdaptiveSparkPlan
val pythonEvals = collect(joinNode.get) {
case p: BatchEvalPythonExec => p
}
- assert(pythonEvals.size == 2)
+ assert(pythonEvals.size == 4)
Review comment:
Yeah, I don't think it's more efficient to have `BatchEvalPythonExec` more. It will require more Python executions which aren't trivial.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-644100154
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-1034340573
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953709709
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-895938308
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46779/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for all children of NullIntolerant expression
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-633929945
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-644098962
**[Test build #124038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124038/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896411160
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896416863
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896064569
**[Test build #142276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142276/testReport)** for PR 28642 at commit [`b100902`](https://github.com/apache/spark/commit/b100902b24463ad8552cbde279f604938c3bc31a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639293643
**[Test build #123552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123552/testReport)** for PR 28642 at commit [`a5f52a8`](https://github.com/apache/spark/commit/a5f52a8c7449bc65c35deb06ddb2b9f4bd059104).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639302208
**[Test build #123553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123553/testReport)** for PR 28642 at commit [`a5f52a8`](https://github.com/apache/spark/commit/a5f52a8c7449bc65c35deb06ddb2b9f4bd059104).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r435725267
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/InferFiltersFromConstraintsSuite.scala
##########
@@ -316,4 +316,19 @@ class InferFiltersFromConstraintsSuite extends PlanTest {
condition)
}
}
+
+ test("Infer IsNotNull for non null-intolerant child of null intolerant join condition") {
+ testConstraintsAfterJoin(
+ testRelation.subquery('left),
+ testRelation.subquery('right),
+ testRelation.where(IsNotNull(Coalesce(Seq('a, 'b)))).subquery('left),
Review comment:
```
hive> EXPLAIN SELECT t1.* FROM t1 JOIN t2 ON coalesce(t1.a, t1.b)=t2.a;
OK
STAGE DEPENDENCIES:
Stage-4 is a root stage
Stage-3 depends on stages: Stage-4
Stage-0 depends on stages: Stage-3
STAGE PLANS:
Stage: Stage-4
Map Reduce Local Work
Alias -> Map Local Tables:
$hdt$_0:t1
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
$hdt$_0:t1
TableScan
alias: t1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: COALESCE(a,b) is not null (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: a (type: string), b (type: string), c (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
HashTable Sink Operator
keys:
0 COALESCE(_col0,_col1) (type: string)
1 _col0 (type: string)
Stage: Stage-3
Map Reduce
Map Operator Tree:
TableScan
alias: t2
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: a is not null (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: a (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Map Join Operator
condition map:
Inner Join 0 to 1
keys:
0 COALESCE(_col0,_col1) (type: string)
1 _col0 (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: vectorized
Local Work:
Map Reduce Local Work
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643948913
**[Test build #124038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124038/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896396583
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-895933596
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46779/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #28642:
URL: https://github.com/apache/spark/pull/28642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953514517
**[Test build #144680 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144680/testReport)** for PR 28642 at commit [`c88566a`](https://github.com/apache/spark/commit/c88566ae4c9b1d14cfc732031d14fca331e7f51f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953666657
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953404661
**[Test build #144680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144680/testReport)** for PR 28642 at commit [`c88566a`](https://github.com/apache/spark/commit/c88566ae4c9b1d14cfc732031d14fca331e7f51f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r738074669
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1215,6 +1215,14 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
}
}
+ // Whether the result of this expression may be null. For example: CAST(strCol AS double)
+ // We will infer an IsNotNull expression for this expression to avoid skew join.
+ private def resultMayBeNull(exp: Expression): Boolean = exp match {
Review comment:
+1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953571266
**[Test build #144703 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144703/testReport)** for PR 28642 at commit [`919492e`](https://github.com/apache/spark/commit/919492ee6095a007077119a11425abd7f08b02f6).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r739105789
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1215,6 +1215,15 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
}
}
+ // Whether the result of this expression may be null. For example: CAST(strCol AS double)
+ // We will infer an IsNotNull expression for this expression to avoid skew join.
+ private def resultMayBeNull(exp: Expression): Boolean = exp match {
+ case e if !e.nullable => false
+ case Cast(child: Attribute, dataType, _, _) => !Cast.canUpCast(child.dataType, dataType)
+ case c: Coalesce if c.children.forall(_.isInstanceOf[Attribute]) => true
Review comment:
We can infer `NullIntolerant` already. For example:
```
spark.sql("create table t1 (id string, value int) using parquet")
spark.sql("create table t2 (id int, value int) using parquet")
spark.sql("select * from t1 join t2 on t1.id = t2.id").explain("extended")
== Optimized Logical Plan ==
Join Inner, (cast(id#0 as int) = id#2)
:- Filter isnotnull(id#0)
: +- Relation default.t1[id#0,value#1] parquet
+- Filter isnotnull(id#2)
+- Relation default.t2[id#2,value#3] parquet
```
`Cast` is `NullIntolerant`. We can infer `IsNotNull(t1.id)` already. But I also want to Infer `isnotnull(cast(t1.id as int))` because `t1.id` may contains many strings that can not be casted to int.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for all children of NullIntolerant expression
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-633929364
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639300351
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-640272817
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-698046330
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639302659
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-640238262
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r735412472
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1102,30 +1103,43 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
filter
}
- case join @ Join(left, right, joinType, conditionOpt, _) =>
- joinType match {
- // For inner join, we can infer additional filters for both sides. LeftSemi is kind of an
- // inner join, it just drops the right side in the final output.
- case _: InnerLike | LeftSemi =>
- val allConstraints = getAllConstraints(left, right, conditionOpt)
- val newLeft = inferNewFilter(left, allConstraints)
- val newRight = inferNewFilter(right, allConstraints)
- join.copy(left = newLeft, right = newRight)
+ case join @ ExtractEquiJoinKeys(_, leftKeys, rightKeys, _, _, _, _) =>
Review comment:
`ExtractEquiJoinKeys` also extract keys, that are compared with `EqualNullSafe` - we should not infer `IsNotNull` for those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r735412472
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1102,30 +1103,43 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
filter
}
- case join @ Join(left, right, joinType, conditionOpt, _) =>
- joinType match {
- // For inner join, we can infer additional filters for both sides. LeftSemi is kind of an
- // inner join, it just drops the right side in the final output.
- case _: InnerLike | LeftSemi =>
- val allConstraints = getAllConstraints(left, right, conditionOpt)
- val newLeft = inferNewFilter(left, allConstraints)
- val newRight = inferNewFilter(right, allConstraints)
- join.copy(left = newLeft, right = newRight)
+ case join @ ExtractEquiJoinKeys(_, leftKeys, rightKeys, _, _, _, _) =>
Review comment:
`ExtractEquiJoinKeys` extracts keys, that are compared with `EqualNullSafe` - we should not infer `IsNotNull` for those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953375266
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144661/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953515380
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144680/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896101595
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46784/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896484364
**[Test build #142299 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142299/testReport)** for PR 28642 at commit [`3643e3f`](https://github.com/apache/spark/commit/3643e3f26ac8755a4da1e83b36e246cb16b48275).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r436442434
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
##########
@@ -1039,7 +1039,7 @@ class JoinSuite extends QueryTest with SharedSparkSession with AdaptiveSparkPlan
val pythonEvals = collect(joinNode.get) {
case p: BatchEvalPythonExec => p
}
- assert(pythonEvals.size == 2)
+ assert(pythonEvals.size == 4)
Review comment:
@HyukjinKwon I'm not sure if this change can optimize python udf?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643949432
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643944449
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639407595
**[Test build #123553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123553/testReport)** for PR 28642 at commit [`a5f52a8`](https://github.com/apache/spark/commit/a5f52a8c7449bc65c35deb06ddb2b9f4bd059104).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639302659
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643949432
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643913716
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r735412472
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1102,30 +1103,43 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
filter
}
- case join @ Join(left, right, joinType, conditionOpt, _) =>
- joinType match {
- // For inner join, we can infer additional filters for both sides. LeftSemi is kind of an
- // inner join, it just drops the right side in the final output.
- case _: InnerLike | LeftSemi =>
- val allConstraints = getAllConstraints(left, right, conditionOpt)
- val newLeft = inferNewFilter(left, allConstraints)
- val newRight = inferNewFilter(right, allConstraints)
- join.copy(left = newLeft, right = newRight)
+ case join @ ExtractEquiJoinKeys(_, leftKeys, rightKeys, _, _, _, _) =>
Review comment:
`ExtractEquiJoinKeys` extracts keys, that are compared with `EqualNullSafe` - we should not infer `IsNotNull` for those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-952861844
**[Test build #144659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144659/testReport)** for PR 28642 at commit [`a9eb7de`](https://github.com/apache/spark/commit/a9eb7de5a76a407c6dfda3ce1d9a91cfb956f3a2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953059910
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49130/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953420520
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49149/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953571315
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144703/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953375266
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144661/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-899283181
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46988/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-895866544
**[Test build #142271 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142271/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
* This patch **fails Python style tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-640238082
**[Test build #123607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123607/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-640272817
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-952896823
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49129/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953709709
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-952939809
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49129/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-952861844
**[Test build #144659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144659/testReport)** for PR 28642 at commit [`a9eb7de`](https://github.com/apache/spark/commit/a9eb7de5a76a407c6dfda3ce1d9a91cfb956f3a2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953404661
**[Test build #144680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144680/testReport)** for PR 28642 at commit [`c88566a`](https://github.com/apache/spark/commit/c88566ae4c9b1d14cfc732031d14fca331e7f51f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953515380
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144680/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r739070103
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1215,6 +1215,15 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
}
}
+ // Whether the result of this expression may be null. For example: CAST(strCol AS double)
+ // We will infer an IsNotNull expression for this expression to avoid skew join.
Review comment:
is it better to infer `IsNotNull(col)` instead of `IsNotNull(CAST(col AS other_type))`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896378066
**[Test build #142299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142299/testReport)** for PR 28642 at commit [`3643e3f`](https://github.com/apache/spark/commit/3643e3f26ac8755a4da1e83b36e246cb16b48275).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-895865723
**[Test build #142271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142271/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-640272578
**[Test build #123607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123607/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-640238082
**[Test build #123607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123607/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643946838
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643914470
**[Test build #124032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124032/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643944467
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124032/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643944449
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639299132
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953442545
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49149/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953571315
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144703/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r737478319
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1215,6 +1215,14 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
}
}
+ // Whether the result of this expression may be null. For example: CAST(strCol AS double)
+ // We will infer an IsNotNull expression for this expression to avoid skew join.
+ private def resultMayBeNull(e: Expression): Boolean = e match {
+ case Cast(child, dataType, _, _) => !Cast.canUpCast(child.dataType, dataType)
+ case _: Coalesce => true
+ case _ => false
+ }
Review comment:
@cloud-fan @HyukjinKwon It will not infer all equality join keys. For example:
Infer | Will not infer
-- | --
cast(strCol AS double) = doubleCol | upper(strCol) = upperStrCol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-952968246
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49129/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953087193
**[Test build #144659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144659/testReport)** for PR 28642 at commit [`a9eb7de`](https://github.com/apache/spark/commit/a9eb7de5a76a407c6dfda3ce1d9a91cfb956f3a2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953088064
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144659/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953078881
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49130/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-895928638
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46779/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896218413
**[Test build #142276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142276/testReport)** for PR 28642 at commit [`b100902`](https://github.com/apache/spark/commit/b100902b24463ad8552cbde279f604938c3bc31a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-644100154
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639299139
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123552/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639407947
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123553/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639302208
**[Test build #123553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123553/testReport)** for PR 28642 at commit [`a5f52a8`](https://github.com/apache/spark/commit/a5f52a8c7449bc65c35deb06ddb2b9f4bd059104).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639299066
**[Test build #123552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123552/testReport)** for PR 28642 at commit [`a5f52a8`](https://github.com/apache/spark/commit/a5f52a8c7449bc65c35deb06ddb2b9f4bd059104).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-895865723
**[Test build #142271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142271/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896064569
**[Test build #142276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142276/testReport)** for PR 28642 at commit [`b100902`](https://github.com/apache/spark/commit/b100902b24463ad8552cbde279f604938c3bc31a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-899283181
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46988/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-952968246
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49129/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r737478319
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1215,6 +1215,14 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
}
}
+ // Whether the result of this expression may be null. For example: CAST(strCol AS double)
+ // We will infer an IsNotNull expression for this expression to avoid skew join.
+ private def resultMayBeNull(e: Expression): Boolean = e match {
+ case Cast(child, dataType, _, _) => !Cast.canUpCast(child.dataType, dataType)
+ case _: Coalesce => true
+ case _ => false
+ }
Review comment:
@cloud-fan @HyukjinKwon It will not infer all equality join keys. For example:
Infer | Will not infer
-- | --
cast(strCol AS double) = doubleCol | upper(strCol) = upperStrCol
cast(bigintCol AS int) = intCol | cast(intCol AS bigint) = bigintCol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953442545
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49149/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-952971582
**[Test build #144661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144661/testReport)** for PR 28642 at commit [`7796e5c`](https://github.com/apache/spark/commit/7796e5ceed08209f7ea6e4da15659418913a84a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953567062
**[Test build #144703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144703/testReport)** for PR 28642 at commit [`919492e`](https://github.com/apache/spark/commit/919492ee6095a007077119a11425abd7f08b02f6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953567062
**[Test build #144703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144703/testReport)** for PR 28642 at commit [`919492e`](https://github.com/apache/spark/commit/919492ee6095a007077119a11425abd7f08b02f6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r739069025
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1215,6 +1215,15 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
}
}
+ // Whether the result of this expression may be null. For example: CAST(strCol AS double)
+ // We will infer an IsNotNull expression for this expression to avoid skew join.
+ private def resultMayBeNull(exp: Expression): Boolean = exp match {
+ case e if !e.nullable => false
+ case Cast(child: Attribute, dataType, _, _) => !Cast.canUpCast(child.dataType, dataType)
+ case c: Coalesce if c.children.forall(_.isInstanceOf[Attribute]) => true
Review comment:
Can't we rely on the `NullIntolerant` interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r440020133
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
##########
@@ -1039,7 +1039,7 @@ class JoinSuite extends QueryTest with SharedSparkSession with AdaptiveSparkPlan
val pythonEvals = collect(joinNode.get) {
case p: BatchEvalPythonExec => p
}
- assert(pythonEvals.size == 2)
+ assert(pythonEvals.size == 4)
Review comment:
I quickly checked:
```
== Physical Plan ==
*(3) Project [a#225, b#226, c#236, d#237]
+- *(3) BroadcastHashJoin [cast(pythonUDF0#256 as int)], [cast(pythonUDF0#257 as int)], Inner, BuildRight
:- BatchEvalPython [udf(cast(a#225 as string))], [pythonUDF0#256]
: +- *(1) Project [_1#220 AS a#225, _2#221 AS b#226]
: +- *(1) Project [_1#220, _2#221]
: +- *(1) Filter isnotnull(cast(pythonUDF0#254 as int))
: +- BatchEvalPython [udf(cast(_1#220 as string))], [pythonUDF0#254]
: +- LocalTableScan [_1#220, _2#221]
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(cast(input[2, string, true] as int) as bigint))), [id=#140]
+- BatchEvalPython [udf(cast(c#236 as string))], [pythonUDF0#257]
+- *(2) Project [_1#231 AS c#236, _2#232 AS d#237]
+- *(2) Project [_1#231, _2#232]
+- *(2) Filter isnotnull(cast(pythonUDF0#255 as int))
+- BatchEvalPython [udf(cast(_1#231 as string))], [pythonUDF0#255]
+- LocalTableScan [_1#231, _2#232]
```
We should probably avoid inferring the is-not-null filter in this case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643914834
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643914834
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639299132
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum closed pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
wangyum closed pull request #28642:
URL: https://github.com/apache/spark/pull/28642
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643948913
**[Test build #124038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124038/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0c7777a9bae7c524b40c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-643994833
cc @cloud-fan FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-640238262
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639407937
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-639293643
**[Test build #123552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123552/testReport)** for PR 28642 at commit [`a5f52a8`](https://github.com/apache/spark/commit/a5f52a8c7449bc65c35deb06ddb2b9f4bd059104).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-899257491
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46988/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953078881
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49130/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r739098037
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1215,6 +1215,15 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
}
}
+ // Whether the result of this expression may be null. For example: CAST(strCol AS double)
+ // We will infer an IsNotNull expression for this expression to avoid skew join.
Review comment:
We can infer `IsNotNull(col)` already. For example:
```scala
spark.sql("create table t1 (id string, value int) using parquet")
spark.sql("create table t2 (id int, value int) using parquet")
spark.sql("select * from t1 join t2 on t1.id = t2.id").explain("extended")
```
Before this pr:
```
== Optimized Logical Plan ==
Join Inner, (cast(id#0 as int) = id#2)
:- Filter isnotnull(id#0)
: +- Relation default.t1[id#0,value#1] parquet
+- Filter isnotnull(id#2)
+- Relation default.t2[id#2,value#3] parquet
```
After this pr:
```
== Optimized Logical Plan ==
Join Inner, (cast(id#0 as int) = id#2)
:- Filter (isnotnull(id#0) AND isnotnull(cast(id#0 as int)))
: +- Relation default.t1[id#0,value#1] parquet
+- Filter isnotnull(id#2)
+- Relation default.t2[id#2,value#3] parquet
```
Infer `isnotnull(cast(t1.id as int))` may filter out many strings that can not be casted to int.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on a change in pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #28642:
URL: https://github.com/apache/spark/pull/28642#discussion_r738038077
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1215,6 +1215,14 @@ object InferFiltersFromConstraints extends Rule[LogicalPlan]
}
}
+ // Whether the result of this expression may be null. For example: CAST(strCol AS double)
+ // We will infer an IsNotNull expression for this expression to avoid skew join.
+ private def resultMayBeNull(exp: Expression): Boolean = exp match {
Review comment:
perhaps also check the nullability of exp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953018237
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49130/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from some special equality join keys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-953441174
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49149/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28642:
URL: https://github.com/apache/spark/pull/28642#issuecomment-896378066
**[Test build #142299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142299/testReport)** for PR 28642 at commit [`3643e3f`](https://github.com/apache/spark/commit/3643e3f26ac8755a4da1e83b36e246cb16b48275).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org