You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by robert3005 <gi...@git.apache.org> on 2018/03/27 15:54:48 UTC
[GitHub] spark pull request #20914: [SPARK-23802][SQL] PropagateEmptyRelation can lea...
GitHub user robert3005 opened a pull request:
https://github.com/apache/spark/pull/20914
[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unresolved state
## What changes were proposed in this pull request?
Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules
## How was this patch tested?
Added unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/robert3005/spark rk/propagate-empty-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20914.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20914
----
commit 1e7dbca93fe5e0e2457ff4ba1015e170c5e59afd
Author: Robert Kruszewski <ro...@...>
Date: 2018-03-12T11:26:04Z
cast
commit 47bed8554e1f73de1f9fcf318885eb9f53629442
Author: Robert Kruszewski <ro...@...>
Date: 2018-03-27T13:22:53Z
it actually works
commit 04dd395272578500e111ccbeb928a3bea9cc19fc
Author: Robert Kruszewski <ro...@...>
Date: 2018-03-27T15:38:27Z
propagate empty fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by robert3005 <gi...@git.apache.org>.
Github user robert3005 commented on the issue:
https://github.com/apache/spark/pull/20914
`org.apache.spark.sql.execution.streaming.RateSourceV2Suite.basic microbatch execution` failed which looks like a flake to me
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88646/testReport)** for PR 20914 at commit [`be950e4`](https://github.com/apache/spark/commit/be950e4cba7334b903b3d9b414a582fe64dcdf66).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88759/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88717/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88717/testReport)** for PR 20914 at commit [`cd96401`](https://github.com/apache/spark/commit/cd964019b34a6a94e723168063d333e924551866).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88705 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88705/testReport)** for PR 20914 at commit [`d9a2ee8`](https://github.com/apache/spark/commit/d9a2ee852c56628fdaef9153463cfbf4b242f1e5).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88630/testReport)** for PR 20914 at commit [`be950e4`](https://github.com/apache/spark/commit/be950e4cba7334b903b3d9b414a582fe64dcdf66).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20914
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88705/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88630/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20914: [SPARK-23802][SQL] PropagateEmptyRelation can lea...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20914
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by robert3005 <gi...@git.apache.org>.
Github user robert3005 commented on the issue:
https://github.com/apache/spark/pull/20914
@gatorsmile how does it look now?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20914: [SPARK-23802][SQL] PropagateEmptyRelation can lea...
Posted by robert3005 <gi...@git.apache.org>.
Github user robert3005 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20914#discussion_r178168714
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala ---
@@ -107,7 +112,7 @@ class PropagateEmptyRelationSuite extends PlanTest {
val query = testRelation1
.where(left)
.join(testRelation2.where(right), joinType = jt, condition = Some('a.attr == 'b.attr))
- val optimized = Optimize.execute(query.analyze)
+ val optimized = Optimize.execute(query.analyze).analyze
--- End diff --
the subsequent analyze here is to fill in timezones on casts inserted by Optimize
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88642/testReport)** for PR 20914 at commit [`be950e4`](https://github.com/apache/spark/commit/be950e4cba7334b903b3d9b414a582fe64dcdf66).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88705 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88705/testReport)** for PR 20914 at commit [`d9a2ee8`](https://github.com/apache/spark/commit/d9a2ee852c56628fdaef9153463cfbf4b242f1e5).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88759/testReport)** for PR 20914 at commit [`1c123bd`](https://github.com/apache/spark/commit/1c123bde8e58a3400def635582c2b6daa3b7b06e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20914
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20914: [SPARK-23802][SQL] PropagateEmptyRelation can lea...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20914#discussion_r178321159
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala ---
@@ -43,7 +43,7 @@ object PropagateEmptyRelation extends Rule[LogicalPlan] with PredicateHelper {
// Construct a project list from plan's output, while the value is always NULL.
private def nullValueProjectList(plan: LogicalPlan): Seq[NamedExpression] =
- plan.output.map{ a => Alias(Literal(null), a.name)(a.exprId) }
+ plan.output.map{ a => Alias(Cast(Literal(null), a.dataType), a.name)(a.exprId) }
--- End diff --
Please use CastSupport.cast
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88642/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/20914
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20914: [SPARK-23802][SQL] PropagateEmptyRelation can lea...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20914#discussion_r178321109
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala ---
@@ -107,7 +112,7 @@ class PropagateEmptyRelationSuite extends PlanTest {
val query = testRelation1
.where(left)
.join(testRelation2.where(right), joinType = jt, condition = Some('a.attr == 'b.attr))
- val optimized = Optimize.execute(query.analyze)
+ val optimized = Optimize.execute(query.analyze).analyze
--- End diff --
We should fill the time zone in the optimizer rule
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88630 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88630/testReport)** for PR 20914 at commit [`be950e4`](https://github.com/apache/spark/commit/be950e4cba7334b903b3d9b414a582fe64dcdf66).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88642/testReport)** for PR 20914 at commit [`be950e4`](https://github.com/apache/spark/commit/be950e4cba7334b903b3d9b414a582fe64dcdf66).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88717/testReport)** for PR 20914 at commit [`cd96401`](https://github.com/apache/spark/commit/cd964019b34a6a94e723168063d333e924551866).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88759/testReport)** for PR 20914 at commit [`1c123bd`](https://github.com/apache/spark/commit/1c123bde8e58a3400def635582c2b6daa3b7b06e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/20914
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20914
**[Test build #88646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88646/testReport)** for PR 20914 at commit [`be950e4`](https://github.com/apache/spark/commit/be950e4cba7334b903b3d9b414a582fe64dcdf66).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20914: [SPARK-23802][SQL] PropagateEmptyRelation can leave quer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20914
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88646/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org