You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/03 23:41:19 UTC
[GitHub] [spark] maryannxue opened a new pull request #27446:
[SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
maryannxue opened a new pull request #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446
### What changes were proposed in this pull request?
This PR is to fix a potential bug in AQE where an `ExecSubqueryExpression` could be mistakenly replaced with another `ExecSubqueryExpression` with the same `ListQuery` but a different `child` expression.
This is because a ListQuery's id can only identify the ListQuery itself, not the parent expression `InSubquery`, but right now the `subqueryMap` in `InsertAdaptiveSparkPlan` uses the `ListQuery`'s id as key and the corresponding `InSubqueryExec` for the `ListQuery`'s parent expression as value. So the fix uses the corresponding `SubqueryExec` for the `ListQuery` itself as the map's value.
### Why are the changes needed?
This logical bug could potentially cause a wrong query plan, which could throw an exception related to unresolved columns.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Passed existing UTs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27446:
[SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581673807
**[Test build #117790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117790/testReport)** for PR 27446 at commit [`d455ff0`](https://github.com/apache/spark/commit/d455ff01c7156426cc77ed8f5e50a98c5736c3de).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27446: [SPARK-30717][SQL] AQE
subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581738235
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27446:
[SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581728269
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117790/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27446: [SPARK-30717][SQL]
AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581728269
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117790/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27446: [SPARK-30717][SQL]
AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581674160
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27446:
[SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581728265
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27446: [SPARK-30717][SQL]
AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581728265
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27446: [SPARK-30717][SQL] AQE
subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581725444
good catch! LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27446:
[SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581674164
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22553/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27446: [SPARK-30717][SQL]
AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581674164
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22553/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27446:
[SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581674160
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27446: [SPARK-30717][SQL] AQE
subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581727926
**[Test build #117790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117790/testReport)** for PR 27446 at commit [`d455ff0`](https://github.com/apache/spark/commit/d455ff01c7156426cc77ed8f5e50a98c5736c3de).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class PlanAdaptiveSubqueries(subqueryMap: Map[Long, SubqueryExec]) extends Rule[SparkPlan] `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27446: [SPARK-30717][SQL] AQE
subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581673807
**[Test build #117790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117790/testReport)** for PR 27446 at commit [`d455ff0`](https://github.com/apache/spark/commit/d455ff01c7156426cc77ed8f5e50a98c5736c3de).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27446: [SPARK-30717][SQL]
AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maryannxue commented on issue #27446: [SPARK-30717][SQL]
AQE subquery map should cache `SubqueryExec` instead of
`ExecSubqueryExpression`
Posted by GitBox <gi...@apache.org>.
maryannxue commented on issue #27446: [SPARK-30717][SQL] AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`
URL: https://github.com/apache/spark/pull/27446#issuecomment-581673139
cc @cloud-fan @JkSelf
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org