You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/28 04:18:12 UTC
[GitHub] [spark] cxzl25 opened a new pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
cxzl25 opened a new pull request #28383:
URL: https://github.com/apache/spark/pull/28383
### What changes were proposed in this pull request?
When the filter used by OptimizeMetadataOnlyQuery is Unevaluable, this rule should not be used.
### Why are the changes needed?
Applying `OptimizeMetadataOnlyQuery` rule will generate `scalar-subquery`.
Apply the `OptimizeMetadataOnlyQuery` rule again, will get the exception `Cannot evaluate expression: scalar-subquery`.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
add UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-620372657
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang edited a comment on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
gengliangwang edited a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623875587
> Shall we remove OptimizeMetadataOnlyQuery? IIRC it has a correcness issue and we disable it by default. cc @gengliangwang
On second thought: I think we should keep it for two reasons:
1. when users are 100% sure about their data won't contain empty partition, they can still turn it on.
2. the future developers may come up with the same idea and create exactly the same rule and enable it by default...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622324894
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623313693
**[Test build #122256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122256/testReport)** for PR 28383 at commit [`d601af4`](https://github.com/apache/spark/commit/d601af42e32cd74e99c691b6870eebf17845ae09).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r419817895
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala
##########
@@ -117,7 +117,7 @@ case class OptimizeMetadataOnlyQuery(catalog: SessionCatalog) extends Rule[Logic
case a: AttributeReference =>
a.withName(relation.output.find(_.semanticEquals(a)).get.name)
}
- }
Review comment:
Could you filter out this unsupported case in advance? e.g., in https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala#L53-L55
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622661598
**[Test build #122189 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122189/testReport)** for PR 28383 at commit [`7046db8`](https://github.com/apache/spark/commit/7046db81d1a830c97a20f8e690cfb507c712da82).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622900591
**[Test build #122210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122210/testReport)** for PR 28383 at commit [`a7638d6`](https://github.com/apache/spark/commit/a7638d602fca15911c99fd14703f144902f68f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622256412
**[Test build #122161 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122161/testReport)** for PR 28383 at commit [`c848508`](https://github.com/apache/spark/commit/c8485086ffee7e5ca2ad7a1e24e10081070d135d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-620372971
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-620372971
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r419817895
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala
##########
@@ -117,7 +117,7 @@ case class OptimizeMetadataOnlyQuery(catalog: SessionCatalog) extends Rule[Logic
case a: AttributeReference =>
a.withName(relation.output.find(_.semanticEquals(a)).get.name)
}
- }
Review comment:
Could you filter out this unsupported case outside `replaceTableScanWithPartitionMetadata`(I think this filtering is not related to normalization)? e.g., in https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala#L53-L55
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622279633
**[Test build #122165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122165/testReport)** for PR 28383 at commit [`4abf2f5`](https://github.com/apache/spark/commit/4abf2f5ac87296497cee68035ad7c0ea91cf3547).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622241833
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622763883
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122189/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622357073
**[Test build #122165 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122165/testReport)** for PR 28383 at commit [`4abf2f5`](https://github.com/apache/spark/commit/4abf2f5ac87296497cee68035ad7c0ea91cf3547).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622660745
@cxzl25 can you revert the test back to the original one and focus on the cleanup? The case before was a valid, and failed in the master. The fix itself seems right too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622280000
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622663517
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418415403
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuerySuite.scala
##########
@@ -150,4 +150,30 @@ class OptimizeMetadataOnlyQuerySuite extends QueryTest with SharedSparkSession {
}
}
}
+
+ test("SPARK-31590 The filter used by Metadata-only queries should not have Unevaluable") {
+ withTable("test_tbl") {
+ withSQLConf(OPTIMIZER_METADATA_ONLY.key -> "true") {
+ sql("CREATE TABLE test_tbl (a INT,d STRING,h STRING) USING PARQUET PARTITIONED BY (d ,h)")
Review comment:
Can you make the test case minimised, and consistent with the style used in this file? I think you can create the partitioned table via `write.parquetby` syntax instead of relying on the SQL syntax here even when you create tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622661681
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622324894
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622763815
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418857861
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala
##########
@@ -117,7 +117,7 @@ case class OptimizeMetadataOnlyQuery(catalog: SessionCatalog) extends Rule[Logic
case a: AttributeReference =>
a.withName(relation.output.find(_.semanticEquals(a)).get.name)
}
- }
+ }.filterNot(SubqueryExpression.hasSubquery)
Review comment:
Yeah let's keep the PR title.and description matched ..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622308348
**[Test build #122157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122157/testReport)** for PR 28383 at commit [`86f28d5`](https://github.com/apache/spark/commit/86f28d51f343becc6a516f726d20efd4d12c7708).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622243294
**[Test build #122157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122157/testReport)** for PR 28383 at commit [`86f28d5`](https://github.com/apache/spark/commit/86f28d51f343becc6a516f726d20efd4d12c7708).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622663422
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623313693
**[Test build #122256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122256/testReport)** for PR 28383 at commit [`d601af4`](https://github.com/apache/spark/commit/d601af42e32cd74e99c691b6870eebf17845ae09).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622309019
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418768888
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuerySuite.scala
##########
@@ -103,6 +103,20 @@ class OptimizeMetadataOnlyQuerySuite extends QueryTest with SharedSparkSession {
"select partcol2, min(partcol1) from srcpart where partcol1 = 0 group by partcol2",
"select max(c1) from (select partcol1 + 1 as c1 from srcpart where partcol1 = 0) t")
+ testMetadataOnly(
+ "SPARK-31590 The filter used by Metadata-only queries should not have Unevaluable",
+ """
+ |SELECT partcol1, MAX(partcol2) AS partcol2
+ |FROM srcpart
+ |WHERE partcol1 = (
+ | SELECT MAX(partcol1)
+ | FROM srcpart
+ |)
+ |AND partcol2= 'event'
+ |GROUP BY partcol1
+ |""".stripMargin
+ )
Review comment:
I run this test on the master and it passed. Could you check this again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418415403
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuerySuite.scala
##########
@@ -150,4 +150,30 @@ class OptimizeMetadataOnlyQuerySuite extends QueryTest with SharedSparkSession {
}
}
}
+
+ test("SPARK-31590 The filter used by Metadata-only queries should not have Unevaluable") {
+ withTable("test_tbl") {
+ withSQLConf(OPTIMIZER_METADATA_ONLY.key -> "true") {
+ sql("CREATE TABLE test_tbl (a INT,d STRING,h STRING) USING PARQUET PARTITIONED BY (d ,h)")
Review comment:
Can you make the test case minimised, and consistent with the style used in this file? I think you can create the partitioned table via `write.parquetby` syntax instead of relying on the SQL syntax here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622256704
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r419036306
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuerySuite.scala
##########
@@ -103,6 +103,21 @@ class OptimizeMetadataOnlyQuerySuite extends QueryTest with SharedSparkSession {
"select partcol2, min(partcol1) from srcpart where partcol1 = 0 group by partcol2",
"select max(c1) from (select partcol1 + 1 as c1 from srcpart where partcol1 = 0) t")
+ testMetadataOnly(
+ "SPARK-31590 " +
+ "The filter used by Metadata-only queries should filter out all the unevaluable expr",
Review comment:
This isn't correct. Can you fix the title and scope it down only about subqueries?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623451828
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622357607
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623451828
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622256412
**[Test build #122161 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122161/testReport)** for PR 28383 at commit [`c848508`](https://github.com/apache/spark/commit/c8485086ffee7e5ca2ad7a1e24e10081070d135d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418764874
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuerySuite.scala
##########
@@ -103,6 +103,20 @@ class OptimizeMetadataOnlyQuerySuite extends QueryTest with SharedSparkSession {
"select partcol2, min(partcol1) from srcpart where partcol1 = 0 group by partcol2",
"select max(c1) from (select partcol1 + 1 as c1 from srcpart where partcol1 = 0) t")
+ testMetadataOnly(
+ "SPARK-31590 The filter used by Metadata-only queries should not have Unevaluable",
+ """
+ |SELECT partcol1, MAX(partcol2) AS partcol2
+ |FROM srcpart
+ |WHERE partcol1 = (
+ | SELECT MAX(partcol1)
+ | FROM srcpart
+ |)
+ |AND partcol2= 'event'
Review comment:
nit: `AND partcol2 = 'event'`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622663422
**[Test build #122190 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122190/testReport)** for PR 28383 at commit [`a7638d6`](https://github.com/apache/spark/commit/a7638d602fca15911c99fd14703f144902f68f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622256704
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418415403
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuerySuite.scala
##########
@@ -150,4 +150,30 @@ class OptimizeMetadataOnlyQuerySuite extends QueryTest with SharedSparkSession {
}
}
}
+
+ test("SPARK-31590 The filter used by Metadata-only queries should not have Unevaluable") {
+ withTable("test_tbl") {
+ withSQLConf(OPTIMIZER_METADATA_ONLY.key -> "true") {
+ sql("CREATE TABLE test_tbl (a INT,d STRING,h STRING) USING PARQUET PARTITIONED BY (d ,h)")
Review comment:
Can you make the test case minimised, and consistent with the style used in this file? I think you can create the partitioned table via `write.parquetby` syntax instead of relying on the SQL syntax here even when you create tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622463844
@viirya and @cloud-fan fyi
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622279633
**[Test build #122165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122165/testReport)** for PR 28383 at commit [`4abf2f5`](https://github.com/apache/spark/commit/4abf2f5ac87296497cee68035ad7c0ea91cf3547).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622280000
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418767134
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala
##########
@@ -117,7 +117,7 @@ case class OptimizeMetadataOnlyQuery(catalog: SessionCatalog) extends Rule[Logic
case a: AttributeReference =>
a.withName(relation.output.find(_.semanticEquals(a)).get.name)
}
- }
+ }.filterNot(SubqueryExpression.hasSubquery)
Review comment:
We don't need to filter out all the unevaluable exprs here? The PR title should say so though...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622663517
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622243652
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622309019
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623314492
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418773629
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuerySuite.scala
##########
@@ -103,6 +103,20 @@ class OptimizeMetadataOnlyQuerySuite extends QueryTest with SharedSparkSession {
"select partcol2, min(partcol1) from srcpart where partcol1 = 0 group by partcol2",
"select max(c1) from (select partcol1 + 1 as c1 from srcpart where partcol1 = 0) t")
+ testMetadataOnly(
+ "SPARK-31590 The filter used by Metadata-only queries should not have Unevaluable",
+ """
+ |SELECT partcol1, MAX(partcol2) AS partcol2
+ |FROM srcpart
+ |WHERE partcol1 = (
+ | SELECT MAX(partcol1)
+ | FROM srcpart
+ |)
+ |AND partcol2= 'event'
+ |GROUP BY partcol1
+ |""".stripMargin
+ )
Review comment:
yea, I just ran it and passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623044277
@cxzl25, I think https://github.com/apache/spark/pull/28383#discussion_r418767134 isn't fully addressed. Can you fix the PR description to explain fully what this PR proposes? This PR doesn't filter unevaluable expressions but only sub-queries because their results are only available during runtime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623875587
> Shall we remove OptimizeMetadataOnlyQuery? IIRC it has a correcness issue and we disable it by default. cc @gengliangwang
I am +1 with that. The correctness issue is the one I reported in https://github.com/apache/spark/pull/23635
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418408328
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala
##########
@@ -119,6 +119,10 @@ case class OptimizeMetadataOnlyQuery(catalog: SessionCatalog) extends Rule[Logic
}
}
+ if (normalizedFilters.exists(_.find(_.isInstanceOf[Unevaluable]).isDefined)) {
+ return child
+ }
Review comment:
Why don't you just filter out subquerties consistently with other normalized filters, by `normalizedFilters.filterNot(SubqueryExpression.hasSubquery)`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622968977
**[Test build #122210 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122210/testReport)** for PR 28383 at commit [`a7638d6`](https://github.com/apache/spark/commit/a7638d602fca15911c99fd14703f144902f68f47).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622969317
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622357607
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622902561
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622763815
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622902561
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622243294
**[Test build #122157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122157/testReport)** for PR 28383 at commit [`86f28d5`](https://github.com/apache/spark/commit/86f28d51f343becc6a516f726d20efd4d12c7708).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418417828
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuerySuite.scala
##########
@@ -150,4 +150,30 @@ class OptimizeMetadataOnlyQuerySuite extends QueryTest with SharedSparkSession {
}
}
}
+
+ test("SPARK-31590 The filter used by Metadata-only queries should not have Unevaluable") {
+ withTable("test_tbl") {
+ withSQLConf(OPTIMIZER_METADATA_ONLY.key -> "true") {
+ sql("CREATE TABLE test_tbl (a INT,d STRING,h STRING) USING PARQUET PARTITIONED BY (d ,h)")
Review comment:
Can we reuse `testMetadataOnly`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-624009214
I think we can just mention that it is discouraged to use that configuration for now. We cant just remove the configuration without deprecation anyway and the fix looks correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622661681
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622893128
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622324253
**[Test build #122161 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122161/testReport)** for PR 28383 at commit [`c848508`](https://github.com/apache/spark/commit/c8485086ffee7e5ca2ad7a1e24e10081070d135d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622243652
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-620372657
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623450486
**[Test build #122256 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122256/testReport)** for PR 28383 at commit [`d601af4`](https://github.com/apache/spark/commit/d601af42e32cd74e99c691b6870eebf17845ae09).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623861615
Shall we remove `OptimizeMetadataOnlyQuery`? IIRC it has a correcness issue and we disable it by default. cc @gengliangwang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622900591
**[Test build #122210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122210/testReport)** for PR 28383 at commit [`a7638d6`](https://github.com/apache/spark/commit/a7638d602fca15911c99fd14703f144902f68f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-624403329
Merged to master, branch-3.0, and branch-2.4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623314492
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28383: [SPARK-31590][SQL] Metadata-only queries should not include subquery in partition filters
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-623791024
> Applying OptimizeMetadataOnlyQuery rule will generate scalar-subquery.
Is this statement true? It seems the test query itself has a subquery.
```
// Analyzed plan of the test query
Aggregate [partcol1#40], [partcol1#40, max(partcol2#41) AS partcol2#71]
+- Filter ((partcol1#40 = scalar-subquery#70 []) AND (partcol2#41 = even))
: +- Aggregate [max(partcol1#40) AS max(partcol1)#73]
: +- SubqueryAlias spark_catalog.default.srcpart
: +- Relation[col1#38,col2#39,partcol1#40,partcol2#41] parquet
+- SubqueryAlias spark_catalog.default.srcpart
+- Relation[col1#38,col2#39,partcol1#40,partcol2#41] parquet
```
I think the root cause is just that unsupported `partitionFilters` (subquery) is passed into `FileIndex.listFiles`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should not have Unevaluable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622248593
Looks fine to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622762904
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on a change in pull request #28383:
URL: https://github.com/apache/spark/pull/28383#discussion_r418869108
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuerySuite.scala
##########
@@ -103,6 +103,20 @@ class OptimizeMetadataOnlyQuerySuite extends QueryTest with SharedSparkSession {
"select partcol2, min(partcol1) from srcpart where partcol1 = 0 group by partcol2",
"select max(c1) from (select partcol1 + 1 as c1 from srcpart where partcol1 = 0) t")
+ testMetadataOnly(
+ "SPARK-31590 The filter used by Metadata-only queries should not have Unevaluable",
+ """
+ |SELECT partcol1, MAX(partcol2) AS partcol2
+ |FROM srcpart
+ |WHERE partcol1 = (
+ | SELECT MAX(partcol1)
+ | FROM srcpart
+ |)
+ |AND partcol2= 'event'
+ |GROUP BY partcol1
+ |""".stripMargin
+ )
Review comment:
Sorry, I typed an extra letter t, it should be the existing partition value to filter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28383: [SPARK-31590][SQL] The filter used by Metadata-only queries should filter out all the unevaluable expr
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28383:
URL: https://github.com/apache/spark/pull/28383#issuecomment-622969317
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org