You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/29 02:50:40 UTC
[GitHub] [spark] cxzl25 opened a new pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
cxzl25 opened a new pull request #34431:
URL: https://github.com/apache/spark/pull/34431
### What changes were proposed in this pull request?
Improve partition filtering speed and reduce metastore pressure.
We can first pull all the partition names, filter by expressions, and then obtain detailed information about the corresponding partitions from the MetaStore Server.
### Why are the changes needed?
When `convertFilters` cannot take effect, cannot filter the queried partitions in advance on the hive MetaStore Server. At this time, `getAllPartitionsOf` will get all partition details.
When the Hive client cannot use the server filter, it will first obtain the values of all partitions, and then filter.
When we have a table with a lot of partitions and there is no way to filter it on the MetaStore Server, we will get all the partition details and filter it on the client side. This is slow and puts a lot of pressure on the MetaStore Server.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Add UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954695534
**[Test build #144755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144755/testReport)** for PR 34431 at commit [`6916668`](https://github.com/apache/spark/commit/6916668aab2801e278a19eedc661f822dff75e34).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954380012
According to the comment here https://github.com/apache/spark/pull/32583#issuecomment-953691402.
There is currently no way to solve the unload class problem in `IsolatedClientLoader`, so a compromise solution of cache hive instance is used in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954488738
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954439666
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144734/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958712541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957097483
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957117831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957117831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744294946
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -577,6 +582,60 @@ class HivePartitionFilteringSuite(version: String)
dateStrValue)
}
+ test("getPartitionsByFilter: substr(chunk,0,1)=a") {
Review comment:
I probably remember that UT does not need to add a jira number if it is not for bug fixes.
I still add the jira number here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744291089
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -62,6 +63,9 @@ class HivePartitionFilteringSuite(version: String)
properties = Map.empty
)
+ // Avoid repeatedly constructing multiple hive instances that do not use direct sql
+ private var disableDirectSqlClient: HiveClient = _
Review comment:
hmm can you explain why this is useful?
looks like by caching the client, we just need to create 1 instead of 3 clients now, for each separate run on a Hive version (2 initialization in the existing code + 1 initialization in the new test case `getPartitionsByFilter: ds=20170101 when $fallbackKey=true`, is that correct?
I'm also curious why the OOM didn't happen before, as we were already using the `init(false)` in the tests.
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -577,6 +582,60 @@ class HivePartitionFilteringSuite(version: String)
dateStrValue)
}
+ test("getPartitionsByFilter: substr(chunk,0,1)=a") {
Review comment:
nit: could we prepend JIRA number to these new tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958712541
**[Test build #144872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144872/testReport)** for PR 34431 at commit [`4e7a37d`](https://github.com/apache/spark/commit/4e7a37d349a233dcf0b1def227a37eee1b48e594).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958991175
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144872/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958782245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954482761
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954487200
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49208/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956244816
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144807/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954904825
**[Test build #144755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144755/testReport)** for PR 34431 at commit [`6916668`](https://github.com/apache/spark/commit/6916668aab2801e278a19eedc661f822dff75e34).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954459679
**[Test build #144739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144739/testReport)** for PR 34431 at commit [`d1adf3c`](https://github.com/apache/spark/commit/d1adf3c647ee1c8a485bff4aaa7c13e363aa8934).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954543195
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49208/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962654854
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144965/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744295770
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -577,6 +582,60 @@ class HivePartitionFilteringSuite(version: String)
dateStrValue)
}
+ test("getPartitionsByFilter: substr(chunk,0,1)=a") {
Review comment:
hm maybe you are right - sorry for the confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962607378
**[Test build #144965 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144965/testReport)** for PR 34431 at commit [`11479df`](https://github.com/apache/spark/commit/11479dfff68d9ea92b965570acc075e8b41e0218).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962697694
**[Test build #144967 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144967/testReport)** for PR 34431 at commit [`df37268`](https://github.com/apache/spark/commit/df3726846397494a42bbc34d3e44f1a22cb91e0b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962654551
**[Test build #144965 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144965/testReport)** for PR 34431 at commit [`11479df`](https://github.com/apache/spark/commit/11479dfff68d9ea92b965570acc075e8b41e0218).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962656786
**[Test build #144968 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144968/testReport)** for PR 34431 at commit [`a5000ce`](https://github.com/apache/spark/commit/a5000ce30a53bdd74e3922530cf84a355fd02a80).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954781039
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49224/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954439651
**[Test build #144734 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144734/testReport)** for PR 34431 at commit [`67b97e0`](https://github.com/apache/spark/commit/67b97e0d5190d1f6db1f6e3dfb39482303c07d1b).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958782245
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49342/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956042518
**[Test build #144807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144807/testReport)** for PR 34431 at commit [`56ce42b`](https://github.com/apache/spark/commit/56ce42b16203135e681c0161b8372662bfdf10e4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961763376
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144917/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956117693
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958782245
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49342/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958772608
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49342/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958991175
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144872/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961624221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] advancedxy commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
advancedxy commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744132868
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
##########
@@ -1106,26 +1109,84 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
// enabled.
recordHiveCall()
getPartitionsByFilterMethod.invoke(hive, table, filter)
- .asInstanceOf[JArrayList[Partition]]
+ .asInstanceOf[JArrayList[Partition]].asScala.toSeq
} catch {
case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] &&
shouldFallback =>
logWarning("Caught Hive MetaException attempting to get partition metadata by " +
"filter from Hive. Falling back to fetching all partition metadata, which will " +
"degrade performance. Modifying your Hive metastore configuration to set " +
s"${tryDirectSqlConfVar.varname} to true (if it is not true already) may resolve " +
- "this problem. Otherwise, to avoid degraded performance you can set " +
+ "this problem. Or you can enable " +
+ s"${SQLConf.HIVE_METASTORE_PARTITION_PRUNING_FAST_FALLBACK.key} " +
+ "to alleviate performance downgrade. " +
+ "Otherwise, to avoid degraded performance you can set " +
s"${SQLConf.HIVE_METASTORE_PARTITION_PRUNING_FALLBACK_ON_EXCEPTION.key} " +
" to false and let the query fail instead.", ex)
// HiveShim clients are expected to handle a superset of the requested partitions
- recordHiveCall()
- getAllPartitionsMethod.invoke(hive, table).asInstanceOf[JSet[Partition]]
+ prunePartitionsFastFallback(hive, table, catalogTable, predicates)
case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] =>
throw QueryExecutionErrors.getPartitionMetadataByFilterError(ex)
}
}
- partitions.asScala.toSeq
+ partitions
+ }
+
+ private def prunePartitionsFastFallback(
+ hive: Hive,
+ table: Table,
+ catalogTable: CatalogTable,
+ predicates: Seq[Expression]): Seq[Partition] = {
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+
+ // Because there is no way to know whether the partition properties has timeZone,
+ // client-side filtering cannot be used with TimeZoneAwareExpression.
+ def hasTimeZoneAwareExpression(e: Expression): Boolean = {
+ e.collectFirst {
+ case t: TimeZoneAwareExpression => t
+ }.isDefined
+ }
+
+ if (!SQLConf.get.metastorePartitionPruningFastFallback ||
+ predicates.isEmpty ||
+ predicates.exists(hasTimeZoneAwareExpression)) {
+ recordHiveCall()
+ getAllPartitionsMethod.invoke(hive, table).asInstanceOf[JSet[Partition]].asScala.toSeq
+ } else {
+ try {
+ val partitionSchema = CharVarcharUtils.replaceCharVarcharWithStringInSchema(
+ catalogTable.partitionSchema)
+ val boundPredicate = ExternalCatalogUtils.generatePartitionPredicateByFilter(
+ catalogTable, partitionSchema, predicates)
+
+ def toRow(spec: TablePartitionSpec): InternalRow = {
+ InternalRow.fromSeq(partitionSchema.map { field =>
+ val partValue = if (spec(field.name) == ExternalCatalogUtils.DEFAULT_PARTITION_NAME) {
+ null
+ } else {
+ spec(field.name)
+ }
+ Cast(Literal(partValue), field.dataType, Option(timeZoneId)).eval()
+ })
+ }
+
+ val allPartitionNames = hive.getPartitionNames(
+ table.getDbName, table.getTableName, -1).asScala
+ val partNames = allPartitionNames.filter { p =>
+ val spec = PartitioningUtils.parsePathFragment(p)
+ boundPredicate.eval(toRow(spec))
+ }
+ recordHiveCall()
+ hive.getPartitionsByNames(table, partNames.asJava).asScala.toSeq
+ } catch {
+ case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] =>
+ logWarning("Caught Hive MetaException attempting to get partition metadata by " +
+ "filter from client side. Falling back to fetching all partition metadata", ex)
+ recordHiveCall()
+ getAllPartitionsMethod.invoke(hive, table).asInstanceOf[JSet[Partition]].asScala.toSeq
Review comment:
NIt: there are multiple `asScala.toSeq` call in this method. How about we just return `JSet[Partition]` for this method, and return `partitions.asScala.toSeq` in `getPartitionsByFilter` as before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744294803
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -62,6 +63,9 @@ class HivePartitionFilteringSuite(version: String)
properties = Map.empty
)
+ // Avoid repeatedly constructing multiple hive instances that do not use direct sql
+ private var disableDirectSqlClient: HiveClient = _
Review comment:
Got it, thanks. One nit: could you rename `disableDirectSqlClient` to something like `clientWithoutDirectSql`? the former looks like a boolean flag :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962624517
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962668230
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49441/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao closed pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
sunchao closed pull request #34431:
URL: https://github.com/apache/spark/pull/34431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954660064
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144739/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954514864
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49208/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-955157746
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956042518
**[Test build #144807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144807/testReport)** for PR 34431 at commit [`56ce42b`](https://github.com/apache/spark/commit/56ce42b16203135e681c0161b8372662bfdf10e4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956244816
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144807/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956033460
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144796/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957241570
I have triggered multiple tests these days and it seems to be working properly.
Can we review this again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954434153
cc @cloud-fan, @juliuszsompolski FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958782245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957076039
**[Test build #144831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144831/testReport)** for PR 34431 at commit [`fa69af0`](https://github.com/apache/spark/commit/fa69af00185773d3e73b62df4bd1c1a1c2ddb198).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957117831
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-955898167
**[Test build #144796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144796/testReport)** for PR 34431 at commit [`822edda`](https://github.com/apache/spark/commit/822eddaa7a6a8b145651aa9c15155be596f668f7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961624221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961624221
**[Test build #144917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144917/testReport)** for PR 34431 at commit [`e3038f4`](https://github.com/apache/spark/commit/e3038f43ac28fa62237a89a0a59949203cec0064).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961653020
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961661217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957241570
I have triggered multiple tests these days and it seems to be working properly.
Can we review this again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744294288
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -62,6 +63,9 @@ class HivePartitionFilteringSuite(version: String)
properties = Map.empty
)
+ // Avoid repeatedly constructing multiple hive instances that do not use direct sql
+ private var disableDirectSqlClient: HiveClient = _
Review comment:
Yes, if I create a client multiple times, it will be OOM due to the limitation of MaxMetaspaceSize=1g.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962696844
**[Test build #144968 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144968/testReport)** for PR 34431 at commit [`a5000ce`](https://github.com/apache/spark/commit/a5000ce30a53bdd74e3922530cf84a355fd02a80).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962697242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954459679
**[Test build #144739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144739/testReport)** for PR 34431 at commit [`d1adf3c`](https://github.com/apache/spark/commit/d1adf3c647ee1c8a485bff4aaa7c13e363aa8934).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954910802
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144755/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954434062
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954781039
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49224/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-955898167
**[Test build #144796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144796/testReport)** for PR 34431 at commit [`822edda`](https://github.com/apache/spark/commit/822eddaa7a6a8b145651aa9c15155be596f668f7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-955157746
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957117831
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957233972
**[Test build #144831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144831/testReport)** for PR 34431 at commit [`fa69af0`](https://github.com/apache/spark/commit/fa69af00185773d3e73b62df4bd1c1a1c2ddb198).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-955944655
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49266/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956033460
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144796/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962621501
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962607378
**[Test build #144965 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144965/testReport)** for PR 34431 at commit [`11479df`](https://github.com/apache/spark/commit/11479dfff68d9ea92b965570acc075e8b41e0218).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958712541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958712541
**[Test build #144872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144872/testReport)** for PR 34431 at commit [`4e7a37d`](https://github.com/apache/spark/commit/4e7a37d349a233dcf0b1def227a37eee1b48e594).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958982714
**[Test build #144872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144872/testReport)** for PR 34431 at commit [`4e7a37d`](https://github.com/apache/spark/commit/4e7a37d349a233dcf0b1def227a37eee1b48e594).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958734713
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49342/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954728663
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49224/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-955944655
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49266/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956030888
**[Test build #144796 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144796/testReport)** for PR 34431 at commit [`822edda`](https://github.com/apache/spark/commit/822eddaa7a6a8b145651aa9c15155be596f668f7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957264005
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144831/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956117693
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957264005
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144831/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957076039
**[Test build #144831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144831/testReport)** for PR 34431 at commit [`fa69af0`](https://github.com/apache/spark/commit/fa69af00185773d3e73b62df4bd1c1a1c2ddb198).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958782245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957076039
**[Test build #144831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144831/testReport)** for PR 34431 at commit [`fa69af0`](https://github.com/apache/spark/commit/fa69af00185773d3e73b62df4bd1c1a1c2ddb198).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957117831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744273834
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
##########
@@ -1106,26 +1109,84 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
// enabled.
recordHiveCall()
getPartitionsByFilterMethod.invoke(hive, table, filter)
- .asInstanceOf[JArrayList[Partition]]
+ .asInstanceOf[JArrayList[Partition]].asScala.toSeq
} catch {
case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] &&
shouldFallback =>
logWarning("Caught Hive MetaException attempting to get partition metadata by " +
"filter from Hive. Falling back to fetching all partition metadata, which will " +
"degrade performance. Modifying your Hive metastore configuration to set " +
s"${tryDirectSqlConfVar.varname} to true (if it is not true already) may resolve " +
- "this problem. Otherwise, to avoid degraded performance you can set " +
+ "this problem. Or you can enable " +
+ s"${SQLConf.HIVE_METASTORE_PARTITION_PRUNING_FAST_FALLBACK.key} " +
+ "to alleviate performance downgrade. " +
+ "Otherwise, to avoid degraded performance you can set " +
s"${SQLConf.HIVE_METASTORE_PARTITION_PRUNING_FALLBACK_ON_EXCEPTION.key} " +
" to false and let the query fail instead.", ex)
// HiveShim clients are expected to handle a superset of the requested partitions
- recordHiveCall()
- getAllPartitionsMethod.invoke(hive, table).asInstanceOf[JSet[Partition]]
+ prunePartitionsFastFallback(hive, table, catalogTable, predicates)
case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] =>
throw QueryExecutionErrors.getPartitionMetadataByFilterError(ex)
}
}
- partitions.asScala.toSeq
+ partitions
+ }
+
+ private def prunePartitionsFastFallback(
+ hive: Hive,
+ table: Table,
+ catalogTable: CatalogTable,
+ predicates: Seq[Expression]): Seq[Partition] = {
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+
+ // Because there is no way to know whether the partition properties has timeZone,
+ // client-side filtering cannot be used with TimeZoneAwareExpression.
+ def hasTimeZoneAwareExpression(e: Expression): Boolean = {
+ e.collectFirst {
+ case t: TimeZoneAwareExpression => t
+ }.isDefined
+ }
+
+ if (!SQLConf.get.metastorePartitionPruningFastFallback ||
+ predicates.isEmpty ||
+ predicates.exists(hasTimeZoneAwareExpression)) {
+ recordHiveCall()
+ getAllPartitionsMethod.invoke(hive, table).asInstanceOf[JSet[Partition]].asScala.toSeq
+ } else {
+ try {
+ val partitionSchema = CharVarcharUtils.replaceCharVarcharWithStringInSchema(
+ catalogTable.partitionSchema)
+ val boundPredicate = ExternalCatalogUtils.generatePartitionPredicateByFilter(
+ catalogTable, partitionSchema, predicates)
+
+ def toRow(spec: TablePartitionSpec): InternalRow = {
+ InternalRow.fromSeq(partitionSchema.map { field =>
+ val partValue = if (spec(field.name) == ExternalCatalogUtils.DEFAULT_PARTITION_NAME) {
+ null
+ } else {
+ spec(field.name)
+ }
+ Cast(Literal(partValue), field.dataType, Option(timeZoneId)).eval()
+ })
+ }
+
+ val allPartitionNames = hive.getPartitionNames(
+ table.getDbName, table.getTableName, -1).asScala
+ val partNames = allPartitionNames.filter { p =>
+ val spec = PartitioningUtils.parsePathFragment(p)
+ boundPredicate.eval(toRow(spec))
+ }
+ recordHiveCall()
+ hive.getPartitionsByNames(table, partNames.asJava).asScala.toSeq
+ } catch {
+ case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] =>
+ logWarning("Caught Hive MetaException attempting to get partition metadata by " +
+ "filter from client side. Falling back to fetching all partition metadata", ex)
+ recordHiveCall()
+ getAllPartitionsMethod.invoke(hive, table).asInstanceOf[JSet[Partition]].asScala.toSeq
Review comment:
Thank you for your suggestion, I modified the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962698159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144967/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961761702
**[Test build #144917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144917/testReport)** for PR 34431 at commit [`e3038f4`](https://github.com/apache/spark/commit/e3038f43ac28fa62237a89a0a59949203cec0064).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744295110
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1015,6 +1015,17 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val HIVE_METASTORE_PARTITION_PRUNING_FAST_FALLBACK =
+ buildConf("spark.sql.hive.metastorePartitionPruningFastFallback")
Review comment:
please also update PR description for this new config, in the user-facing change section, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744293837
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -62,6 +63,9 @@ class HivePartitionFilteringSuite(version: String)
properties = Map.empty
)
+ // Avoid repeatedly constructing multiple hive instances that do not use direct sql
+ private var disableDirectSqlClient: HiveClient = _
Review comment:
so you mean the test was fine before when we creates 2 clients with each Hive version, but start to failing when creating 3 clients?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962654854
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144965/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954454330
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954437652
**[Test build #144734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144734/testReport)** for PR 34431 at commit [`67b97e0`](https://github.com/apache/spark/commit/67b97e0d5190d1f6db1f6e3dfb39482303c07d1b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954397213
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954543195
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49208/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-955938267
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49266/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956073901
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-955913061
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49266/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961638996
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961661217
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961624221
**[Test build #144917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144917/testReport)** for PR 34431 at commit [`e3038f4`](https://github.com/apache/spark/commit/e3038f43ac28fa62237a89a0a59949203cec0064).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954652918
**[Test build #144739 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144739/testReport)** for PR 34431 at commit [`d1adf3c`](https://github.com/apache/spark/commit/d1adf3c647ee1c8a485bff4aaa7c13e363aa8934).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954910802
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144755/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957117831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958782245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958712541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962662181
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49438/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962756736
Merged to master, thanks @cxzl25 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961661217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956117671
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954760600
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49224/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957117818
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954695534
**[Test build #144755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144755/testReport)** for PR 34431 at commit [`6916668`](https://github.com/apache/spark/commit/6916668aab2801e278a19eedc661f822dff75e34).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954437652
**[Test build #144734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144734/testReport)** for PR 34431 at commit [`67b97e0`](https://github.com/apache/spark/commit/67b97e0d5190d1f6db1f6e3dfb39482303c07d1b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954488738
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957076039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957241570
I have triggered multiple tests these days and it seems to be working properly.
Can we review this again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744293312
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -62,6 +63,9 @@ class HivePartitionFilteringSuite(version: String)
properties = Map.empty
)
+ // Avoid repeatedly constructing multiple hive instances that do not use direct sql
+ private var disableDirectSqlClient: HiveClient = _
Review comment:
First, the size of `METASPACE_SIZE` is 1g
Then `HiveClientVersions` needs to test "0.14", "1.0", "1.1", "1.2", "2.0", "2.1", "2.2", "2.3", "3.0", "3.1", 11 hive versions.
Each version is initialized 3 times with `IsolatedClientLoader`, and hive-related classes cannot be unloaded after ut ends.
If we create hive client several times (val client = init(false)), it will cause `OutOfMemoryError: Metaspace`
For example, this test
https://github.com/cxzl25/spark/commit/888ac1ef068ed88f4802079d8120cac11b46999c
https://github.com/cxzl25/spark/runs/4023095713
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744295213
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -577,6 +582,60 @@ class HivePartitionFilteringSuite(version: String)
dateStrValue)
}
+ test("SPARK-35437 getPartitionsByFilter: substr(chunk,0,1)=a") {
Review comment:
nit nit: it should be "SPARK-35437: getPartitionsByFilter" :)
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -577,6 +582,60 @@ class HivePartitionFilteringSuite(version: String)
dateStrValue)
}
+ test("SPARK-35437 getPartitionsByFilter: substr(chunk,0,1)=a") {
Review comment:
nit nit: it should be `SPARK-35437: getPartitionsByFilter` :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962656194
**[Test build #144967 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144967/testReport)** for PR 34431 at commit [`df37268`](https://github.com/apache/spark/commit/df3726846397494a42bbc34d3e44f1a22cb91e0b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744295770
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -577,6 +582,60 @@ class HivePartitionFilteringSuite(version: String)
dateStrValue)
}
+ test("getPartitionsByFilter: substr(chunk,0,1)=a") {
Review comment:
hm maybe you are right - sorry for the confusion. feel free to remove the JIRA number if you prefer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962670376
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49438/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962670376
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49438/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962697242
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144968/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958712541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961624221
**[Test build #144917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144917/testReport)** for PR 34431 at commit [`e3038f4`](https://github.com/apache/spark/commit/e3038f43ac28fa62237a89a0a59949203cec0064).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962613870
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961661217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954397213
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954660064
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144739/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
juliuszsompolski commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954609966
cc @databricks-david-lewis FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-954439666
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144734/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957076039
**[Test build #144831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144831/testReport)** for PR 34431 at commit [`fa69af0`](https://github.com/apache/spark/commit/fa69af00185773d3e73b62df4bd1c1a1c2ddb198).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-956243273
**[Test build #144807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144807/testReport)** for PR 34431 at commit [`56ce42b`](https://github.com/apache/spark/commit/56ce42b16203135e681c0161b8372662bfdf10e4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-957076039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-961624221
**[Test build #144917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144917/testReport)** for PR 34431 at commit [`e3038f4`](https://github.com/apache/spark/commit/e3038f43ac28fa62237a89a0a59949203cec0064).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958782245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958782245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-958712541
**[Test build #144872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144872/testReport)** for PR 34431 at commit [`4e7a37d`](https://github.com/apache/spark/commit/4e7a37d349a233dcf0b1def227a37eee1b48e594).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962624517
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744293312
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala
##########
@@ -62,6 +63,9 @@ class HivePartitionFilteringSuite(version: String)
properties = Map.empty
)
+ // Avoid repeatedly constructing multiple hive instances that do not use direct sql
+ private var disableDirectSqlClient: HiveClient = _
Review comment:
First, the size of `METASPACE_SIZE` is 1g.
Then `HiveClientVersions` needs to test "0.14", "1.0", "1.1", "1.2", "2.0", "2.1", "2.2", "2.3", "3.0", "3.1", 11 hive versions.
Each version is initialized 3 times with `IsolatedClientLoader`, and hive-related classes cannot be unloaded after ut ends.
If we create hive client several times (val client = init(false)), it will cause `OutOfMemoryError: Metaspace`
For example, this test
https://github.com/cxzl25/spark/commit/888ac1ef068ed88f4802079d8120cac11b46999c
https://github.com/cxzl25/spark/runs/4023095713
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962667997
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49438/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on a change in pull request #34431:
URL: https://github.com/apache/spark/pull/34431#discussion_r744273834
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
##########
@@ -1106,26 +1109,84 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
// enabled.
recordHiveCall()
getPartitionsByFilterMethod.invoke(hive, table, filter)
- .asInstanceOf[JArrayList[Partition]]
+ .asInstanceOf[JArrayList[Partition]].asScala.toSeq
} catch {
case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] &&
shouldFallback =>
logWarning("Caught Hive MetaException attempting to get partition metadata by " +
"filter from Hive. Falling back to fetching all partition metadata, which will " +
"degrade performance. Modifying your Hive metastore configuration to set " +
s"${tryDirectSqlConfVar.varname} to true (if it is not true already) may resolve " +
- "this problem. Otherwise, to avoid degraded performance you can set " +
+ "this problem. Or you can enable " +
+ s"${SQLConf.HIVE_METASTORE_PARTITION_PRUNING_FAST_FALLBACK.key} " +
+ "to alleviate performance downgrade. " +
+ "Otherwise, to avoid degraded performance you can set " +
s"${SQLConf.HIVE_METASTORE_PARTITION_PRUNING_FALLBACK_ON_EXCEPTION.key} " +
" to false and let the query fail instead.", ex)
// HiveShim clients are expected to handle a superset of the requested partitions
- recordHiveCall()
- getAllPartitionsMethod.invoke(hive, table).asInstanceOf[JSet[Partition]]
+ prunePartitionsFastFallback(hive, table, catalogTable, predicates)
case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] =>
throw QueryExecutionErrors.getPartitionMetadataByFilterError(ex)
}
}
- partitions.asScala.toSeq
+ partitions
+ }
+
+ private def prunePartitionsFastFallback(
+ hive: Hive,
+ table: Table,
+ catalogTable: CatalogTable,
+ predicates: Seq[Expression]): Seq[Partition] = {
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+
+ // Because there is no way to know whether the partition properties has timeZone,
+ // client-side filtering cannot be used with TimeZoneAwareExpression.
+ def hasTimeZoneAwareExpression(e: Expression): Boolean = {
+ e.collectFirst {
+ case t: TimeZoneAwareExpression => t
+ }.isDefined
+ }
+
+ if (!SQLConf.get.metastorePartitionPruningFastFallback ||
+ predicates.isEmpty ||
+ predicates.exists(hasTimeZoneAwareExpression)) {
+ recordHiveCall()
+ getAllPartitionsMethod.invoke(hive, table).asInstanceOf[JSet[Partition]].asScala.toSeq
+ } else {
+ try {
+ val partitionSchema = CharVarcharUtils.replaceCharVarcharWithStringInSchema(
+ catalogTable.partitionSchema)
+ val boundPredicate = ExternalCatalogUtils.generatePartitionPredicateByFilter(
+ catalogTable, partitionSchema, predicates)
+
+ def toRow(spec: TablePartitionSpec): InternalRow = {
+ InternalRow.fromSeq(partitionSchema.map { field =>
+ val partValue = if (spec(field.name) == ExternalCatalogUtils.DEFAULT_PARTITION_NAME) {
+ null
+ } else {
+ spec(field.name)
+ }
+ Cast(Literal(partValue), field.dataType, Option(timeZoneId)).eval()
+ })
+ }
+
+ val allPartitionNames = hive.getPartitionNames(
+ table.getDbName, table.getTableName, -1).asScala
+ val partNames = allPartitionNames.filter { p =>
+ val spec = PartitioningUtils.parsePathFragment(p)
+ boundPredicate.eval(toRow(spec))
+ }
+ recordHiveCall()
+ hive.getPartitionsByNames(table, partNames.asJava).asScala.toSeq
+ } catch {
+ case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] =>
+ logWarning("Caught Hive MetaException attempting to get partition metadata by " +
+ "filter from client side. Falling back to fetching all partition metadata", ex)
+ recordHiveCall()
+ getAllPartitionsMethod.invoke(hive, table).asInstanceOf[JSet[Partition]].asScala.toSeq
Review comment:
Thank you for your suggestion, I modified the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962676569
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49441/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962678791
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49441/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962678791
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49441/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962656786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34431: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34431:
URL: https://github.com/apache/spark/pull/34431#issuecomment-962500617
Thanks @cxzl25 , will take a look again soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org