You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/07 12:51:12 UTC
[GitHub] [spark] Shockang opened a new pull request #33674: [WIP][Spark-36328][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Shockang opened a new pull request #33674:
URL: https://github.com/apache/spark/pull/33674
### What changes were proposed in this pull request?
Add the credentials from previous JobConf into the new JobConf to reuse the FileSystem Delegation Token.
### Why are the changes needed?
Spark Job creates a new JobConf (which will have a new Credentials) for every hive table partition, the token is not reused and gets fetched for every partition. This is slowing down the query as each delegation token has to go through KDC and SSL handshake on Secure Clusters.
### Does this PR introduce _any_ user-facing change?
Yes, while user querying partitioned hive table.
### How was this patch tested?
new test added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904711470
**[Test build #142733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142733/testReport)** for PR 33674 at commit [`01f8eee`](https://github.com/apache/spark/commit/01f8eee3012dbed0fc5ede653c7717cd7edb8080).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-894652060
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904747020
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47233/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn edited a comment on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
yaooqinn edited a comment on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-905152050
IIUC, the executors recieve tokens from the driver periodically, and it has nothing to do with an actual job.
> delegation token has to go through KDC
IIUC, this is not true either. Clients need to get TGT from KDC via keytabs but do not need to negotiate with KDC every time as long as they start use tokens
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904848047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142733/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904807111
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47233/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904711470
**[Test build #142733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142733/testReport)** for PR 33674 at commit [`01f8eee`](https://github.com/apache/spark/commit/01f8eee3012dbed0fc5ede653c7717cd7edb8080).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-1073134804
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Shockang commented on a change in pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
Shockang commented on a change in pull request #33674:
URL: https://github.com/apache/spark/pull/33674#discussion_r698021092
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
##########
@@ -378,6 +387,9 @@ class HadoopTableReader(
private def createNewHadoopRDD(partDesc: PartitionDesc, path: String): RDD[Writable] = {
val newJobConf = new JobConf(hadoopConf)
HadoopTableReader.initializeLocalJobConfFunc(path, partDesc.getTableDesc)(newJobConf)
+ // SPARK-36328: Reuse the FileSystem delegation token while querying partitioned hive table.
+ SparkHadoopUtil.get.addCurrentCredentials(newJobConf,
+ _broadcastedCredentials.value.value)
Review comment:
I'll delete it. I added it because considering that the new Hadoop API will make it easier to reuse tokens, but it violates the principle of minimizing modification points. @c21
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-907814821
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-907807771
**[Test build #142849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142849/testReport)** for PR 33674 at commit [`1a8e29f`](https://github.com/apache/spark/commit/1a8e29f82c5284440c6a955ce82f56d9ffdaa322).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904807111
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47233/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #33674:
URL: https://github.com/apache/spark/pull/33674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-907814490
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Shockang removed a comment on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
Shockang removed a comment on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-894800937
cc @cloud-fan Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] PrabhuJoseph commented on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
PrabhuJoseph commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-910519315
@sunchao Have shared the repro code. The Driver is issuing multiple get Delegation Token requests one for each partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904848047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142733/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904841008
**[Test build #142733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142733/testReport)** for PR 33674 at commit [`01f8eee`](https://github.com/apache/spark/commit/01f8eee3012dbed0fc5ede653c7717cd7edb8080).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-907839699
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-907836231
**[Test build #142849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142849/testReport)** for PR 33674 at commit [`1a8e29f`](https://github.com/apache/spark/commit/1a8e29f82c5284440c6a955ce82f56d9ffdaa322).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-908633394
I don' understand the issue this PR is trying to address either. As @yaooqinn mentioned, isn't delegation tokens (DT) distributed by drivers to executors and Spark will use DT to authenticate with the remote servers (e.g., HDFS NameNode) instead of going to KDC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-907807771
**[Test build #142849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142849/testReport)** for PR 33674 at commit [`1a8e29f`](https://github.com/apache/spark/commit/1a8e29f82c5284440c6a955ce82f56d9ffdaa322).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-907816923
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-907839699
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #33674:
URL: https://github.com/apache/spark/pull/33674#discussion_r695404049
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
##########
@@ -378,6 +387,9 @@ class HadoopTableReader(
private def createNewHadoopRDD(partDesc: PartitionDesc, path: String): RDD[Writable] = {
val newJobConf = new JobConf(hadoopConf)
HadoopTableReader.initializeLocalJobConfFunc(path, partDesc.getTableDesc)(newJobConf)
+ // SPARK-36328: Reuse the FileSystem delegation token while querying partitioned hive table.
+ SparkHadoopUtil.get.addCurrentCredentials(newJobConf,
+ _broadcastedCredentials.value.value)
Review comment:
wondering why we need this now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904695290
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904695603
cc @c21 @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-908633394
I don' understand the issue this PR is trying to address either. As @yaooqinn mentioned, isn't delegation tokens (DT) distributed by drivers to executors and Spark will use DT to authenticate with the remote servers (e.g., HDFS NameNode) instead of going to KDC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-905152050
IIUC, the executors get tokens from the driver periodically, and it has nothing to do with an actual job.
> delegation token has to go through KDC
IIUC, this is not true either. Clients need to get TGT from KDC via keytabs but do not need to negotiate with KDC every time as long as they start use tokens
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-907816923
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] PrabhuJoseph commented on pull request #33674: [Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
PrabhuJoseph commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-989729420
@Shockang Have tested the patch but it does not address the reported issue. The spark job runs in yarn-client mode where the Driver runs along with Client creates a new JobConf in HadoopRDD for every partition which internally fetches a FileSystem Delegation Token. So when there are 1000 partitions - 1000 time delegation token will be fetched.
The Spark Client gets the FileSystem Delegation Token at start (Client.scala - setupSecurityToken) and places in the Token file and pass it to the Spark Application Master & Executors to use. But the Client uses different credentials which does not have FileSystem Delegation Token as it is using TGT. (Refer SPARK-15754)
And so every call Driver (Client Mode) makes to list the path creates a separate JobConf and adds the Client Credentials which does not have FileSystem token and so obtains a new token.
One simple fix is to expose a config which adds obtained hadoop filesystem delegation token into the client user credentials if enabled. This will improve the performance by fetching delegation token only once when running query on a partitioned table.
Client.scala
private val hadoopConf = new YarnConfiguration(SparkHadoopUtil.newConfiguration(sparkConf))
private val isClusterMode = sparkConf.get("spark.submit.deployMode", "client") == "cluster"
+ private val useDelegationToken = sparkConf.getBoolean("spark.client.useDelegationToken", false)
// AM related configurations
private val amMemory = if (isClusterMode) {
// and adding delegation tokens could lead to expired or cancelled tokens being used
// later, as reported in SPARK-15754.
val currentUser = UserGroupInformation.getCurrentUser()
if (SparkHadoopUtil.get.isProxyUser(currentUser)) {
+ if (SparkHadoopUtil.get.isProxyUser(currentUser) || useDelegationToken) {
+ logInfo("Adding obtained Hadoop Delegation Tokens into User Credentials")
currentUser.addCredentials(credentials)
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-894652060
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Shockang commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
Shockang commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-894800937
cc @cloud-fan Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33674: [WIP][Spark-36328][CORE][SQL] Reuse the FileSystem delegation token while querying partitioned hive table.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33674:
URL: https://github.com/apache/spark/pull/33674#issuecomment-904791640
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47233/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org