You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/31 02:52:41 UTC
[GitHub] [spark] ulysses-you opened a new pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
ulysses-you opened a new pull request #32009:
URL: https://github.com/apache/spark/pull/32009
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Add updateDelegationTokens in `LocalSchedulerBackend`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Make local mode support update tokens. Include hive token if package with hive.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, local mode support update token.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
currently, test manual with
```
./bin/spark-shell --master local --proxy-user user_name
> spark.sql("show tables").show
```
before
```
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:477)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
```
after
```
+---------+--------------------+-----------+
|namespace| tableName|isTemporary|
+---------+--------------------+-----------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811616050
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41371/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-821117630
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137477/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810769301
**[Test build #136746 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136746/testReport)** for PR 32009 at commit [`369e08b`](https://github.com/apache/spark/commit/369e08b2e39b09868238db00b14db4a0eb526ddc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-821003649
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-821117630
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137477/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811161700
**[Test build #136767 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136767/testReport)** for PR 32009 at commit [`f528cda`](https://github.com/apache/spark/commit/f528cda3b3ea8217e80857f60eb07cc4f3922bb5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810739994
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41328/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-828135393
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138016/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-816472048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-816580929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137117/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811046832
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41350/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-816564720
**[Test build #137117 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137117/testReport)** for PR 32009 at commit [`8fcb62f`](https://github.com/apache/spark/commit/8fcb62fd33d07ee4cb37bc9b85118c5c4ced3b01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810780681
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136746/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810767033
thank you for taking a look @HyukjinKwon @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-827662120
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137999/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-821056641
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42053/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811577317
thank you @tgravescs
yeah, the basic requirement is proxy. As the admin of cluster, we don't have user keytab but we need to debug and re-produce some production envirmenont problem that required use proxy to do some option. And you kown, it's not possible to get all keytab from user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-828135393
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138016/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-827570316
**[Test build #137999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137999/testReport)** for PR 32009 at commit [`084cb55`](https://github.com/apache/spark/commit/084cb5522089921f2feebe1383f81081dd0ac745).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811616050
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41371/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-816441638
**[Test build #137117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137117/testReport)** for PR 32009 at commit [`8fcb62f`](https://github.com/apache/spark/commit/8fcb62fd33d07ee4cb37bc9b85118c5c4ced3b01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-816472098
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41696/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811591207
> NOte that if we decide to support this we should document exactly what is and what is not supported: http://spark.apache.org/docs/latest/security.html#kerberos
thanks for the reminder, removed the word `non-local`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810780681
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136746/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-821022733
**[Test build #137477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137477/testReport)** for PR 32009 at commit [`8fcb62f`](https://github.com/apache/spark/commit/8fcb62fd33d07ee4cb37bc9b85118c5c4ced3b01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-827600137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-821050251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-827570316
**[Test build #137999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137999/testReport)** for PR 32009 at commit [`084cb55`](https://github.com/apache/spark/commit/084cb5522089921f2feebe1383f81081dd0ac745).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811046832
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41350/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811000303
**[Test build #136767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136767/testReport)** for PR 32009 at commit [`f528cda`](https://github.com/apache/spark/commit/f528cda3b3ea8217e80857f60eb07cc4f3922bb5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811658544
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136788/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811612908
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41371/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-827662120
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137999/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-897245185
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-828094769
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42535/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811598016
**[Test build #136788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136788/testReport)** for PR 32009 at commit [`8fcb62f`](https://github.com/apache/spark/commit/8fcb62fd33d07ee4cb37bc9b85118c5c4ced3b01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-828093109
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-827606683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42519/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810739994
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41328/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-828072919
**[Test build #138016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138016/testReport)** for PR 32009 at commit [`084cb55`](https://github.com/apache/spark/commit/084cb5522089921f2feebe1383f81081dd0ac745).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-827606683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42519/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811611447
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41371/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-827660439
**[Test build #137999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137999/testReport)** for PR 32009 at commit [`084cb55`](https://github.com/apache/spark/commit/084cb5522089921f2feebe1383f81081dd0ac745).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810719159
**[Test build #136746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136746/testReport)** for PR 32009 at commit [`369e08b`](https://github.com/apache/spark/commit/369e08b2e39b09868238db00b14db4a0eb526ddc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-828072919
**[Test build #138016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138016/testReport)** for PR 32009 at commit [`084cb55`](https://github.com/apache/spark/commit/084cb5522089921f2feebe1383f81081dd0ac745).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810999164
thank you @gaborgsomogyi , the suggestions are useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811046779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811658544
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136788/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810739984
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41328/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811598016
**[Test build #136788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136788/testReport)** for PR 32009 at commit [`8fcb62f`](https://github.com/apache/spark/commit/8fcb62fd33d07ee4cb37bc9b85118c5c4ced3b01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811099824
so can you give more details on what exactly you want to support here? local mode is meant as basically test and development mode, if you have something that needs token updates that kind of implies its a long running production thing.
is this just to support proxy users or are you actually trying to support a long lived process where tokens need updating?
If proxy user, I guess I would also ask why do you need a proxy user in local mode just becomes that user and run it. Perhaps you don't have access to that user key tab and its configured for proxy user only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811000303
**[Test build #136767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136767/testReport)** for PR 32009 at commit [`f528cda`](https://github.com/apache/spark/commit/f528cda3b3ea8217e80857f60eb07cc4f3922bb5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810734716
cc @tgravescs, @mridulm, @jerryshao, @gaborgsomogyi FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #32009:
URL: https://github.com/apache/spark/pull/32009
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-816580929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137117/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32009:
URL: https://github.com/apache/spark/pull/32009#discussion_r604826491
##########
File path: core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
##########
@@ -76,6 +81,9 @@ private[spark] class LocalEndpoint(
case KillTask(taskId, interruptThread, reason) =>
executor.killTask(taskId, interruptThread, reason)
+
+ case UpdateDelegationTokens(tokens) =>
Review comment:
I thought about this, and seems it's the most easily approach to leave it here, since the `HadoopDelegationTokenManager ` would renew token through endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-816441638
**[Test build #137117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137117/testReport)** for PR 32009 at commit [`8fcb62f`](https://github.com/apache/spark/commit/8fcb62fd33d07ee4cb37bc9b85118c5c4ced3b01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811101550
NOte that if we decide to support this we should document exactly what is and what is not supported: http://spark.apache.org/docs/latest/security.html#kerberos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-821056641
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42053/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-821022733
**[Test build #137477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137477/testReport)** for PR 32009 at commit [`8fcb62f`](https://github.com/apache/spark/commit/8fcb62fd33d07ee4cb37bc9b85118c5c4ced3b01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #32009:
URL: https://github.com/apache/spark/pull/32009#discussion_r604842570
##########
File path: core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
##########
@@ -178,4 +188,23 @@ private[spark] class LocalSchedulerBackend(
}
}
+ private def updateDelegationTokens(): Unit = {
+ if (UserGroupInformation.isSecurityEnabled()) {
+ val delegationTokenManager =
Review comment:
Since we added as `var` now it works, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-828094769
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42535/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811167195
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136767/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #32009:
URL: https://github.com/apache/spark/pull/32009#discussion_r604772518
##########
File path: core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
##########
@@ -178,4 +188,23 @@ private[spark] class LocalSchedulerBackend(
}
}
+ private def updateDelegationTokens(): Unit = {
+ if (UserGroupInformation.isSecurityEnabled()) {
+ val delegationTokenManager =
+ new HadoopDelegationTokenManager(conf, scheduler.sc.hadoopConfiguration, localEndpoint)
+ val ugi = UserGroupInformation.getCurrentUser()
+ val tokens = if (delegationTokenManager.renewalEnabled) {
+ delegationTokenManager.start()
Review comment:
`delegationTokenManager` is never stopped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #32009:
URL: https://github.com/apache/spark/pull/32009#discussion_r604759649
##########
File path: core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
##########
@@ -178,4 +188,23 @@ private[spark] class LocalSchedulerBackend(
}
}
+ private def updateDelegationTokens(): Unit = {
+ if (UserGroupInformation.isSecurityEnabled()) {
+ val delegationTokenManager =
+ new HadoopDelegationTokenManager(conf, scheduler.sc.hadoopConfiguration, localEndpoint)
+ val ugi = UserGroupInformation.getCurrentUser()
+ val tokens = if (delegationTokenManager.renewalEnabled) {
+ delegationTokenManager.start()
+ } else {
+ val creds = ugi.getCredentials()
+ delegationTokenManager.obtainDelegationTokens(creds)
+ if (creds.numberOfTokens() > 0 || creds.numberOfSecretKeys() > 0) {
+ SparkHadoopUtil.get.serialize(creds)
+ } else {
+ null
+ }
+ }
+ SparkHadoopUtil.get.addDelegationTokens(tokens, conf)
Review comment:
`tokens` can be null, right?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
##########
@@ -76,6 +81,9 @@ private[spark] class LocalEndpoint(
case KillTask(taskId, interruptThread, reason) =>
executor.killTask(taskId, interruptThread, reason)
+
+ case UpdateDelegationTokens(tokens) =>
Review comment:
Just thinking out loud. Because in local version of Spark the executor, backend, and master all run in the same JVM it's not necessary to send the tokens to the `executorEndpoint` so it's fine as it.
##########
File path: core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
##########
@@ -178,4 +188,23 @@ private[spark] class LocalSchedulerBackend(
}
}
+ private def updateDelegationTokens(): Unit = {
+ if (UserGroupInformation.isSecurityEnabled()) {
+ val delegationTokenManager =
Review comment:
Since we don't store the reference of the `HadoopDelegationTokenManager` how does this renew the tokens? Or renewal is not considered?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
##########
@@ -76,6 +81,9 @@ private[spark] class LocalEndpoint(
case KillTask(taskId, interruptThread, reason) =>
executor.killTask(taskId, interruptThread, reason)
+
+ case UpdateDelegationTokens(tokens) =>
+ SparkHadoopUtil.get.addDelegationTokens(tokens, scheduler.conf)
Review comment:
It would be good to add some info like in `CoarseGrainedExecutorBackend` to increase debug possibilities:
```
logInfo(s"Received tokens of ${tokenBytes.length} bytes")
```
##########
File path: core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
##########
@@ -178,4 +188,23 @@ private[spark] class LocalSchedulerBackend(
}
}
+ private def updateDelegationTokens(): Unit = {
+ if (UserGroupInformation.isSecurityEnabled()) {
+ val delegationTokenManager =
+ new HadoopDelegationTokenManager(conf, scheduler.sc.hadoopConfiguration, localEndpoint)
+ val ugi = UserGroupInformation.getCurrentUser()
+ val tokens = if (delegationTokenManager.renewalEnabled) {
+ delegationTokenManager.start()
Review comment:
`delegationTokenManager` is never stopper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-821107114
**[Test build #137477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137477/testReport)** for PR 32009 at commit [`8fcb62f`](https://github.com/apache/spark/commit/8fcb62fd33d07ee4cb37bc9b85118c5c4ced3b01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-828128696
**[Test build #138016 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138016/testReport)** for PR 32009 at commit [`084cb55`](https://github.com/apache/spark/commit/084cb5522089921f2feebe1383f81081dd0ac745).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810737551
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41328/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-816440115
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-828064877
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-811641059
**[Test build #136788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136788/testReport)** for PR 32009 at commit [`8fcb62f`](https://github.com/apache/spark/commit/8fcb62fd33d07ee4cb37bc9b85118c5c4ced3b01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-816472098
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41696/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32009:
URL: https://github.com/apache/spark/pull/32009#discussion_r604824965
##########
File path: core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
##########
@@ -178,4 +188,23 @@ private[spark] class LocalSchedulerBackend(
}
}
+ private def updateDelegationTokens(): Unit = {
+ if (UserGroupInformation.isSecurityEnabled()) {
+ val delegationTokenManager =
Review comment:
The `renewal` would run at `HadoopDelegationTokenManager` if the conditions are matched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32009: [SPARK-34914][CORE] Local scheduler backend support update token
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32009:
URL: https://github.com/apache/spark/pull/32009#issuecomment-810719159
**[Test build #136746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136746/testReport)** for PR 32009 at commit [`369e08b`](https://github.com/apache/spark/commit/369e08b2e39b09868238db00b14db4a0eb526ddc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org