You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yaooqinn <gi...@git.apache.org> on 2017/03/22 13:23:06 UTC
[GitHub] spark pull request #17387: [SPARK-20060][Deploy][Kerberos][Spark Shell] Obta...
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/17387
[SPARK-20060][Deploy][Kerberos][Spark Shell] Obtain credentials for proxy user before talking to hive metastore
## What changes were proposed in this pull request?
For **Spark on non-Yarn** mode on a kerberized hdfs, we don't obtain credentials from hive metastore, hdfs, etc and just use the local kinited user to connecting them. But if we specify the --proxy-user argument on non-yarn mode, such as local, standalone, after we simply use `UGI.createProxyUser` to get a proxy ugi as the effective user and wrap the code in doAs, the proxy ugi has no credentials to talk to hive metastore. Thus, we need to obtain credentials via the real user and add them to the proxy ugi.
## How was this patch tested?
unit tests
manual tests on kerberized metastore service and hdfs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yaooqinn/spark proxy-on-non-yarn
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17387.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17387
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos]Support Standalone visiti...
Posted by mgummelt <gi...@git.apache.org>.
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17387
@yaooqinn I'm trying to add Kerberos support for Mesos, and creating `HadoopRDD`s fail for me because YARN isn't configured: https://issues.apache.org/jira/browse/SPARK-20328
Did you run into this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos][Spark Shell] Obtain cred...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17387
@jerryshao This may also fix issue SPARK-19995 and SPARK-19997 on yarn with those apps which take `SparkSQLCLIDriver.main` as entrance, plz take a look.
@vanzin @tgravescs @mridulm @dongjoon-hyun can you help to review this, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos]Support Standalone visiti...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17387
@yaooqinn normally such a big behavior change requires design doc and well discussion. It is not a good idea to push bunch of codes silently without any discussion.
Besides, we're working on refactoring current credential mechanism to support push tokens from external service and get rid of distribution of keytab, please see [SPARK-19143](https://issues.apache.org/jira/browse/SPARK-19143). I think after that supporting kerberized services for standalone mode and others are much more easier and straightforward.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos]Support Standalone visiti...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17387
@yaooqinn I think the patch here is quite old and cannot be merged anymore, can you please close it.
If you still want to address this issue, can you please create a new PR, thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17387: [SPARK-20060][Deploy][Kerberos]Support Standalone...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn closed the pull request at:
https://github.com/apache/spark/pull/17387
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos]Support Standalone visiti...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/17387
@yaooqinn Shall we listen to @tgravescs and start with more design docs and SPIP?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos]Support Standalone visiti...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/17387
Yeah if you plan on adding support for secure hdfs access in standalone mode, it needs a feature jira, probably go through SPIP and make sure everything truly works and is documented. I remember someone in the past tried to just revert that jira that broke it but others wanted to make sure it was fully supported.
I also think there is a jira for adding kerberos support to the mesos side.
So I think we should start with more design in the jira and SPIP to make sure people agree with this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos][Spark Shell] Obtain cred...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17387
Does kerberos authentication really work in non-yarn cluster mode? AFAIK I don't see any code which will ship delegation tokens to executors other than yarn.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos][Spark Shell] Obtain cred...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/17387
I didn't think Spark officially supported kerberos in standalone mode. I'm pretty sure it doesn't work at all even if kinit'd due to a change that went in a long time back in runAsSparkUser, unless that changed fairly recently. This was brought up before as well but would have to dig up those details as to why it didn't go in. I think it was because if we are going to add support we should do it fully and document support for it and such.
I'm definitely fine with adding it, this would just be a feature vs a bug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos][Spark Shell] Obtain cred...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17387
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: WIP[SPARK-20060][Deploy][Kerberos]Support Standalone vis...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17387
cc @tgravescs, tested with a secured HDFS with standalone and works fine. And this pr has a lot of yarn specified security arguments to rename.
cc @jerryshao plz take a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos][Spark Shell] Obtain cred...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17387
yes\uff0cthis seem only fix the local mode\uff0cin standalone mode it still got problems of hdfs token lost @tgravescs @jerryshao .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos][Spark Shell] Obtain cred...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/17387
Here is a jira from a long time back: https://issues.apache.org/jira/browse/SPARK-2541
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org