You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2016/07/06 08:31:56 UTC
[GitHub] spark pull request #14065: [SPARK-16342][YARN][WIP] Add a configurable token...
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/14065
[SPARK-16342][YARN][WIP] Add a configurable token manager for Spark running on YARN
## What changes were proposed in this pull request?
Add a configurable token manager for Spark on running on yarn.
### Current Problems ###
1. Supported token provider is hard-coded, currently only hdfs, hbase and hive are supported and it is impossible for user to add new token provider without code changes.
2. Also this problem exits in timely token renewer and updater.
### Changes In This Proposal ###
In this proposal, to address the problems mentioned above and make the current code more cleaner and easier to understand, mainly has 3 changes:
1. Abstract a `ServiceTokenProvider` as well as `ServiceTokenRenewable` interface for token provider. Each service wants to communicate with Spark through token way needs to implement this interface.
2. Provide a `ConfigurableTokenManager` to manage all the register token providers, also token renewer and updater. Also this class offers the API for other modules to obtain tokens, get renewal interval and so on.
3. Implement 3 built-in token providers `HDFSTokenProvider`, `HiveTokenProvider` and `HBaseTokenProvider` to keep the same semantics as supported today. Whether to load in these built-in token providers is controlled by configuration "spark.yarn.security.tokens.${service}.enabled", by default for all the built-in token providers are loaded.
### Behavior Changes ###
For the end user there's no behavior change, we still use the same configuration `spark.yarn.security.tokens.${service}.enabled` to decide which token provider is enabled (hbase or hive).
For user implemented token provider (assume the name of token provider is "test") needs to add into this class should have two configurations:
1. `spark.yarn.security.tokens.test.enabled` to true
2. `spark.yarn.security.tokens.test.class` to the full qualified class name.
So we still keep the same semantics as current code while add one new configuration.
### Current Status ###
- [x] token provider interface and management framework.
- [x] implement built-in token providers (hdfs, hbase, hive).
- [ ] Coverage of unit test.
- [ ] Integrated test with security cluster.
## How was this patch tested?
Unit test and integrated test.
Please suggest and review, any comment is greatly appreciated.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-16342
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14065.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14065
----
commit 9e9311cd956eb0b2f900625b042c5c22d1016a08
Author: jerryshao <ss...@hortonworks.com>
Date: 2016-07-01T09:40:41Z
Add ConfigurableTokenManager initial commit
commit 3aaf0706d71321c2c150e0ddd21fee1cd218a4e1
Author: jerryshao <ss...@hortonworks.com>
Date: 2016-07-05T06:07:58Z
Further change on ConfigurableTokenManager
commit 9e8702140614f0551cad889e26f98ed36d7f6f15
Author: jerryshao <ss...@hortonworks.com>
Date: 2016-07-05T09:18:43Z
Some refactory works and unit test added
commit 8c0821b2074799a05c3dbb448368b3f195eff661
Author: jerryshao <ss...@hortonworks.com>
Date: 2016-07-06T06:38:29Z
Add more unit tests
commit 90f194e34ffd198d2b1ee5b04f24afd8c4454d90
Author: jerryshao <ss...@hortonworks.com>
Date: 2016-07-06T07:01:31Z
Add more comments
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62148/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69990265
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/token/ConfigurableTokenManager.scala ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.token
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A [[ConfigurableTokenManager]] to manage all the token providers register in this class. Also
+ * it provides other modules the functionality to obtain tokens, get token renewal interval and
+ * calculate the time length till next renewal.
+ *
+ * By default ConfigurableTokenManager has 3 built-in token providers, HDFSTokenProvider,
+ * HiveTokenProvider and HBaseTokenProvider, and this 3 token providers can also be controlled
+ * by configuration spark.yarn.security.tokens.{service}.enabled, if it is set to false, this
+ * provider will not be loaded.
+ *
+ * For other token providers which need to be loaded in should:
+ * 1. Implement [[ServiceTokenProvider]] or [[ServiceTokenRenewable]] if token renewal is
+ * required for this service.
+ * 2. set spark.yarn.security.tokens.{service}.enabled to true
+ * 3. Specify the class name through spark.yarn.security.tokens.{service}.class
+ *
+ */
+final class ConfigurableTokenManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val tokenProviderEnabledConfig = "spark\\.yarn\\.security\\.tokens\\.(.+)\\.enabled".r
+ private val tokenProviderClsConfig = "spark.yarn.security.tokens.%s.class"
+
+ // Maintain all the registered token providers
+ private val tokenProviders = mutable.HashMap[String, ServiceTokenProvider]()
+
+ private val defaultTokenProviders = Map(
+ "hdfs" -> "org.apache.spark.deploy.yarn.token.HDFSTokenProvider",
+ "hive" -> "org.apache.spark.deploy.yarn.token.HiveTokenProvider",
+ "hbase" -> "org.apache.spark.deploy.yarn.token.HBaseTokenProvider"
+ )
+
+ // AMDelegationTokenRenewer, this will only be create and started in the AM
+ private var _delegationTokenRenewer: AMDelegationTokenRenewer = null
+
+ // ExecutorDelegationTokenUpdater, this will only be created and started in the driver and
+ // executor side.
+ private var _delegationTokenUpdater: ExecutorDelegationTokenUpdater = null
+
+ def initialize(): Unit = {
+ // Copy SparkConf and add default enabled token provider configurations to SparkConf.
+ val clonedConf = sparkConf.clone
+ defaultTokenProviders.keys.foreach { key =>
+ clonedConf.setIfMissing(s"spark.yarn.security.tokens.$key.enabled", "true")
+ }
+
+ // Instantialize all the service token providers according to the configurations.
+ clonedConf.getAll.filter { case (key, value) =>
+ if (tokenProviderEnabledConfig.findPrefixOf(key).isDefined) {
+ value.toBoolean
+ } else {
+ false
+ }
+ }.map { case (key, _) =>
+ val tokenProviderEnabledConfig(service) = key
+ val cls = sparkConf.getOption(tokenProviderClsConfig.format(service))
+ .orElse(defaultTokenProviders.get(service))
+ (service, cls)
+ }.foreach { case (service, cls) =>
+ if (cls.isDefined) {
+ try {
+ val tokenProvider =
+ Utils.classForName(cls.get).newInstance().asInstanceOf[ServiceTokenProvider]
+ tokenProviders += (service -> tokenProvider)
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Fail to instantiate class ${cls.get}", e)
+ }
+ }
+ }
+ }
+
+ /**
+ * Get service token provider by name.
+ */
+ def getServiceTokenProvider(service: String): Option[ServiceTokenProvider] = {
+ tokenProviders.get(service)
+ }
+
+ /**
+ * Obtain tokens from all the token providers and add into credentials, also return as an array.
+ */
+ def obtainTokens(conf: Configuration, creds: Credentials): Array[Token[_]] = {
+ val tokenBuf = mutable.ArrayBuffer[Token[_]]()
+ tokenProviders.values.foreach { provider =>
+ if (provider.isTokenRequired(conf)) {
--- End diff --
Seems like you could remote providers that don't require tokens during `initialize` if you make a `SparkConf` available to it. Then you could avoid passing `SparkConf` to all of these methods.
You could also make `Configuration` a constructor args for the same reasons.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73424456
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HiveCredentialProvider.scala ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import scala.reflect.runtime.universe
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier
+import org.apache.hadoop.io.Text
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+private[security] class HiveCredentialProvider extends ServiceCredentialProvider with Logging {
+
+ override def serviceName: String = "hive"
+
+ private def hiveConf(hadoopConf: Configuration): Configuration = {
+ try {
+ val mirror = universe.runtimeMirror(Utils.getContextOrSparkClassLoader)
+ // the hive configuration class is a subclass of Hadoop Configuration, so can be cast down
+ // to a Configuration and used without reflection
+ val hiveConfClass = mirror.classLoader.loadClass("org.apache.hadoop.hive.conf.HiveConf")
+ // using the (Configuration, Class) constructor allows the current configuration to be
+ // included in the hive config.
+ val ctor = hiveConfClass.getDeclaredConstructor(classOf[Configuration],
+ classOf[Object].getClass)
+ ctor.newInstance(hadoopConf, hiveConfClass).asInstanceOf[Configuration]
+ } catch {
+ case NonFatal(e) =>
+ logDebug("Fail to create Hive Configuration", e)
+ hadoopConf
+ }
+ }
+
+ override def credentialsRequired(hadoopConf: Configuration): Boolean = {
+ if (!UserGroupInformation.isSecurityEnabled) {
--- End diff --
`UserGroupInformation.isSecurityEnabled && hiveConf...`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72749483
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -41,16 +43,18 @@ private[spark] class ExecutorDelegationTokenUpdater(
SparkHadoopUtil.get.getConfBypassingFSCache(
hadoopConf, new Path(credentialsFile).toUri.getScheme)
- private val delegationTokenRenewer =
+ private val credentialUpdater =
Executors.newSingleThreadScheduledExecutor(
- ThreadUtils.namedThreadFactory("Delegation Token Refresh Thread"))
+ ThreadUtils.namedThreadFactory("Credential Refresh Thread"))
- // On the executor, this thread wakes up and picks up new tokens from HDFS, if any.
- private val executorUpdaterRunnable =
+ // This thread wakes up and picks up new credentials from HDFS, if any.
+ private val credentialUpdaterRunnable =
new Runnable {
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
+ @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
--- End diff --
This method `updateCredentialsIfRequired ` will periodically call itself, so it is easy to code to put it as a field.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72490919
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -41,16 +43,18 @@ private[spark] class ExecutorDelegationTokenUpdater(
SparkHadoopUtil.get.getConfBypassingFSCache(
hadoopConf, new Path(credentialsFile).toUri.getScheme)
- private val delegationTokenRenewer =
+ private val credentialUpdater =
Executors.newSingleThreadScheduledExecutor(
- ThreadUtils.namedThreadFactory("Delegation Token Refresh Thread"))
+ ThreadUtils.namedThreadFactory("Credential Refresh Thread"))
- // On the executor, this thread wakes up and picks up new tokens from HDFS, if any.
- private val executorUpdaterRunnable =
+ // This thread wakes up and picks up new credentials from HDFS, if any.
+ private val credentialUpdaterRunnable =
new Runnable {
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
+ @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
--- End diff --
This is only ever used inside `updateCredentialsIfRequired`, do you need to keep it in a field?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73931365
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -53,50 +53,58 @@ private[spark] class CredentialUpdater(
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
- @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
+ /** Start the credentail updater thread periodically to get the credentials */
+ def bootstrap(): Unit = {
+ val bootstrapTime = sparkConf.get(CREDENTIALS_RENEWAL_TIME)
+ val remainingTime = bootstrapTime - System.currentTimeMillis()
+ if (remainingTime <= 0) {
+ // We just checked for new credentials but none were there, wait a minute and retry.
+ // This handles the shutdown case where the staging directory may have been removed(see
+ // SPARK-12316 for more details).
+ credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.MINUTES)
+ } else {
+ logInfo(s"Scheduling credentials refresh from HDFS in $remainingTime millis.")
+ credentialUpdater.schedule(credentialUpdaterRunnable, remainingTime, TimeUnit.MILLISECONDS)
+ }
+ }
- def updateCredentialsIfRequired(): Unit = {
- try {
+ private def updateCredentialsIfRequired(): Unit = {
+ val timeToNextUpdate = try {
val credentialsFilePath = new Path(credentialsFile)
val remoteFs = FileSystem.get(freshHadoopConf)
SparkHadoopUtil.get.listFilesSorted(
remoteFs, credentialsFilePath.getParent,
credentialsFilePath.getName, SparkHadoopUtil.SPARK_YARN_CREDS_TEMP_EXTENSION)
- .lastOption.foreach { credentialsStatus =>
+ .lastOption.map { credentialsStatus =>
val suffix = SparkHadoopUtil.get.getSuffixForCredentialsPath(credentialsStatus.getPath)
if (suffix > lastCredentialsFileSuffix) {
logInfo("Reading new credentials from " + credentialsStatus.getPath)
val newCredentials = getCredentialsFromHDFSFile(remoteFs, credentialsStatus.getPath)
lastCredentialsFileSuffix = suffix
UserGroupInformation.getCurrentUser.addCredentials(newCredentials)
logInfo("Credentials updated from credentials file.")
- timeOfNextUpdate = getTimeOfNextUpdateFromFileName(credentialsStatus.getPath)
+
+ val remainingTime =
+ getTimeOfNextUpdateFromFileName(credentialsStatus.getPath) - System.currentTimeMillis()
+ if (remainingTime <= 0) 60L * 1000 else remainingTime
} else {
- // Check every hour to see if new credentials arrived.
- logInfo("Updated credentials were expected, but the AM has not updated the " +
- "credentials yet, will check again in an hour.")
- credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.HOURS)
- return
+ // If current credential file is elder than expected, sleep 1 hour and check again.
+ 3600L * 1000
}
- }
- val remainingTime = timeOfNextUpdate - System.currentTimeMillis()
- if (remainingTime <= 0) {
- // We just checked for new credentials but none were there, wait a minute and retry.
- // This handles the shutdown case where the staging directory may have been removed(see
- // SPARK-12316 for more details).
- credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.MINUTES)
- } else {
- logInfo(s"Scheduling credentials refresh from HDFS in $remainingTime millis.")
- credentialUpdater.schedule(
- credentialUpdaterRunnable, remainingTime, TimeUnit.MILLISECONDS)
+ }.getOrElse {
+ // Wait for 1 minute to check again if there's no credential file currently
+ 60L * 1000
}
} catch {
// Since the file may get deleted while we are reading it, catch the Exception and come
// back in an hour to try again
case NonFatal(e) =>
logWarning("Error while trying to update credentials, will try again in 1 hour", e)
- credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.HOURS)
+ 3600L * 1000
--- End diff --
`TimeUnit...` you get the gist.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71048935
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in, any plugged-in credential provider wants to be managed by
+ * ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]] interface and put
+ * into resources to be loaded by ServiceLoader.
+ *
+ * Also the specific credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+final class ConfigurableCredentialManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = mutable.HashMap[String, ServiceCredentialProvider]()
+
+ // Default crendetial providers that will be loaded automatically, unless specifically disabled.
+ private val defaultCredentialProviders = Map(
+ "hdfs" -> "org.apache.spark.deploy.yarn.security.HDFSCredentialProvider",
+ "hive" -> "org.apache.spark.deploy.yarn.security.HiveCredentialProvider",
+ "hbase" -> "org.apache.spark.deploy.yarn.security.HBaseCredentialProvider"
+ )
+
+ // AMDelegationTokenRenewer, this will only be create and started in the AM
+ private var _delegationTokenRenewer: AMDelegationTokenRenewer = null
+
+ // ExecutorDelegationTokenUpdater, this will only be created and started in the driver and
+ // executor side.
+ private var _delegationTokenUpdater: ExecutorDelegationTokenUpdater = null
+
+ def initialize(): Unit = {
+ val providers = ServiceLoader.load(classOf[ServiceCredentialProvider],
+ Utils.getContextOrSparkClassLoader).asScala
+
+ // Filter out credentials in which spark.yarn.security.credentials.{service}.enabled is false.
+ providers.filter { p =>
+ sparkConf.getOption(providerEnabledConfig.format(p.serviceName))
+ .orElse {
+ sparkConf.getOption(deprecatedProviderEnabledConfig.format(p.serviceName)).map { c =>
+ logWarning(s"${deprecatedProviderEnabledConfig.format(p.serviceName)} is deprecated, " +
+ s"using ${providerEnabledConfig.format(p.serviceName)} instead")
+ c
+ }
+ }
+ .getOrElse(defaultCredentialProviders.keySet.find(_ == p.serviceName).isDefined.toString)
+ .toBoolean
+ }.foreach {
+ p => credentialProviders(p.serviceName) = p
+ }
+ }
+
+ /**
+ * Get credential provider for the specified service.
+ */
+ def getServiceCredentialProvider(service: String): Option[ServiceCredentialProvider] = {
+ credentialProviders.get(service)
+ }
+
+ /**
+ * Obtain credentials for specified service.
+ * @return time of next renewal if this service credential is renewable.
+ */
+ def obtainCredentialsFromService(
+ service: String,
+ hadoopConf: Configuration,
+ creds: Credentials): Option[Long] = {
+ getServiceCredentialProvider(service).flatMap { provider =>
+ if (provider.isCredentialRequired(hadoopConf)) {
+ provider.obtainCredentials(hadoopConf, creds)
+ } else {
+ None
+ }
+ }
+ }
+
+ /**
+ * Obtain credentials from all the registered providers.
+ * @return Array of time of next renewal if credential can be renewed.
+ * Otherwise will return None instead.
+ */
+ def obtainCredentials(hadoopConf: Configuration, creds: Credentials): Array[Option[Long]] = {
+ val timeOfNextRenewals = mutable.ArrayBuffer[Option[Long]]()
+ credentialProviders.values.foreach { provider =>
+ if (provider.isCredentialRequired(hadoopConf)) {
+ timeOfNextRenewals += provider.obtainCredentials(hadoopConf, creds)
+ } else {
+ logDebug(s"Service ${provider.serviceName} does not require a token." +
+ s" Check your configuration to see if security is disabled or not.")
+ }
+ }
+
+ timeOfNextRenewals.toArray
+ }
+
+ def delegationTokenRenewer(hadoopConf: Configuration): AMDelegationTokenRenewer = synchronized {
--- End diff --
This method has that weird singleton-like pattern again. Maybe it should just return a new renewer on every call, since it doesn't seem this class actually has any use for it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72336372
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMCredentialRenewer.scala ---
@@ -171,16 +172,30 @@ private[yarn] class AMDelegationTokenRenewer(
val tempCreds = keytabLoggedInUGI.getCredentials
val credentialsPath = new Path(credentialsFile)
val dst = credentialsPath.getParent
+ var nearestNextRenewalTime = Long.MaxValue
keytabLoggedInUGI.doAs(new PrivilegedExceptionAction[Void] {
// Get a copy of the credentials
override def run(): Void = {
- val nearestNextTime = credentialManager.obtainCredentials(freshHadoopConf, tempCreds)
- require(nearestNextTime > System.currentTimeMillis(),
- s"Time of next renewal $nearestNextTime is earlier than now")
- timeOfNextRenewal = nearestNextTime
+ nearestNextRenewalTime = credentialManager.obtainCredentials(freshHadoopConf, tempCreds)
null
}
})
+
+ val currTime = System.currentTimeMillis()
+ val timeOfNextUpdate = if (nearestNextRenewalTime <= currTime) {
+ // If next renewal time is earlier than current time, we set next renewal time to current
+ // time, this will trigger next renewal immediately. Also set next update time to current
+ // time. There still has a gap between token renewal and update will potentially introduce
+ // issue.
--- End diff --
It might be useful here to log a message here as we don't really expect this to happen. If it keeps happening it could be we aren't renewing them soon enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72483453
--- Diff: docs/running-on-yarn.md ---
@@ -525,16 +534,22 @@ token for the cluster's HDFS filesystem, and potentially for HBase and Hive.
An HBase token will be obtained if HBase is in on classpath, the HBase configuration declares
the application is secure (i.e. `hbase-site.xml` sets `hbase.security.authentication` to `kerberos`),
-and `spark.yarn.security.tokens.hbase.enabled` is not set to `false`.
+and `spark.yarn.security.credentials.hbase.enabled` is not set to `false`.
Similarly, a Hive token will be obtained if Hive is on the classpath, its configuration
includes a URI of the metadata store in `"hive.metastore.uris`, and
-`spark.yarn.security.tokens.hive.enabled` is not set to `false`.
+`spark.yarn.security.credentials.hive.enabled` is not set to `false`.
If an application needs to interact with other secure HDFS clusters, then
the tokens needed to access these clusters must be explicitly requested at
launch time. This is done by listing them in the `spark.yarn.access.namenodes` property.
+Also Spark supports interacting with other security services through plug-in mechanism, user must implement
--- End diff --
Rephrase:
Spark supports integrating with other security-aware services through the Java Services mechanism (see
`java.util.ServiceLoader`). To do that, implementations of `org.apache.spark.deploy.yarn.security.ServiceCredentialProvider` should be made available to Spark by listing their ames in the corresponding file in the jar's `META-INF/services` directory. These plug-ins can be disabled by setting...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71048752
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in, any plugged-in credential provider wants to be managed by
+ * ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]] interface and put
+ * into resources to be loaded by ServiceLoader.
+ *
+ * Also the specific credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+final class ConfigurableCredentialManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = mutable.HashMap[String, ServiceCredentialProvider]()
+
+ // Default crendetial providers that will be loaded automatically, unless specifically disabled.
+ private val defaultCredentialProviders = Map(
+ "hdfs" -> "org.apache.spark.deploy.yarn.security.HDFSCredentialProvider",
+ "hive" -> "org.apache.spark.deploy.yarn.security.HiveCredentialProvider",
+ "hbase" -> "org.apache.spark.deploy.yarn.security.HBaseCredentialProvider"
+ )
+
+ // AMDelegationTokenRenewer, this will only be create and started in the AM
+ private var _delegationTokenRenewer: AMDelegationTokenRenewer = null
+
+ // ExecutorDelegationTokenUpdater, this will only be created and started in the driver and
+ // executor side.
+ private var _delegationTokenUpdater: ExecutorDelegationTokenUpdater = null
+
+ def initialize(): Unit = {
+ val providers = ServiceLoader.load(classOf[ServiceCredentialProvider],
+ Utils.getContextOrSparkClassLoader).asScala
+
+ // Filter out credentials in which spark.yarn.security.credentials.{service}.enabled is false.
+ providers.filter { p =>
+ sparkConf.getOption(providerEnabledConfig.format(p.serviceName))
+ .orElse {
+ sparkConf.getOption(deprecatedProviderEnabledConfig.format(p.serviceName)).map { c =>
+ logWarning(s"${deprecatedProviderEnabledConfig.format(p.serviceName)} is deprecated, " +
+ s"using ${providerEnabledConfig.format(p.serviceName)} instead")
+ c
+ }
+ }
+ .getOrElse(defaultCredentialProviders.keySet.find(_ == p.serviceName).isDefined.toString)
+ .toBoolean
+ }.foreach {
+ p => credentialProviders(p.serviceName) = p
--- End diff --
Don't use a mutable hash map. Instead:
```
.map { p => (p.serviceName, p) }.toMap
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71358595
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -171,10 +174,10 @@ private[yarn] class AMDelegationTokenRenewer(
keytabLoggedInUGI.doAs(new PrivilegedExceptionAction[Void] {
// Get a copy of the credentials
override def run(): Void = {
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + dst
- hadoopUtil.obtainTokensForNamenodes(nns, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHiveMetastore(sparkConf, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHBase(sparkConf, freshHadoopConf, tempCreds)
+ val nearestNextTime = credentialManager.obtainCredentials(freshHadoopConf, tempCreds)
+ require(nearestNextTime > System.currentTimeMillis(),
+ s"Time of next renewal $nearestNextTime is earlier than now")
+ timeOfNextRenewal = nearestNextTime
--- End diff --
Nevermind, ignore this I misread the code... that is only if exception occurs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14065
Looks fine. There are some possible enhancements (e.g. what looks like some code repetition in the HDFS provider, neither Hive nor HBase return a token renewal time, etc) but those can be done separately.
@tgravescs did you have any remaining comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #63068 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63068/consoleFull)** for PR 14065 at commit [`ac0062c`](https://github.com/apache/spark/commit/ac0062cf9d8a6f48d03fe335ba2ea7ec5f00e8e7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61912/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14065
I still see a few, mostly minor, issues. I think the provider initialization should be cleaned up to avoid things like the hacky code in the HDFS provider.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72494408
--- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/security/HDFSCredentialProviderSuite.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.scalatest.{Matchers, PrivateMethodTester}
+
+import org.apache.spark.{SparkConf, SparkException, SparkFunSuite}
+
+class HDFSCredentialProviderSuite
+ extends SparkFunSuite
+ with PrivateMethodTester
+ with Matchers {
+ private val _nnsToAccess = PrivateMethod[Set[Path]]('nnsToAccess)
+ private val _getTokenRenewer = PrivateMethod[String]('getTokenRenewer)
+
+ private def nnsToAccess(
+ hdfsCredentialProvider: HDFSCredentialProvider, sparkConf: SparkConf): Set[Path] = {
+ hdfsCredentialProvider invokePrivate _nnsToAccess(sparkConf)
+ }
+
+ private def getTokenRenewer(
+ hdfsCredentialProvider: HDFSCredentialProvider, conf: Configuration): String = {
+ hdfsCredentialProvider invokePrivate _getTokenRenewer(conf)
+ }
+
+ private var hdfsCredentialProvider: HDFSCredentialProvider = null
+
+ override def beforeAll() {
+ super.beforeAll()
+
+ if (hdfsCredentialProvider == null) {
+ hdfsCredentialProvider = new HDFSCredentialProvider()
+ }
+ }
+
+ override def afterAll() {
+ if (hdfsCredentialProvider != null) {
+ hdfsCredentialProvider = null
+ }
+
+ super.afterAll()
+ }
+
+ test("check access nns empty") {
+ val sparkConf = new SparkConf()
+ .set("spark.yarn.access.namenodes", "")
--- End diff --
I don't see a whole lot of value in these tests; you're basically testing the config parsing code, which is already tested thoroughly in other places. There's no custom parsing going on the HDFS provider code, aside from the transformation from String to Path.
If you could instead test that the HDFS provider were actually getting tokens for all NNs that would be great, but I'm not sure you can do that easily with the current test infrastructure (e.g. you'd have to somehow be able to mock the `FileSystem` instance to see if the interesting methods were called).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14065
I'm having trouble finding the bandwidth to look at the updated patch, but it's on my list... there were some replies to my comments that I want to take a closer look at.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72493335
--- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManagerSuite.scala ---
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.io.Text
+import org.apache.hadoop.security.Credentials
+import org.apache.hadoop.security.token.Token
+import org.scalatest.{BeforeAndAfter, Matchers}
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.deploy.yarn.config._
+
+class ConfigurableCredentialManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfter {
+ private var credentialManager: ConfigurableCredentialManager = null
+ private var sparkConf: SparkConf = null
+ private var hadoopConf: Configuration = null
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+
+ sparkConf = new SparkConf()
+ hadoopConf = new Configuration()
+ System.setProperty("SPARK_YARN_MODE", "true")
+ }
+
+ override def afterAll(): Unit = {
+ System.clearProperty("SPARK_YARN_MODE")
+
+ super.afterAll()
+ }
+
+ test("Correctly load default credential providers") {
+ credentialManager = new ConfigurableCredentialManager(sparkConf, hadoopConf)
+
+ credentialManager.getServiceCredentialProvider("hdfs") should not be (None)
+ credentialManager.getServiceCredentialProvider("hbase") should not be (None)
+ credentialManager.getServiceCredentialProvider("hive") should not be (None)
+ }
+
+ test("disable hive credential provider") {
+ sparkConf.set("spark.yarn.security.credentials.hive.enabled", "false")
+ credentialManager = new ConfigurableCredentialManager(sparkConf, hadoopConf)
+
+ credentialManager.getServiceCredentialProvider("hdfs") should not be (None)
+ credentialManager.getServiceCredentialProvider("hbase") should not be (None)
+ credentialManager.getServiceCredentialProvider("hive") should be (None)
+ }
+
+ test("using deprecated configurations") {
+ sparkConf.set("spark.yarn.security.tokens.hdfs.enabled", "false")
+ sparkConf.set("spark.yarn.security.tokens.hive.enabled", "false")
+ credentialManager = new ConfigurableCredentialManager(sparkConf, hadoopConf)
+
+ credentialManager.getServiceCredentialProvider("hdfs") should be (None)
+ credentialManager.getServiceCredentialProvider("hive") should be (None)
+ credentialManager.getServiceCredentialProvider("test") should not be (None)
+ credentialManager.getServiceCredentialProvider("hbase") should not be (None)
+ }
+
+ test("plug in customized credential provider") {
+ credentialManager = new ConfigurableCredentialManager(sparkConf, hadoopConf)
+
+ credentialManager.getServiceCredentialProvider("test") should not be (None)
--- End diff --
minor but this is implicitly tested by the previous test. maybe remove the check for the "test" provider from there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/14065
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62468 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62468/consoleFull)** for PR 14065 at commit [`9863ca4`](https://github.com/apache/spark/commit/9863ca42427479bf79731f204f606d1ccae20491).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69989212
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/token/ConfigurableTokenManager.scala ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.token
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A [[ConfigurableTokenManager]] to manage all the token providers register in this class. Also
+ * it provides other modules the functionality to obtain tokens, get token renewal interval and
+ * calculate the time length till next renewal.
+ *
+ * By default ConfigurableTokenManager has 3 built-in token providers, HDFSTokenProvider,
+ * HiveTokenProvider and HBaseTokenProvider, and this 3 token providers can also be controlled
+ * by configuration spark.yarn.security.tokens.{service}.enabled, if it is set to false, this
+ * provider will not be loaded.
+ *
+ * For other token providers which need to be loaded in should:
+ * 1. Implement [[ServiceTokenProvider]] or [[ServiceTokenRenewable]] if token renewal is
+ * required for this service.
+ * 2. set spark.yarn.security.tokens.{service}.enabled to true
+ * 3. Specify the class name through spark.yarn.security.tokens.{service}.class
+ *
+ */
+final class ConfigurableTokenManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val tokenProviderEnabledConfig = "spark\\.yarn\\.security\\.tokens\\.(.+)\\.enabled".r
+ private val tokenProviderClsConfig = "spark.yarn.security.tokens.%s.class"
+
+ // Maintain all the registered token providers
+ private val tokenProviders = mutable.HashMap[String, ServiceTokenProvider]()
+
+ private val defaultTokenProviders = Map(
--- End diff --
I'd rather use `java.util.ServiceLoader` for this. You'll need something like that at some point anyway, to support other token providers. Doing that now has the extra benefit of using the same code for built-in and third party providers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #63516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63516/consoleFull)** for PR 14065 at commit [`bce8cd6`](https://github.com/apache/spark/commit/bce8cd6a5796fc95a432c169c8c40bea552382f0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14065
Also thinking about one example to land this feature, I think Kafka might be one candidate, they also have delegation token based proposal [KIP-48](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71046990
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -69,6 +71,9 @@ private[yarn] class AMDelegationTokenRenewer(
private val freshHadoopConf =
hadoopUtil.getConfBypassingFSCache(hadoopConf, new Path(credentialsFile).toUri.getScheme)
+ @volatile private var timeOfNextRenewal =
+ sparkConf.get(CREDENTIALS_RENEWAL_TIME).getOrElse(Long.MaxValue)
--- End diff --
From my understanding you should never get to this code when `CREDENTIALS_RENEWAL_TIME` is not set, so you should use `.get` to force and exception in case the config is not set.
If indeed that situation can occur and you really need a default value, it should be declared in the constant itself and not here where it's used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71357664
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -171,10 +174,10 @@ private[yarn] class AMDelegationTokenRenewer(
keytabLoggedInUGI.doAs(new PrivilegedExceptionAction[Void] {
// Get a copy of the credentials
override def run(): Void = {
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + dst
- hadoopUtil.obtainTokensForNamenodes(nns, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHiveMetastore(sparkConf, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHBase(sparkConf, freshHadoopConf, tempCreds)
+ val nearestNextTime = credentialManager.obtainCredentials(freshHadoopConf, tempCreds)
+ require(nearestNextTime > System.currentTimeMillis(),
+ s"Time of next renewal $nearestNextTime is earlier than now")
+ timeOfNextRenewal = nearestNextTime
--- End diff --
driverTokenRenewerRunnable is hardcoded to renew every hour:
delegationTokenRenewer.schedule(this, 1, TimeUnit.HOURS)
We should probably make this configurable. Ideally nothing needs to be renewed that often but if we are wanting to support arbitrary services making it configurable would make sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72752972
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HDFSCredentialProvider.scala ---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.io.{ByteArrayInputStream, DataInputStream}
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier
+import org.apache.hadoop.mapred.Master
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.deploy.yarn.config._
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+
+private[security] class HDFSCredentialProvider extends ServiceCredentialProvider with Logging {
+
+ private val sparkConf = new SparkConf()
--- End diff --
I would like to add this `sparkConf` as an argument to `obtainCredentials`. `isCredentialRequired` would be better not to do initialization things, since it will be called every time before obtaining credentials.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73417870
--- Diff: docs/running-on-yarn.md ---
@@ -461,15 +461,24 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
</tr>
<tr>
- <td><code>spark.yarn.security.tokens.${service}.enabled</code></td>
+ <td><code>spark.yarn.security.credentials.${service}.enabled</code></td>
<td><code>true</code></td>
<td>
- Controls whether to retrieve delegation tokens for non-HDFS services when security is enabled.
- By default, delegation tokens for all supported services are retrieved when those services are
+ Controls whether to obtain credentials for services when security is enabled.
+ By default, credentials for all supported services are retrieved when those services are
configured, but it's possible to disable that behavior if it somehow conflicts with the
- application being run.
+ application being run. For the details please see
--- End diff --
nit: "For further details, please..."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71047809
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -171,10 +175,13 @@ private[yarn] class AMDelegationTokenRenewer(
keytabLoggedInUGI.doAs(new PrivilegedExceptionAction[Void] {
// Get a copy of the credentials
override def run(): Void = {
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + dst
- hadoopUtil.obtainTokensForNamenodes(nns, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHiveMetastore(sparkConf, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHBase(sparkConf, freshHadoopConf, tempCreds)
+ var nearestNextTime = Long.MaxValue
+ credentialManager.obtainCredentials(freshHadoopConf, tempCreds).foreach { nextTime =>
--- End diff --
I also noticed that the logic will renew all tokens when the first token needs renewal. It's probably fine since this should happen very rarely (once every few days). But that also means that `obtainCredentialsFromService` is completely unused and can be deleted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71362442
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ExecutorDelegationTokenUpdater.scala ---
@@ -107,6 +110,14 @@ private[spark] class ExecutorDelegationTokenUpdater(
}
}
+ private def getTimeOfNextUpdateFromFileName(credentialsPath: Path): Long = {
+ val name = credentialsPath.getName
+ val index = name.lastIndexOf(SparkHadoopUtil.SPARK_YARN_CREDS_COUNTER_DELIM)
+ val slice = name.substring(0, index)
+ val last2index = slice.lastIndexOf(SparkHadoopUtil.SPARK_YARN_CREDS_COUNTER_DELIM)
+ name.substring(last2index).toLong
--- End diff --
is this giving you the right time? It looks like its giving you the credentials file name instead of the next time to update.
name is: credentials-UUID-timestamp-1... slice = credentials-UUID-timestamp, name.substring of that is credentials-UUID. don't you want name.substring(last2index, index) ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14065
couple minor nits, otherwise seems fine, but I want to run a couple tests locally.
@vanzin were all your comments addressed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71102254
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in, any plugged-in credential provider wants to be managed by
+ * ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]] interface and put
+ * into resources to be loaded by ServiceLoader.
+ *
+ * Also the specific credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+final class ConfigurableCredentialManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = mutable.HashMap[String, ServiceCredentialProvider]()
+
+ // Default crendetial providers that will be loaded automatically, unless specifically disabled.
+ private val defaultCredentialProviders = Map(
--- End diff --
Sorry my fault, it is not necessary now, I will remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71104553
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -390,8 +393,22 @@ private[spark] class Client(
// Upload Spark and the application JAR to the remote file system if necessary,
// and add them as local resources to the application master.
val fs = destDir.getFileSystem(hadoopConf)
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + destDir
- YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
+
+ // Merge credentials obtained from registered providers
+ var nearestTimeOfNextRenewal = Long.MaxValue
+ credentialManager.obtainCredentials(hadoopConf, credentials).foreach { t =>
--- End diff --
Yes, in the current implementation of `HDFSCredentialProvider`, time of next token renewal will only be calculated when principal is provided (`loginFromKeytab` is true).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62148 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62148/consoleFull)** for PR 14065 at commit [`0fbf25b`](https://github.com/apache/spark/commit/0fbf25b51285ea433a3dcf733b92a1d785b3d017).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62506/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72492363
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HDFSCredentialProvider.scala ---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.io.{ByteArrayInputStream, DataInputStream}
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier
+import org.apache.hadoop.mapred.Master
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.deploy.yarn.config._
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+
+private[security] class HDFSCredentialProvider extends ServiceCredentialProvider with Logging {
+
+ private val sparkConf = new SparkConf()
+
+ // NameNode to access, used to get tokens from different FileSystems
+ private val unionNNsToAccess = {
+ val tmpConf = SparkHadoopUtil.get.newConfiguration(sparkConf)
+ val stageDir = sparkConf.get(STAGING_DIR).map(new Path(_))
+ .getOrElse(FileSystem.get(tmpConf).getHomeDirectory)
+ nnsToAccess(sparkConf) + stageDir
+ }
+
+ // Token renewal interval, this value will be set in the first call,
+ // if None means no token renewer specified, so cannot get token renewal interval.
+ private var tokenRenewalInterval: Option[Long] = null
+
+ override val serviceName: String = "hdfs"
+
+ override def obtainCredentials(hadoopConf: Configuration, creds: Credentials): Option[Long] = {
+ // Obtain tokens from HDFS and add into Credentials
+ unionNNsToAccess.foreach { dst =>
+ val dstFs = dst.getFileSystem(hadoopConf)
+ logInfo("getting token for namenode: " + dst)
+ dstFs.addDelegationTokens(getTokenRenewer(hadoopConf), creds)
+ }
+
+ // Get the token renewal interval if it is not set. It will only be called once.
+ if (tokenRenewalInterval == null) {
+ tokenRenewalInterval = getTokenRenewalInterval(hadoopConf)
+ }
+
+ // Get the time of next renewal.
+ tokenRenewalInterval.map { interval =>
+ creds.getAllTokens.asScala
+ .filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
+ .map { t =>
+ val identifier = new DelegationTokenIdentifier()
+ identifier.readFields(new DataInputStream(new ByteArrayInputStream(t.getIdentifier)))
+ identifier.getIssueDate + interval
+ }.foldLeft(0L)(math.max)
+ }
+ }
+
+ private def getTokenRenewalInterval(hadoopConf: Configuration): Option[Long] = {
+ // We cannot use the tokens generated with renewer yarn. Trying to renew
+ // those will fail with an access control issue. So create new tokens with the logged in
+ // user as renewer.
+ sparkConf.get(PRINCIPAL).map { renewer =>
+ val creds = new Credentials()
+ unionNNsToAccess.foreach { dst =>
+ val dstFs = dst.getFileSystem(hadoopConf)
+ dstFs.addDelegationTokens(renewer, creds)
+ }
+ val t = creds.getAllTokens.asScala
+ .filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
+ .head
+ val newExpiration = t.renew(hadoopConf)
+ val identifier = new DelegationTokenIdentifier()
+ identifier.readFields(new DataInputStream(new ByteArrayInputStream(t.getIdentifier)))
+ val interval = newExpiration - identifier.getIssueDate
+ logInfo(s"Renewal Interval is $interval")
+ interval
+ }
+ }
+
+ private def getTokenRenewer(conf: Configuration): String = {
+ val delegTokenRenewer = Master.getMasterPrincipal(conf)
+ logDebug("delegation token renewer is: " + delegTokenRenewer)
+ if (delegTokenRenewer == null || delegTokenRenewer.length() == 0) {
+ val errorMessage = "Can't get Master Kerberos principal for use as renewer"
+ logError(errorMessage)
+ throw new SparkException(errorMessage)
+ }
+
+ delegTokenRenewer
+ }
+
+ private def nnsToAccess(sparkConf: SparkConf): Set[Path] = {
--- End diff --
This seems to only be called from a single place, might as well inline it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72493513
--- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManagerSuite.scala ---
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.io.Text
+import org.apache.hadoop.security.Credentials
+import org.apache.hadoop.security.token.Token
+import org.scalatest.{BeforeAndAfter, Matchers}
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.deploy.yarn.config._
+
+class ConfigurableCredentialManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfter {
+ private var credentialManager: ConfigurableCredentialManager = null
+ private var sparkConf: SparkConf = null
+ private var hadoopConf: Configuration = null
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+
+ sparkConf = new SparkConf()
+ hadoopConf = new Configuration()
+ System.setProperty("SPARK_YARN_MODE", "true")
+ }
+
+ override def afterAll(): Unit = {
+ System.clearProperty("SPARK_YARN_MODE")
+
+ super.afterAll()
+ }
+
+ test("Correctly load default credential providers") {
+ credentialManager = new ConfigurableCredentialManager(sparkConf, hadoopConf)
+
+ credentialManager.getServiceCredentialProvider("hdfs") should not be (None)
+ credentialManager.getServiceCredentialProvider("hbase") should not be (None)
+ credentialManager.getServiceCredentialProvider("hive") should not be (None)
+ }
+
+ test("disable hive credential provider") {
+ sparkConf.set("spark.yarn.security.credentials.hive.enabled", "false")
+ credentialManager = new ConfigurableCredentialManager(sparkConf, hadoopConf)
+
+ credentialManager.getServiceCredentialProvider("hdfs") should not be (None)
+ credentialManager.getServiceCredentialProvider("hbase") should not be (None)
+ credentialManager.getServiceCredentialProvider("hive") should be (None)
+ }
+
+ test("using deprecated configurations") {
+ sparkConf.set("spark.yarn.security.tokens.hdfs.enabled", "false")
+ sparkConf.set("spark.yarn.security.tokens.hive.enabled", "false")
+ credentialManager = new ConfigurableCredentialManager(sparkConf, hadoopConf)
+
+ credentialManager.getServiceCredentialProvider("hdfs") should be (None)
+ credentialManager.getServiceCredentialProvider("hive") should be (None)
+ credentialManager.getServiceCredentialProvider("test") should not be (None)
+ credentialManager.getServiceCredentialProvider("hbase") should not be (None)
+ }
+
+ test("plug in customized credential provider") {
+ credentialManager = new ConfigurableCredentialManager(sparkConf, hadoopConf)
+
+ credentialManager.getServiceCredentialProvider("test") should not be (None)
+ }
+
+ test("verify obtaining credentials from provider") {
+ credentialManager = new ConfigurableCredentialManager(sparkConf, hadoopConf)
+ val creds = new Credentials()
+
+ // Tokens can only be obtained from TestTokenProvider, for hdfs, hbase and hive tokens cannot
+ // be obtained.
+ credentialManager.obtainCredentials(hadoopConf, creds)
+ val tokens = creds.getAllTokens
+ tokens.size() should be (1)
+ tokens.iterator().next().getService should be (new Text("test"))
+ }
+
+ test("verify getting credential renewal info") {
+ credentialManager = new ConfigurableCredentialManager(sparkConf, hadoopConf)
+ val creds = new Credentials()
+
+ val testCredentialProvider = credentialManager.getServiceCredentialProvider("test").get
+ .asInstanceOf[TestCredentialProvider]
+ // Only TestTokenProvider can get the time of next token renewal
+ val nextRenewal = credentialManager.obtainCredentials(hadoopConf, creds)
+ nextRenewal should be (testCredentialProvider.timeOfNextTokenRenewal)
+ }
+
+ test("Obtain tokens For HiveMetastore") {
--- End diff --
nit: all test descriptions start with lower case, so should this one (and the next)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63200/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14065
@vanzin , I did refactoring on some interfaces, especially for `obtainCredentials` method, and the implementation of `HDFSCredentialProvider` and `HiveCredential`, would you please help to review again? Thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73420791
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in if not explicitly disabled, any plugged-in credential provider wants to be
+ * managed by ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]]
+ * interface and put into resources/META-INF/services to be loaded by ServiceLoader.
+ *
+ * Also each credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+private[yarn] final class ConfigurableCredentialManager (
--- End diff --
nit: no space before '('
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72490136
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in if not explicitly disabled, any plugged-in credential provider wants to be
+ * managed by ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]]
+ * interface and put into resources/META-INF/services to be loaded by ServiceLoader.
+ *
+ * Also each credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+private[yarn] final class ConfigurableCredentialManager (
+ sparkConf: SparkConf, hadoopConf: Configuration) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = {
+ val providers = ServiceLoader.load(classOf[ServiceCredentialProvider],
+ Utils.getContextOrSparkClassLoader).asScala
+
+ // Filter out credentials in which spark.yarn.security.credentials.{service}.enabled is false.
+ providers.filter { p =>
+ sparkConf.getOption(providerEnabledConfig.format(p.serviceName)).map(_.toBoolean)
+ .orElse {
+ sparkConf.getOption(deprecatedProviderEnabledConfig.format(p.serviceName)).map { c =>
+ logWarning(s"${deprecatedProviderEnabledConfig.format(p.serviceName)} is deprecated, " +
+ s"using ${providerEnabledConfig.format(p.serviceName)} instead")
+ c.toBoolean
+ }
+ }.getOrElse(true)
+ }.map { p => (p.serviceName, p) }.toMap
+ }
+
+ /**
+ * Get credential provider for the specified service.
+ */
+ def getServiceCredentialProvider(service: String): Option[ServiceCredentialProvider] = {
+ credentialProviders.get(service)
+ }
+
+ /**
+ * Obtain credentials from all the registered providers.
+ * @return nearest time of next renewal, Long.MaxValue if all the credentials aren't renewable,
+ * otherwise the nearest renewal time of any credentials will be returned.
+ */
+ def obtainCredentials(hadoopConf: Configuration, creds: Credentials): Long = {
+ credentialProviders.values.flatMap { provider =>
+ if (provider.isCredentialRequired(hadoopConf)) {
+ provider.obtainCredentials(hadoopConf, creds)
+ } else {
+ logDebug(s"Service ${provider.serviceName} does not require a token." +
+ s" Check your configuration to see if security is disabled or not.")
+ None
+ }
+ }.foldLeft(Long.MaxValue)(math.min)
+ }
+
+ def credentialRenewer: AMCredentialRenewer = {
--- End diff --
The method is named as if it were a property, yet it returns something different on every call. It should have a signature that reflects that. e.g. `newCredentialRenewer()`. scaladoc would be nice too, especially to clarify that the caller is responsible for the lifecycle of the returned instance.
Also applies to `credentialUpdater`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73418370
--- Diff: docs/running-on-yarn.md ---
@@ -525,16 +534,23 @@ token for the cluster's HDFS filesystem, and potentially for HBase and Hive.
An HBase token will be obtained if HBase is in on classpath, the HBase configuration declares
the application is secure (i.e. `hbase-site.xml` sets `hbase.security.authentication` to `kerberos`),
-and `spark.yarn.security.tokens.hbase.enabled` is not set to `false`.
+and `spark.yarn.security.credentials.hbase.enabled` is not set to `false`.
Similarly, a Hive token will be obtained if Hive is on the classpath, its configuration
includes a URI of the metadata store in `"hive.metastore.uris`, and
-`spark.yarn.security.tokens.hive.enabled` is not set to `false`.
+`spark.yarn.security.credentials.hive.enabled` is not set to `false`.
If an application needs to interact with other secure HDFS clusters, then
the tokens needed to access these clusters must be explicitly requested at
launch time. This is done by listing them in the `spark.yarn.access.namenodes` property.
+Spark supports integrating with other security-aware services through Java Services mechanism (see
+`java.util.ServiceLoader`). To do that, implementations of `org.apache.spark.deploy.yarn.security.ServiceCredentialProvider`
+ should be available to Spark by listing their names in the corresponding file in the jar's
+ `META-INF/services` directory. These plug-ins can be disabled by setting
--- End diff --
nit: there's a leading space in this line and the next
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #63068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63068/consoleFull)** for PR 14065 at commit [`ac0062c`](https://github.com/apache/spark/commit/ac0062cf9d8a6f48d03fe335ba2ea7ec5f00e8e7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62915/consoleFull)** for PR 14065 at commit [`dc3be43`](https://github.com/apache/spark/commit/dc3be432c5467d1d6f0fedbcb0cd172ab0fa8bba).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r70074645
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/token/ServiceTokenProvider.scala ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.token
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.SparkConf
+
+/**
+ * An interface to provide tokens for service, any service wants to communicate with Spark
+ * through token way needs to implement this interface and register into
+ * [[ConfigurableTokenManager]] through configurations.
+ */
+trait ServiceTokenProvider {
+
+ /**
+ * Name of the ServiceTokenProvider, should be unique. Using this to distinguish different
+ * service.
+ */
+ def serviceName: String
+
+ /**
+ * Used to indicate whether a token is required.
+ */
+ def isTokenRequired(conf: Configuration): Boolean = {
+ UserGroupInformation.isSecurityEnabled
+ }
+
+ /**
+ * Obtain tokens from this service, tokens will be added into Credentials and return as array.
+ */
+ def obtainTokensFromService(
--- End diff --
This looks much more consolidate, I will change to this, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73931146
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -53,50 +53,58 @@ private[spark] class CredentialUpdater(
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
- @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
+ /** Start the credentail updater thread periodically to get the credentials */
+ def bootstrap(): Unit = {
+ val bootstrapTime = sparkConf.get(CREDENTIALS_RENEWAL_TIME)
+ val remainingTime = bootstrapTime - System.currentTimeMillis()
+ if (remainingTime <= 0) {
+ // We just checked for new credentials but none were there, wait a minute and retry.
+ // This handles the shutdown case where the staging directory may have been removed(see
+ // SPARK-12316 for more details).
+ credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.MINUTES)
+ } else {
+ logInfo(s"Scheduling credentials refresh from HDFS in $remainingTime millis.")
+ credentialUpdater.schedule(credentialUpdaterRunnable, remainingTime, TimeUnit.MILLISECONDS)
+ }
+ }
- def updateCredentialsIfRequired(): Unit = {
- try {
+ private def updateCredentialsIfRequired(): Unit = {
+ val timeToNextUpdate = try {
val credentialsFilePath = new Path(credentialsFile)
val remoteFs = FileSystem.get(freshHadoopConf)
SparkHadoopUtil.get.listFilesSorted(
remoteFs, credentialsFilePath.getParent,
credentialsFilePath.getName, SparkHadoopUtil.SPARK_YARN_CREDS_TEMP_EXTENSION)
- .lastOption.foreach { credentialsStatus =>
+ .lastOption.map { credentialsStatus =>
val suffix = SparkHadoopUtil.get.getSuffixForCredentialsPath(credentialsStatus.getPath)
if (suffix > lastCredentialsFileSuffix) {
logInfo("Reading new credentials from " + credentialsStatus.getPath)
val newCredentials = getCredentialsFromHDFSFile(remoteFs, credentialsStatus.getPath)
lastCredentialsFileSuffix = suffix
UserGroupInformation.getCurrentUser.addCredentials(newCredentials)
logInfo("Credentials updated from credentials file.")
- timeOfNextUpdate = getTimeOfNextUpdateFromFileName(credentialsStatus.getPath)
+
+ val remainingTime =
+ getTimeOfNextUpdateFromFileName(credentialsStatus.getPath) - System.currentTimeMillis()
+ if (remainingTime <= 0) 60L * 1000 else remainingTime
--- End diff --
`TimeUnit.HOUR.toMillis(1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71363930
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ServiceCredentialProvider.scala ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * ::DeveloperApi::
+ * A credential provider for this service, any service want to access by spark through security
--- End diff --
slight wording change:
A credentials provider for a service. A user must implement this if they need to access a secure service from spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69988196
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -390,8 +390,9 @@ private[spark] class Client(
// Upload Spark and the application JAR to the remote file system if necessary,
// and add them as local resources to the application master.
val fs = destDir.getFileSystem(hadoopConf)
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + destDir
- YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
+ hdfsTokenProvider(sparkConf).setNameNodesToAccess(sparkConf, Set(destDir))
--- End diff --
+1; it would be better if all interactions with token providers were done through the common interface; it seems like these HDFS-specific calls could easily be moved to the HDFS token provider.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62219/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71044195
--- Diff: dev/.rat-excludes ---
@@ -99,3 +99,4 @@ spark-deps-.*
.*tsv
org.apache.spark.scheduler.ExternalClusterManager
.*\.sql
+org.apache.spark.deploy.yarn.security.ServiceCredentialProvider
--- End diff --
Instead of adding these files like this, isn't it possible to just say `META-INF/services/*`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72750018
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -107,8 +110,16 @@ private[spark] class ExecutorDelegationTokenUpdater(
}
}
+ private def getTimeOfNextUpdateFromFileName(credentialsPath: Path): Long = {
+ val name = credentialsPath.getName
--- End diff --
IIUC credential file name is combined by `credentials-uuid-timestamp-integer`, uuid has `-` in it. Maybe it is safer to use the current way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73931312
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -53,50 +53,58 @@ private[spark] class CredentialUpdater(
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
- @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
+ /** Start the credentail updater thread periodically to get the credentials */
+ def bootstrap(): Unit = {
+ val bootstrapTime = sparkConf.get(CREDENTIALS_RENEWAL_TIME)
+ val remainingTime = bootstrapTime - System.currentTimeMillis()
+ if (remainingTime <= 0) {
+ // We just checked for new credentials but none were there, wait a minute and retry.
+ // This handles the shutdown case where the staging directory may have been removed(see
+ // SPARK-12316 for more details).
+ credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.MINUTES)
+ } else {
+ logInfo(s"Scheduling credentials refresh from HDFS in $remainingTime millis.")
+ credentialUpdater.schedule(credentialUpdaterRunnable, remainingTime, TimeUnit.MILLISECONDS)
+ }
+ }
- def updateCredentialsIfRequired(): Unit = {
- try {
+ private def updateCredentialsIfRequired(): Unit = {
+ val timeToNextUpdate = try {
val credentialsFilePath = new Path(credentialsFile)
val remoteFs = FileSystem.get(freshHadoopConf)
SparkHadoopUtil.get.listFilesSorted(
remoteFs, credentialsFilePath.getParent,
credentialsFilePath.getName, SparkHadoopUtil.SPARK_YARN_CREDS_TEMP_EXTENSION)
- .lastOption.foreach { credentialsStatus =>
+ .lastOption.map { credentialsStatus =>
val suffix = SparkHadoopUtil.get.getSuffixForCredentialsPath(credentialsStatus.getPath)
if (suffix > lastCredentialsFileSuffix) {
logInfo("Reading new credentials from " + credentialsStatus.getPath)
val newCredentials = getCredentialsFromHDFSFile(remoteFs, credentialsStatus.getPath)
lastCredentialsFileSuffix = suffix
UserGroupInformation.getCurrentUser.addCredentials(newCredentials)
logInfo("Credentials updated from credentials file.")
- timeOfNextUpdate = getTimeOfNextUpdateFromFileName(credentialsStatus.getPath)
+
+ val remainingTime =
+ getTimeOfNextUpdateFromFileName(credentialsStatus.getPath) - System.currentTimeMillis()
+ if (remainingTime <= 0) 60L * 1000 else remainingTime
} else {
- // Check every hour to see if new credentials arrived.
- logInfo("Updated credentials were expected, but the AM has not updated the " +
- "credentials yet, will check again in an hour.")
- credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.HOURS)
- return
+ // If current credential file is elder than expected, sleep 1 hour and check again.
+ 3600L * 1000
}
- }
- val remainingTime = timeOfNextUpdate - System.currentTimeMillis()
- if (remainingTime <= 0) {
- // We just checked for new credentials but none were there, wait a minute and retry.
- // This handles the shutdown case where the staging directory may have been removed(see
- // SPARK-12316 for more details).
- credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.MINUTES)
- } else {
- logInfo(s"Scheduling credentials refresh from HDFS in $remainingTime millis.")
- credentialUpdater.schedule(
- credentialUpdaterRunnable, remainingTime, TimeUnit.MILLISECONDS)
+ }.getOrElse {
+ // Wait for 1 minute to check again if there's no credential file currently
+ 60L * 1000
--- End diff --
`TimeUnit.MINUTE.toMillis(1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62658/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72489363
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in if not explicitly disabled, any plugged-in credential provider wants to be
+ * managed by ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]]
+ * interface and put into resources/META-INF/services to be loaded by ServiceLoader.
+ *
+ * Also each credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+private[yarn] final class ConfigurableCredentialManager (
+ sparkConf: SparkConf, hadoopConf: Configuration) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = {
+ val providers = ServiceLoader.load(classOf[ServiceCredentialProvider],
+ Utils.getContextOrSparkClassLoader).asScala
+
+ // Filter out credentials in which spark.yarn.security.credentials.{service}.enabled is false.
+ providers.filter { p =>
+ sparkConf.getOption(providerEnabledConfig.format(p.serviceName)).map(_.toBoolean)
--- End diff --
Minor but you can move the `toBoolean` transformation to after the `orElse` and save a tiny bit of code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62149 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62149/consoleFull)** for PR 14065 at commit [`60a275f`](https://github.com/apache/spark/commit/60a275f9487eacab8ce8c70d6917d3aebe16d131).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r74104234
--- Diff: docs/running-on-yarn.md ---
@@ -525,16 +524,23 @@ token for the cluster's HDFS filesystem, and potentially for HBase and Hive.
An HBase token will be obtained if HBase is in on classpath, the HBase configuration declares
the application is secure (i.e. `hbase-site.xml` sets `hbase.security.authentication` to `kerberos`),
-and `spark.yarn.security.tokens.hbase.enabled` is not set to `false`.
+and `spark.yarn.security.credentials.hbase.enabled` is not set to `false`.
Similarly, a Hive token will be obtained if Hive is on the classpath, its configuration
includes a URI of the metadata store in `"hive.metastore.uris`, and
-`spark.yarn.security.tokens.hive.enabled` is not set to `false`.
+`spark.yarn.security.credentials.hive.enabled` is not set to `false`.
If an application needs to interact with other secure HDFS clusters, then
the tokens needed to access these clusters must be explicitly requested at
launch time. This is done by listing them in the `spark.yarn.access.namenodes` property.
+Spark supports integrating with other security-aware services through Java Services mechanism (see
+`java.util.ServiceLoader`). To do that, implementations of `org.apache.spark.deploy.yarn.security.ServiceCredentialProvider`
+should be available to Spark by listing their names in the corresponding file in the jar's
+`META-INF/services` directory. These plug-ins can be disabled by setting
+`spark.yarn.security.tokens.{service}.enabled` to `false`, where `{service}` is the name of
+credential provider.
+
```
spark.yarn.access.namenodes hdfs://ireland.example.org:8020/,hdfs://frankfurt.example.org:8020/
--- End diff --
Hmm, it seems like this example should be before the paragraph you're adding.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73418133
--- Diff: docs/running-on-yarn.md ---
@@ -461,15 +461,24 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
</tr>
<tr>
- <td><code>spark.yarn.security.tokens.${service}.enabled</code></td>
+ <td><code>spark.yarn.security.credentials.${service}.enabled</code></td>
<td><code>true</code></td>
<td>
- Controls whether to retrieve delegation tokens for non-HDFS services when security is enabled.
- By default, delegation tokens for all supported services are retrieved when those services are
+ Controls whether to obtain credentials for services when security is enabled.
+ By default, credentials for all supported services are retrieved when those services are
configured, but it's possible to disable that behavior if it somehow conflicts with the
- application being run.
+ application being run. For the details please see
+ [Running in a Secure Cluster](running-on-yarn.html#running-in-a-secure-cluster)
<p/>
- Currently supported services are: <code>hive</code>, <code>hbase</code>
+ Currently supported services are: <code>hdfs</code>, <code>hive</code>, <code>hbase</code>
--- End diff --
This sentence is now a little misleading; should probably either not mention this, or say that these are the built-in services.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61896/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72750534
--- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/security/HDFSCredentialProviderSuite.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.scalatest.{Matchers, PrivateMethodTester}
+
+import org.apache.spark.{SparkConf, SparkException, SparkFunSuite}
+
+class HDFSCredentialProviderSuite
+ extends SparkFunSuite
+ with PrivateMethodTester
+ with Matchers {
+ private val _nnsToAccess = PrivateMethod[Set[Path]]('nnsToAccess)
+ private val _getTokenRenewer = PrivateMethod[String]('getTokenRenewer)
+
+ private def nnsToAccess(
+ hdfsCredentialProvider: HDFSCredentialProvider, sparkConf: SparkConf): Set[Path] = {
+ hdfsCredentialProvider invokePrivate _nnsToAccess(sparkConf)
+ }
+
+ private def getTokenRenewer(
+ hdfsCredentialProvider: HDFSCredentialProvider, conf: Configuration): String = {
+ hdfsCredentialProvider invokePrivate _getTokenRenewer(conf)
+ }
+
+ private var hdfsCredentialProvider: HDFSCredentialProvider = null
+
+ override def beforeAll() {
+ super.beforeAll()
+
+ if (hdfsCredentialProvider == null) {
+ hdfsCredentialProvider = new HDFSCredentialProvider()
+ }
+ }
+
+ override def afterAll() {
+ if (hdfsCredentialProvider != null) {
+ hdfsCredentialProvider = null
+ }
+
+ super.afterAll()
+ }
+
+ test("check access nns empty") {
+ val sparkConf = new SparkConf()
+ .set("spark.yarn.access.namenodes", "")
--- End diff --
these bunch of tests existed in the original code, I just moved it to `HDFSCredentialProviderSuite`. I can delete them if you think it is not so necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62219/consoleFull)** for PR 14065 at commit [`a8141a5`](https://github.com/apache/spark/commit/a8141a54dd87f77c3ce6eb0bccfabacd64841b4b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71046243
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -390,8 +393,22 @@ private[spark] class Client(
// Upload Spark and the application JAR to the remote file system if necessary,
// and add them as local resources to the application master.
val fs = destDir.getFileSystem(hadoopConf)
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + destDir
- YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
+
+ // Merge credentials obtained from registered providers
+ var nearestTimeOfNextRenewal = Long.MaxValue
+ credentialManager.obtainCredentials(hadoopConf, credentials).foreach { t =>
--- End diff --
You don't really need to calculate `nearestTimeOfNextRenewal` if `loginFromKeytab` is false, right?
Also, this all could be:
```
val nearestTimeOfNextRenewal = credentialManager.obtainCredentials(hadoopConf, credentials).flatten.min
```
Just be careful to:
- validate that all returned times are after the current time
- watch out for when the result of `.flatten` is empty
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63349/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62148/consoleFull)** for PR 14065 at commit [`0fbf25b`](https://github.com/apache/spark/commit/0fbf25b51285ea433a3dcf733b92a1d785b3d017).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #63405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63405/consoleFull)** for PR 14065 at commit [`71df428`](https://github.com/apache/spark/commit/71df4281404907e2143f1d23bf2a4ec606ab6ef2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14065
sorry I haven't had time to review fully, taking a quick look I think it would be nice to add at least a blurb to the docs (running on yarn) about how to use this and pointer to scaladocs for ServiceCredentialProvider and the spark.yarn.security.credentials.%s.enabled configs and how things get loaded.
We also need to update docs for spark.yarn.security.credentials.%s.enabled.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69916415
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/token/AMDelegationTokenRenewer.scala ---
@@ -171,10 +174,9 @@ private[yarn] class AMDelegationTokenRenewer(
keytabLoggedInUGI.doAs(new PrivilegedExceptionAction[Void] {
// Get a copy of the credentials
override def run(): Void = {
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + dst
- hadoopUtil.obtainTokensForNamenodes(nns, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHiveMetastore(sparkConf, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHBase(sparkConf, freshHadoopConf, tempCreds)
+ hdfsTokenProvider(sparkConf).setNameNodesToAccess(sparkConf, Set(dst))
+ hdfsTokenProvider(sparkConf).setTokenRenewer(null)
--- End diff --
why are we setting this to null here? is it supposed to indicate that its not supposed to be renewed internally?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71048326
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in, any plugged-in credential provider wants to be managed by
+ * ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]] interface and put
+ * into resources to be loaded by ServiceLoader.
+ *
+ * Also the specific credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+final class ConfigurableCredentialManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = mutable.HashMap[String, ServiceCredentialProvider]()
+
+ // Default crendetial providers that will be loaded automatically, unless specifically disabled.
+ private val defaultCredentialProviders = Map(
--- End diff --
Why do you still need this now that the code is using `ServiceLoader`?
If `ServiceLoader.load` is not loading these providers, something is wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69990872
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/token/ServiceTokenProvider.scala ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.token
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.SparkConf
+
+/**
+ * An interface to provide tokens for service, any service wants to communicate with Spark
+ * through token way needs to implement this interface and register into
+ * [[ConfigurableTokenManager]] through configurations.
+ */
+trait ServiceTokenProvider {
+
+ /**
+ * Name of the ServiceTokenProvider, should be unique. Using this to distinguish different
+ * service.
+ */
+ def serviceName: String
+
+ /**
+ * Used to indicate whether a token is required.
+ */
+ def isTokenRequired(conf: Configuration): Boolean = {
+ UserGroupInformation.isSecurityEnabled
+ }
+
+ /**
+ * Obtain tokens from this service, tokens will be added into Credentials and return as array.
+ */
+ def obtainTokensFromService(
+ sparkConf: SparkConf,
+ serviceConf: Configuration,
--- End diff --
Note the name here is misleading. It won't be the service's conf, but really a `YarnConfiguration`. Note how both Hive and HBase providers have to load their own configuration to be able to see service-specific settings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #61896 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61896/consoleFull)** for PR 14065 at commit [`2b17f2e`](https://github.com/apache/spark/commit/2b17f2e906b2fd67e1c2398fd8ba88b64b797550).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72491288
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -107,8 +110,16 @@ private[spark] class ExecutorDelegationTokenUpdater(
}
}
+ private def getTimeOfNextUpdateFromFileName(credentialsPath: Path): Long = {
+ val name = credentialsPath.getName
--- End diff --
I wonder if using `name.split(SparkHadoopUtil.SPARK_YARN_CREDS_COUNTER_DELIM)` wouldn't make this method easier to read.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62468/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71354857
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -193,10 +196,17 @@ private[yarn] class AMDelegationTokenRenewer(
}
}
val nextSuffix = lastCredentialsFileSuffix + 1
+ // Time of next update should be later than renewal,
+ val timeOfNextUpdate =
+ (timeOfNextRenewal - System.currentTimeMillis()) * 1.1 + System.currentTimeMillis()
--- End diff --
I'm a little concerned with the 1.1 here. If renewal is every 24 hours, 1.1 times that is another 144 minutes past renewal, this is fine for hdfs which return 75% of renewal but its still quite long, it was 70 minutes before with 75% and 80% but I'm a little concerned with possibly other services and what the user could return. So following my comment above perhaps we should move the logic from hdfs credentials into here and do like the 75%/80% (or 85%) again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63516/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71049066
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in, any plugged-in credential provider wants to be managed by
+ * ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]] interface and put
+ * into resources to be loaded by ServiceLoader.
+ *
+ * Also the specific credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+final class ConfigurableCredentialManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = mutable.HashMap[String, ServiceCredentialProvider]()
+
+ // Default crendetial providers that will be loaded automatically, unless specifically disabled.
+ private val defaultCredentialProviders = Map(
+ "hdfs" -> "org.apache.spark.deploy.yarn.security.HDFSCredentialProvider",
+ "hive" -> "org.apache.spark.deploy.yarn.security.HiveCredentialProvider",
+ "hbase" -> "org.apache.spark.deploy.yarn.security.HBaseCredentialProvider"
+ )
+
+ // AMDelegationTokenRenewer, this will only be create and started in the AM
+ private var _delegationTokenRenewer: AMDelegationTokenRenewer = null
+
+ // ExecutorDelegationTokenUpdater, this will only be created and started in the driver and
+ // executor side.
+ private var _delegationTokenUpdater: ExecutorDelegationTokenUpdater = null
+
+ def initialize(): Unit = {
+ val providers = ServiceLoader.load(classOf[ServiceCredentialProvider],
+ Utils.getContextOrSparkClassLoader).asScala
+
+ // Filter out credentials in which spark.yarn.security.credentials.{service}.enabled is false.
+ providers.filter { p =>
+ sparkConf.getOption(providerEnabledConfig.format(p.serviceName))
+ .orElse {
+ sparkConf.getOption(deprecatedProviderEnabledConfig.format(p.serviceName)).map { c =>
+ logWarning(s"${deprecatedProviderEnabledConfig.format(p.serviceName)} is deprecated, " +
+ s"using ${providerEnabledConfig.format(p.serviceName)} instead")
+ c
+ }
+ }
+ .getOrElse(defaultCredentialProviders.keySet.find(_ == p.serviceName).isDefined.toString)
+ .toBoolean
+ }.foreach {
+ p => credentialProviders(p.serviceName) = p
+ }
+ }
+
+ /**
+ * Get credential provider for the specified service.
+ */
+ def getServiceCredentialProvider(service: String): Option[ServiceCredentialProvider] = {
+ credentialProviders.get(service)
+ }
+
+ /**
+ * Obtain credentials for specified service.
+ * @return time of next renewal if this service credential is renewable.
+ */
+ def obtainCredentialsFromService(
+ service: String,
+ hadoopConf: Configuration,
+ creds: Credentials): Option[Long] = {
+ getServiceCredentialProvider(service).flatMap { provider =>
+ if (provider.isCredentialRequired(hadoopConf)) {
+ provider.obtainCredentials(hadoopConf, creds)
+ } else {
+ None
+ }
+ }
+ }
+
+ /**
+ * Obtain credentials from all the registered providers.
+ * @return Array of time of next renewal if credential can be renewed.
+ * Otherwise will return None instead.
+ */
+ def obtainCredentials(hadoopConf: Configuration, creds: Credentials): Array[Option[Long]] = {
+ val timeOfNextRenewals = mutable.ArrayBuffer[Option[Long]]()
+ credentialProviders.values.foreach { provider =>
+ if (provider.isCredentialRequired(hadoopConf)) {
+ timeOfNextRenewals += provider.obtainCredentials(hadoopConf, creds)
+ } else {
+ logDebug(s"Service ${provider.serviceName} does not require a token." +
+ s" Check your configuration to see if security is disabled or not.")
+ }
+ }
+
+ timeOfNextRenewals.toArray
+ }
+
+ def delegationTokenRenewer(hadoopConf: Configuration): AMDelegationTokenRenewer = synchronized {
--- End diff --
You have two options here:
- pass the hadoop configuration in the manager's constructor, and lazily instantiate the token renewer (and updated) - a `lazy val` would suffice for that
- just return a new one for the given hadoop configuration and let the caller handle its lifecycle
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71047265
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -85,15 +90,14 @@ private[yarn] class AMDelegationTokenRenewer(
* will synchronously create new ones.
*/
def scheduleRenewal(runnable: Runnable): Unit = {
- val credentials = UserGroupInformation.getCurrentUser.getCredentials
- val renewalInterval = hadoopUtil.getTimeFromNowToRenewal(sparkConf, 0.75, credentials)
// Run now!
- if (renewalInterval <= 0) {
+ val leftTime = timeOfNextRenewal - System.currentTimeMillis()
--- End diff --
s/leftTime/remainingTime
The previous name (renewalInterval) was also fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72489576
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in if not explicitly disabled, any plugged-in credential provider wants to be
+ * managed by ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]]
+ * interface and put into resources/META-INF/services to be loaded by ServiceLoader.
+ *
+ * Also each credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+private[yarn] final class ConfigurableCredentialManager (
+ sparkConf: SparkConf, hadoopConf: Configuration) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = {
+ val providers = ServiceLoader.load(classOf[ServiceCredentialProvider],
+ Utils.getContextOrSparkClassLoader).asScala
+
+ // Filter out credentials in which spark.yarn.security.credentials.{service}.enabled is false.
+ providers.filter { p =>
+ sparkConf.getOption(providerEnabledConfig.format(p.serviceName)).map(_.toBoolean)
+ .orElse {
+ sparkConf.getOption(deprecatedProviderEnabledConfig.format(p.serviceName)).map { c =>
+ logWarning(s"${deprecatedProviderEnabledConfig.format(p.serviceName)} is deprecated, " +
+ s"using ${providerEnabledConfig.format(p.serviceName)} instead")
+ c.toBoolean
+ }
+ }.getOrElse(true)
+ }.map { p => (p.serviceName, p) }.toMap
+ }
+
+ /**
+ * Get credential provider for the specified service.
+ */
+ def getServiceCredentialProvider(service: String): Option[ServiceCredentialProvider] = {
+ credentialProviders.get(service)
+ }
+
+ /**
+ * Obtain credentials from all the registered providers.
+ * @return nearest time of next renewal, Long.MaxValue if all the credentials aren't renewable,
+ * otherwise the nearest renewal time of any credentials will be returned.
+ */
+ def obtainCredentials(hadoopConf: Configuration, creds: Credentials): Long = {
--- End diff --
Do you need `hadoopConf` here again? It's already provided in the class's constructor.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #63349 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63349/consoleFull)** for PR 14065 at commit [`32c16c0`](https://github.com/apache/spark/commit/32c16c06bd54eb26abc97c4fe047dd2a9af0c4ad).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14065
what all testing have you done on this? Ran on actual cluster with secure hive, hbase, hdfs?
Test out tokens and renewal intervals to make sure things propogated?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #63349 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63349/consoleFull)** for PR 14065 at commit [`32c16c0`](https://github.com/apache/spark/commit/32c16c06bd54eb26abc97c4fe047dd2a9af0c4ad).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71348341
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -390,8 +391,21 @@ private[spark] class Client(
// Upload Spark and the application JAR to the remote file system if necessary,
// and add them as local resources to the application master.
val fs = destDir.getFileSystem(hadoopConf)
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + destDir
- YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
+
+ // Merge credentials obtained from registered providers
+ val nearestTimeOfNextRenewal = credentialManager.obtainCredentials(hadoopConf, credentials)
+
+ // If we use principal and keytab to login, also credentials can be renewed some time
+ // after current time, we should pass the next renewal and updating time to delegation token
+ // renewer and updater.
+ if (loginFromKeytab && nearestTimeOfNextRenewal > System.currentTimeMillis() &&
+ nearestTimeOfNextRenewal != Long.MaxValue) {
+ sparkConf.set(CREDENTIALS_RENEWAL_TIME, nearestTimeOfNextRenewal)
+ val nearestTimeOfNextUpdate =
+ (nearestTimeOfNextRenewal - System.currentTimeMillis()) * 1.1 + System.currentTimeMillis()
--- End diff --
what is the 1.1 here? just some time slightly in the future for executors to fetch? Note that if they really return the time to renew from obtainTokens this could be to late. With hdfs with return 75% of renew time. We either need to make sure to document the user should do something similar or we need to handle here. Its probably safer to handle here. The only downside would be if they make renewal interval really really small. We should probably put warning/documentation in around obtain tokens about that too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73419090
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMCredentialRenewer.scala ---
@@ -27,39 +27,42 @@ import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.SparkConf
import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
import org.apache.spark.deploy.yarn.config._
import org.apache.spark.internal.Logging
import org.apache.spark.internal.config._
import org.apache.spark.util.ThreadUtils
-/*
+/**
* The following methods are primarily meant to make sure long-running apps like Spark
- * Streaming apps can run without interruption while writing to secure HDFS. The
- * scheduleLoginFromKeytab method is called on the driver when the
- * CoarseGrainedScheduledBackend starts up. This method wakes up a thread that logs into the KDC
- * once 75% of the renewal interval of the original delegation tokens used for the container
- * has elapsed. It then creates new delegation tokens and writes them to HDFS in a
+ * Streaming apps can run without interruption while accessing security services. The
+ * scheduleLoginFromKeytab method is called on the AM to get the new credentials.
+ * This method wakes up a thread that logs into the KDC
+ * once 75% of the renewal interval of the original credentials used for the container
+ * has elapsed. It then obtain new credentials and writes them to HDFS in a
--- End diff --
nit: obtains
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71111198
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -193,8 +200,14 @@ private[yarn] class AMDelegationTokenRenewer(
}
}
val nextSuffix = lastCredentialsFileSuffix + 1
+ // Time of next update should be late than renewal,
+ // which is around new token issue date + 0.8 * renew interval
+ val timeOfNextUpdate =
--- End diff --
Not sure how this part could be consolidate into `obtainCredentials` method, the return value of `obtainCredentials` is time of next renewal, here it is time of next updating, which should be slightly later than next renewal.
So in shortly IIUC if we also want to consolidate this code, so we should change the return value `obtainCredentials` to `(timeOfNextRenewal, timeOfNextUpdate)`, what do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69989547
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/token/ConfigurableTokenManager.scala ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.token
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A [[ConfigurableTokenManager]] to manage all the token providers register in this class. Also
+ * it provides other modules the functionality to obtain tokens, get token renewal interval and
+ * calculate the time length till next renewal.
+ *
+ * By default ConfigurableTokenManager has 3 built-in token providers, HDFSTokenProvider,
+ * HiveTokenProvider and HBaseTokenProvider, and this 3 token providers can also be controlled
+ * by configuration spark.yarn.security.tokens.{service}.enabled, if it is set to false, this
+ * provider will not be loaded.
+ *
+ * For other token providers which need to be loaded in should:
+ * 1. Implement [[ServiceTokenProvider]] or [[ServiceTokenRenewable]] if token renewal is
+ * required for this service.
+ * 2. set spark.yarn.security.tokens.{service}.enabled to true
+ * 3. Specify the class name through spark.yarn.security.tokens.{service}.class
+ *
+ */
+final class ConfigurableTokenManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val tokenProviderEnabledConfig = "spark\\.yarn\\.security\\.tokens\\.(.+)\\.enabled".r
+ private val tokenProviderClsConfig = "spark.yarn.security.tokens.%s.class"
+
+ // Maintain all the registered token providers
+ private val tokenProviders = mutable.HashMap[String, ServiceTokenProvider]()
+
+ private val defaultTokenProviders = Map(
+ "hdfs" -> "org.apache.spark.deploy.yarn.token.HDFSTokenProvider",
+ "hive" -> "org.apache.spark.deploy.yarn.token.HiveTokenProvider",
+ "hbase" -> "org.apache.spark.deploy.yarn.token.HBaseTokenProvider"
+ )
+
+ // AMDelegationTokenRenewer, this will only be create and started in the AM
+ private var _delegationTokenRenewer: AMDelegationTokenRenewer = null
+
+ // ExecutorDelegationTokenUpdater, this will only be created and started in the driver and
+ // executor side.
+ private var _delegationTokenUpdater: ExecutorDelegationTokenUpdater = null
+
+ def initialize(): Unit = {
--- End diff --
A lot of this method would go away by using `java.util.ServiceLoader`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69917965
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/token/HDFSTokenProvider.scala ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.token
+
+import java.io.{ByteArrayInputStream, DataInputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier
+import org.apache.hadoop.mapred.Master
+import org.apache.hadoop.security.Credentials
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.deploy.yarn.config._
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+
+private[yarn] class HDFSTokenProvider
+ extends ServiceTokenProvider with ServiceTokenRenewable with Logging {
+
+ private var nnsToAccess: Set[Path] = Set.empty
+ private var tokenRenewer: Option[String] = None
+
+ override val serviceName: String = "hdfs"
+
+ override def obtainTokensFromService(
+ sparkConf: SparkConf,
+ serviceConf: Configuration,
+ creds: Credentials)
+ : Array[Token[_]] = {
+ val tokens = ArrayBuffer[Token[_]]()
+ val renewer = tokenRenewer.getOrElse(getTokenRenewer(serviceConf))
+ nnsToAccess.foreach { dst =>
+ val dstFs = dst.getFileSystem(serviceConf)
+ logInfo("getting token for namenode: " + dst)
+ tokens ++= dstFs.addDelegationTokens(renewer, creds)
+ }
+
+ tokens.toArray
+ }
+
+ override def getTokenRenewalInterval(sparkConf: SparkConf, serviceConf: Configuration): Long = {
+ // We cannot use the tokens generated above since those have renewer yarn. Trying to renew
--- End diff --
update comment to be more specific then "above" since this has moved
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73931240
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -53,50 +53,58 @@ private[spark] class CredentialUpdater(
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
- @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
+ /** Start the credentail updater thread periodically to get the credentials */
+ def bootstrap(): Unit = {
+ val bootstrapTime = sparkConf.get(CREDENTIALS_RENEWAL_TIME)
+ val remainingTime = bootstrapTime - System.currentTimeMillis()
+ if (remainingTime <= 0) {
+ // We just checked for new credentials but none were there, wait a minute and retry.
+ // This handles the shutdown case where the staging directory may have been removed(see
+ // SPARK-12316 for more details).
+ credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.MINUTES)
+ } else {
+ logInfo(s"Scheduling credentials refresh from HDFS in $remainingTime millis.")
+ credentialUpdater.schedule(credentialUpdaterRunnable, remainingTime, TimeUnit.MILLISECONDS)
+ }
+ }
- def updateCredentialsIfRequired(): Unit = {
- try {
+ private def updateCredentialsIfRequired(): Unit = {
+ val timeToNextUpdate = try {
val credentialsFilePath = new Path(credentialsFile)
val remoteFs = FileSystem.get(freshHadoopConf)
SparkHadoopUtil.get.listFilesSorted(
remoteFs, credentialsFilePath.getParent,
credentialsFilePath.getName, SparkHadoopUtil.SPARK_YARN_CREDS_TEMP_EXTENSION)
- .lastOption.foreach { credentialsStatus =>
+ .lastOption.map { credentialsStatus =>
val suffix = SparkHadoopUtil.get.getSuffixForCredentialsPath(credentialsStatus.getPath)
if (suffix > lastCredentialsFileSuffix) {
logInfo("Reading new credentials from " + credentialsStatus.getPath)
val newCredentials = getCredentialsFromHDFSFile(remoteFs, credentialsStatus.getPath)
lastCredentialsFileSuffix = suffix
UserGroupInformation.getCurrentUser.addCredentials(newCredentials)
logInfo("Credentials updated from credentials file.")
- timeOfNextUpdate = getTimeOfNextUpdateFromFileName(credentialsStatus.getPath)
+
+ val remainingTime =
+ getTimeOfNextUpdateFromFileName(credentialsStatus.getPath) - System.currentTimeMillis()
+ if (remainingTime <= 0) 60L * 1000 else remainingTime
} else {
- // Check every hour to see if new credentials arrived.
- logInfo("Updated credentials were expected, but the AM has not updated the " +
- "credentials yet, will check again in an hour.")
- credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.HOURS)
- return
+ // If current credential file is elder than expected, sleep 1 hour and check again.
+ 3600L * 1000
--- End diff --
`TimeUnit.HOUR.toMillis(1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #61912 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61912/consoleFull)** for PR 14065 at commit [`c9d9ed0`](https://github.com/apache/spark/commit/c9d9ed0cd0aef6b8017c04e635f27ef123a48887).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62506/consoleFull)** for PR 14065 at commit [`b8eeb28`](https://github.com/apache/spark/commit/b8eeb28b141b678cf4ccace36564f24536758132).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #63200 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63200/consoleFull)** for PR 14065 at commit [`127d85e`](https://github.com/apache/spark/commit/127d85ed54f057581a35c88fc7f85e1b8e13de38).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14065
Thanks a lot @tgravescs and @vanzin for your suggestions, I will change the codes accordingly, greatly appreciate your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62658/consoleFull)** for PR 14065 at commit [`bd04a2b`](https://github.com/apache/spark/commit/bd04a2b9590e59800e1206103d32779d190056ae).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73419026
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMCredentialRenewer.scala ---
@@ -27,39 +27,42 @@ import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.SparkConf
import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
import org.apache.spark.deploy.yarn.config._
import org.apache.spark.internal.Logging
import org.apache.spark.internal.config._
import org.apache.spark.util.ThreadUtils
-/*
+/**
* The following methods are primarily meant to make sure long-running apps like Spark
- * Streaming apps can run without interruption while writing to secure HDFS. The
- * scheduleLoginFromKeytab method is called on the driver when the
- * CoarseGrainedScheduledBackend starts up. This method wakes up a thread that logs into the KDC
- * once 75% of the renewal interval of the original delegation tokens used for the container
- * has elapsed. It then creates new delegation tokens and writes them to HDFS in a
+ * Streaming apps can run without interruption while accessing security services. The
--- End diff --
nit: s/security/secured
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71048595
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in, any plugged-in credential provider wants to be managed by
+ * ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]] interface and put
+ * into resources to be loaded by ServiceLoader.
+ *
+ * Also the specific credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+final class ConfigurableCredentialManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = mutable.HashMap[String, ServiceCredentialProvider]()
+
+ // Default crendetial providers that will be loaded automatically, unless specifically disabled.
+ private val defaultCredentialProviders = Map(
+ "hdfs" -> "org.apache.spark.deploy.yarn.security.HDFSCredentialProvider",
+ "hive" -> "org.apache.spark.deploy.yarn.security.HiveCredentialProvider",
+ "hbase" -> "org.apache.spark.deploy.yarn.security.HBaseCredentialProvider"
+ )
+
+ // AMDelegationTokenRenewer, this will only be create and started in the AM
+ private var _delegationTokenRenewer: AMDelegationTokenRenewer = null
+
+ // ExecutorDelegationTokenUpdater, this will only be created and started in the driver and
+ // executor side.
+ private var _delegationTokenUpdater: ExecutorDelegationTokenUpdater = null
+
+ def initialize(): Unit = {
+ val providers = ServiceLoader.load(classOf[ServiceCredentialProvider],
+ Utils.getContextOrSparkClassLoader).asScala
+
+ // Filter out credentials in which spark.yarn.security.credentials.{service}.enabled is false.
+ providers.filter { p =>
+ sparkConf.getOption(providerEnabledConfig.format(p.serviceName))
+ .orElse {
+ sparkConf.getOption(deprecatedProviderEnabledConfig.format(p.serviceName)).map { c =>
+ logWarning(s"${deprecatedProviderEnabledConfig.format(p.serviceName)} is deprecated, " +
+ s"using ${providerEnabledConfig.format(p.serviceName)} instead")
+ c
+ }
+ }
+ .getOrElse(defaultCredentialProviders.keySet.find(_ == p.serviceName).isDefined.toString)
--- End diff --
What's the purpose of this line? Seems like you want to enable internal providers and disable 3rd-party ones by default? Why?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72493667
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HDFSCredentialProvider.scala ---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.io.{ByteArrayInputStream, DataInputStream}
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier
+import org.apache.hadoop.mapred.Master
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.deploy.yarn.config._
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+
+private[security] class HDFSCredentialProvider extends ServiceCredentialProvider with Logging {
+
+ private val sparkConf = new SparkConf()
+
+ // NameNode to access, used to get tokens from different FileSystems
+ private val unionNNsToAccess = {
+ val tmpConf = SparkHadoopUtil.get.newConfiguration(sparkConf)
+ val stageDir = sparkConf.get(STAGING_DIR).map(new Path(_))
+ .getOrElse(FileSystem.get(tmpConf).getHomeDirectory)
+ nnsToAccess(sparkConf) + stageDir
+ }
+
+ // Token renewal interval, this value will be set in the first call,
+ // if None means no token renewer specified, so cannot get token renewal interval.
+ private var tokenRenewalInterval: Option[Long] = null
+
+ override val serviceName: String = "hdfs"
+
+ override def obtainCredentials(hadoopConf: Configuration, creds: Credentials): Option[Long] = {
+ // Obtain tokens from HDFS and add into Credentials
+ unionNNsToAccess.foreach { dst =>
+ val dstFs = dst.getFileSystem(hadoopConf)
+ logInfo("getting token for namenode: " + dst)
+ dstFs.addDelegationTokens(getTokenRenewer(hadoopConf), creds)
+ }
+
+ // Get the token renewal interval if it is not set. It will only be called once.
+ if (tokenRenewalInterval == null) {
+ tokenRenewalInterval = getTokenRenewalInterval(hadoopConf)
+ }
+
+ // Get the time of next renewal.
+ tokenRenewalInterval.map { interval =>
+ creds.getAllTokens.asScala
+ .filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
+ .map { t =>
+ val identifier = new DelegationTokenIdentifier()
+ identifier.readFields(new DataInputStream(new ByteArrayInputStream(t.getIdentifier)))
+ identifier.getIssueDate + interval
+ }.foldLeft(0L)(math.max)
+ }
+ }
+
+ private def getTokenRenewalInterval(hadoopConf: Configuration): Option[Long] = {
+ // We cannot use the tokens generated with renewer yarn. Trying to renew
+ // those will fail with an access control issue. So create new tokens with the logged in
+ // user as renewer.
+ sparkConf.get(PRINCIPAL).map { renewer =>
+ val creds = new Credentials()
+ unionNNsToAccess.foreach { dst =>
+ val dstFs = dst.getFileSystem(hadoopConf)
+ dstFs.addDelegationTokens(renewer, creds)
+ }
+ val t = creds.getAllTokens.asScala
+ .filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
+ .head
+ val newExpiration = t.renew(hadoopConf)
+ val identifier = new DelegationTokenIdentifier()
+ identifier.readFields(new DataInputStream(new ByteArrayInputStream(t.getIdentifier)))
+ val interval = newExpiration - identifier.getIssueDate
+ logInfo(s"Renewal Interval is $interval")
+ interval
+ }
+ }
+
+ private def getTokenRenewer(conf: Configuration): String = {
+ val delegTokenRenewer = Master.getMasterPrincipal(conf)
+ logDebug("delegation token renewer is: " + delegTokenRenewer)
+ if (delegTokenRenewer == null || delegTokenRenewer.length() == 0) {
+ val errorMessage = "Can't get Master Kerberos principal for use as renewer"
+ logError(errorMessage)
+ throw new SparkException(errorMessage)
+ }
+
+ delegTokenRenewer
+ }
+
+ private def nnsToAccess(sparkConf: SparkConf): Set[Path] = {
--- End diff --
(I see it's used in the tests too, let me read that code first.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71363393
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ServiceCredentialProvider.scala ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * ::DeveloperApi::
+ * A credential provider for this service, any service want to access by spark through security
+ * way must implement this interface.
+ */
+@DeveloperApi
--- End diff --
I dont think this should be a developerAPI, we want people to extends this and not worry about it breaking. Unless your intention is to make it develop for now until we get a little run time on it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69992687
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/token/ServiceTokenProvider.scala ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.token
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.SparkConf
+
+/**
+ * An interface to provide tokens for service, any service wants to communicate with Spark
+ * through token way needs to implement this interface and register into
+ * [[ConfigurableTokenManager]] through configurations.
+ */
+trait ServiceTokenProvider {
+
+ /**
+ * Name of the ServiceTokenProvider, should be unique. Using this to distinguish different
+ * service.
+ */
+ def serviceName: String
+
+ /**
+ * Used to indicate whether a token is required.
+ */
+ def isTokenRequired(conf: Configuration): Boolean = {
+ UserGroupInformation.isSecurityEnabled
+ }
+
+ /**
+ * Obtain tokens from this service, tokens will be added into Credentials and return as array.
+ */
+ def obtainTokensFromService(
--- End diff --
If you follow Tom's suggestion and turn this into a generic "obtainCredentials" method, then you could potentially merge it with `getTimeFromNowToRenewal` too.
e.g. the provider is responsible for adding the tokens to the `Credentials` object, and it returns when it should be called again to renew those tokens (or obtain new credentials). One less method in the interface!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #63405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63405/consoleFull)** for PR 14065 at commit [`71df428`](https://github.com/apache/spark/commit/71df4281404907e2143f1d23bf2a4ec606ab6ef2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62219 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62219/consoleFull)** for PR 14065 at commit [`a8141a5`](https://github.com/apache/spark/commit/a8141a54dd87f77c3ce6eb0bccfabacd64841b4b).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71343103
--- Diff: docs/running-on-yarn.md ---
@@ -461,15 +461,28 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
</tr>
<tr>
- <td><code>spark.yarn.security.tokens.${service}.enabled</code></td>
+ <td><code>spark.yarn.security.credentials.${service}.enabled</code></td>
<td><code>true</code></td>
<td>
- Controls whether to retrieve delegation tokens for non-HDFS services when security is enabled.
- By default, delegation tokens for all supported services are retrieved when those services are
+ Controls whether to obtain credentials for services when security is enabled.
+ By default, credentials for all supported services are retrieved when those services are
configured, but it's possible to disable that behavior if it somehow conflicts with the
application being run.
<p/>
- Currently supported services are: <code>hive</code>, <code>hbase</code>
+ Currently supported services are: <code>hdfs</code>, <code>hive</code>, <code>hbase</code>
+ <p/>
+ For other 3rd party services need to be communicate with Spark through security way should implement
--- End diff --
a few grammar things here and I think we should separate this out into the "Running in a Secure Cluster" section.
It also has a few places that need updated from tokens. to credentials.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71121047
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in, any plugged-in credential provider wants to be managed by
+ * ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]] interface and put
+ * into resources to be loaded by ServiceLoader.
+ *
+ * Also the specific credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+final class ConfigurableCredentialManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = mutable.HashMap[String, ServiceCredentialProvider]()
+
+ // Default crendetial providers that will be loaded automatically, unless specifically disabled.
+ private val defaultCredentialProviders = Map(
+ "hdfs" -> "org.apache.spark.deploy.yarn.security.HDFSCredentialProvider",
+ "hive" -> "org.apache.spark.deploy.yarn.security.HiveCredentialProvider",
+ "hbase" -> "org.apache.spark.deploy.yarn.security.HBaseCredentialProvider"
+ )
+
+ // AMDelegationTokenRenewer, this will only be create and started in the AM
+ private var _delegationTokenRenewer: AMDelegationTokenRenewer = null
+
+ // ExecutorDelegationTokenUpdater, this will only be created and started in the driver and
+ // executor side.
+ private var _delegationTokenUpdater: ExecutorDelegationTokenUpdater = null
+
+ def initialize(): Unit = {
+ val providers = ServiceLoader.load(classOf[ServiceCredentialProvider],
+ Utils.getContextOrSparkClassLoader).asScala
+
+ // Filter out credentials in which spark.yarn.security.credentials.{service}.enabled is false.
+ providers.filter { p =>
+ sparkConf.getOption(providerEnabledConfig.format(p.serviceName))
+ .orElse {
+ sparkConf.getOption(deprecatedProviderEnabledConfig.format(p.serviceName)).map { c =>
+ logWarning(s"${deprecatedProviderEnabledConfig.format(p.serviceName)} is deprecated, " +
+ s"using ${providerEnabledConfig.format(p.serviceName)} instead")
+ c
+ }
+ }
+ .getOrElse(defaultCredentialProviders.keySet.find(_ == p.serviceName).isDefined.toString)
--- End diff --
This means that by default hdfs, hive and hbase credential providers will be loaded unless explicitly disabled by configuration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14065
@vazin, looks like I missed that comment, I will address that today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14065
I took a quick look through.
It might be nice to think about how we could handle other credentials.
For instance Apache Kafka currently doesn't have tokens so you need keytab or TGT and jaas conf file. Yes they are adding tokens but in in the mean time how does that work. Are there other services similar to that.
Can we handle things other then Tokens? it does appear that I could implement my own ServiceTokenProvider that goes off to really any service and I can put things into the Credentials object as Token or as a Secret so perhaps we are covered here. But perhaps that means we should rename things to be obtainCredentials rather then obtainTokens.
Are there specific services you were thinking about here? We could atleast use those as examples to make sure interface fits those.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71481579
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ExecutorDelegationTokenUpdater.scala ---
@@ -107,6 +110,14 @@ private[spark] class ExecutorDelegationTokenUpdater(
}
}
+ private def getTimeOfNextUpdateFromFileName(credentialsPath: Path): Long = {
+ val name = credentialsPath.getName
+ val index = name.lastIndexOf(SparkHadoopUtil.SPARK_YARN_CREDS_COUNTER_DELIM)
+ val slice = name.substring(0, index)
+ val last2index = slice.lastIndexOf(SparkHadoopUtil.SPARK_YARN_CREDS_COUNTER_DELIM)
+ name.substring(last2index).toLong
--- End diff --
Sorry about it, I will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71358746
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -171,10 +174,10 @@ private[yarn] class AMDelegationTokenRenewer(
keytabLoggedInUGI.doAs(new PrivilegedExceptionAction[Void] {
// Get a copy of the credentials
override def run(): Void = {
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + dst
- hadoopUtil.obtainTokensForNamenodes(nns, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHiveMetastore(sparkConf, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHBase(sparkConf, freshHadoopConf, tempCreds)
+ val nearestNextTime = credentialManager.obtainCredentials(freshHadoopConf, tempCreds)
+ require(nearestNextTime > System.currentTimeMillis(),
--- End diff --
do we really want this to be a require? This is going to throw an exception. The exception is caught in driverTokenRenewerRunnable and then logged but then sets a timer to call again in an hour. We don't want to wait an hour if renewal is < current time we want it to renew right now so if we just set timeOfNextRenewal and return it would do that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14065
all my original comments were addressed and I won't have time to do another review until next week so I'm good with it if you are.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72492707
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HiveCredentialProvider.scala ---
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import scala.reflect.runtime.universe
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier
+import org.apache.hadoop.io.Text
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+private[security] class HiveCredentialProvider extends ServiceCredentialProvider with Logging {
+
+ override def serviceName: String = "hive"
+
+ override def obtainCredentials(hadoopConf: Configuration, creds: Credentials): Option[Long] = {
+ val mirror = universe.runtimeMirror(Utils.getContextOrSparkClassLoader)
+
+ // the hive configuration class is a subclass of Hadoop Configuration, so can be cast down
+ // to a Configuration and used without reflection
+ val hiveConfClass = mirror.classLoader.loadClass("org.apache.hadoop.hive.conf.HiveConf")
+ // using the (Configuration, Class) constructor allows the current configuration to be included
+ // in the hive config.
+ val ctor = hiveConfClass.getDeclaredConstructor(classOf[Configuration],
+ classOf[Object].getClass)
+ val hiveConf = ctor.newInstance(hadoopConf, hiveConfClass).asInstanceOf[Configuration]
+ val metastoreUri = hiveConf.getTrimmed("hive.metastore.uris", "")
+
+ // Check for local metastore
+ if (metastoreUri.nonEmpty) {
--- End diff --
Should some of the code before this condition be in `isCredentialRequired`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72747890
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in if not explicitly disabled, any plugged-in credential provider wants to be
+ * managed by ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]]
+ * interface and put into resources/META-INF/services to be loaded by ServiceLoader.
+ *
+ * Also each credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+private[yarn] final class ConfigurableCredentialManager (
+ sparkConf: SparkConf, hadoopConf: Configuration) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = {
+ val providers = ServiceLoader.load(classOf[ServiceCredentialProvider],
+ Utils.getContextOrSparkClassLoader).asScala
+
+ // Filter out credentials in which spark.yarn.security.credentials.{service}.enabled is false.
+ providers.filter { p =>
+ sparkConf.getOption(providerEnabledConfig.format(p.serviceName)).map(_.toBoolean)
+ .orElse {
+ sparkConf.getOption(deprecatedProviderEnabledConfig.format(p.serviceName)).map { c =>
+ logWarning(s"${deprecatedProviderEnabledConfig.format(p.serviceName)} is deprecated, " +
+ s"using ${providerEnabledConfig.format(p.serviceName)} instead")
+ c.toBoolean
+ }
+ }.getOrElse(true)
+ }.map { p => (p.serviceName, p) }.toMap
+ }
+
+ /**
+ * Get credential provider for the specified service.
+ */
+ def getServiceCredentialProvider(service: String): Option[ServiceCredentialProvider] = {
+ credentialProviders.get(service)
+ }
+
+ /**
+ * Obtain credentials from all the registered providers.
+ * @return nearest time of next renewal, Long.MaxValue if all the credentials aren't renewable,
+ * otherwise the nearest renewal time of any credentials will be returned.
+ */
+ def obtainCredentials(hadoopConf: Configuration, creds: Credentials): Long = {
--- End diff --
Yes, in `AMCredentialRenewer` it uses a `freshHadoopConf` to obtain credentials instead of the one passed from constructor. So here I added this parameter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69989897
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/token/ConfigurableTokenManager.scala ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.token
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A [[ConfigurableTokenManager]] to manage all the token providers register in this class. Also
+ * it provides other modules the functionality to obtain tokens, get token renewal interval and
+ * calculate the time length till next renewal.
+ *
+ * By default ConfigurableTokenManager has 3 built-in token providers, HDFSTokenProvider,
+ * HiveTokenProvider and HBaseTokenProvider, and this 3 token providers can also be controlled
+ * by configuration spark.yarn.security.tokens.{service}.enabled, if it is set to false, this
+ * provider will not be loaded.
+ *
+ * For other token providers which need to be loaded in should:
+ * 1. Implement [[ServiceTokenProvider]] or [[ServiceTokenRenewable]] if token renewal is
+ * required for this service.
+ * 2. set spark.yarn.security.tokens.{service}.enabled to true
+ * 3. Specify the class name through spark.yarn.security.tokens.{service}.class
+ *
+ */
+final class ConfigurableTokenManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val tokenProviderEnabledConfig = "spark\\.yarn\\.security\\.tokens\\.(.+)\\.enabled".r
+ private val tokenProviderClsConfig = "spark.yarn.security.tokens.%s.class"
+
+ // Maintain all the registered token providers
+ private val tokenProviders = mutable.HashMap[String, ServiceTokenProvider]()
+
+ private val defaultTokenProviders = Map(
+ "hdfs" -> "org.apache.spark.deploy.yarn.token.HDFSTokenProvider",
+ "hive" -> "org.apache.spark.deploy.yarn.token.HiveTokenProvider",
+ "hbase" -> "org.apache.spark.deploy.yarn.token.HBaseTokenProvider"
+ )
+
+ // AMDelegationTokenRenewer, this will only be create and started in the AM
+ private var _delegationTokenRenewer: AMDelegationTokenRenewer = null
+
+ // ExecutorDelegationTokenUpdater, this will only be created and started in the driver and
+ // executor side.
+ private var _delegationTokenUpdater: ExecutorDelegationTokenUpdater = null
+
+ def initialize(): Unit = {
+ // Copy SparkConf and add default enabled token provider configurations to SparkConf.
+ val clonedConf = sparkConf.clone
+ defaultTokenProviders.keys.foreach { key =>
+ clonedConf.setIfMissing(s"spark.yarn.security.tokens.$key.enabled", "true")
+ }
+
+ // Instantialize all the service token providers according to the configurations.
+ clonedConf.getAll.filter { case (key, value) =>
+ if (tokenProviderEnabledConfig.findPrefixOf(key).isDefined) {
+ value.toBoolean
+ } else {
+ false
+ }
+ }.map { case (key, _) =>
+ val tokenProviderEnabledConfig(service) = key
+ val cls = sparkConf.getOption(tokenProviderClsConfig.format(service))
+ .orElse(defaultTokenProviders.get(service))
+ (service, cls)
+ }.foreach { case (service, cls) =>
+ if (cls.isDefined) {
+ try {
+ val tokenProvider =
+ Utils.classForName(cls.get).newInstance().asInstanceOf[ServiceTokenProvider]
+ tokenProviders += (service -> tokenProvider)
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Fail to instantiate class ${cls.get}", e)
+ }
+ }
+ }
+ }
+
+ /**
+ * Get service token provider by name.
+ */
+ def getServiceTokenProvider(service: String): Option[ServiceTokenProvider] = {
+ tokenProviders.get(service)
+ }
+
+ /**
+ * Obtain tokens from all the token providers and add into credentials, also return as an array.
+ */
+ def obtainTokens(conf: Configuration, creds: Credentials): Array[Token[_]] = {
+ val tokenBuf = mutable.ArrayBuffer[Token[_]]()
+ tokenProviders.values.foreach { provider =>
--- End diff --
minor: you can use `flatMap` instead of a mutable buffer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73931091
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -53,50 +53,58 @@ private[spark] class CredentialUpdater(
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
- @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
+ /** Start the credentail updater thread periodically to get the credentials */
+ def bootstrap(): Unit = {
+ val bootstrapTime = sparkConf.get(CREDENTIALS_RENEWAL_TIME)
+ val remainingTime = bootstrapTime - System.currentTimeMillis()
+ if (remainingTime <= 0) {
+ // We just checked for new credentials but none were there, wait a minute and retry.
+ // This handles the shutdown case where the staging directory may have been removed(see
+ // SPARK-12316 for more details).
+ credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.MINUTES)
+ } else {
+ logInfo(s"Scheduling credentials refresh from HDFS in $remainingTime millis.")
+ credentialUpdater.schedule(credentialUpdaterRunnable, remainingTime, TimeUnit.MILLISECONDS)
+ }
+ }
- def updateCredentialsIfRequired(): Unit = {
- try {
+ private def updateCredentialsIfRequired(): Unit = {
+ val timeToNextUpdate = try {
val credentialsFilePath = new Path(credentialsFile)
val remoteFs = FileSystem.get(freshHadoopConf)
SparkHadoopUtil.get.listFilesSorted(
remoteFs, credentialsFilePath.getParent,
credentialsFilePath.getName, SparkHadoopUtil.SPARK_YARN_CREDS_TEMP_EXTENSION)
- .lastOption.foreach { credentialsStatus =>
+ .lastOption.map { credentialsStatus =>
val suffix = SparkHadoopUtil.get.getSuffixForCredentialsPath(credentialsStatus.getPath)
--- End diff --
Doesn't look like your fault, but this whole block code is indented incorrectly. Should be one level deeper.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71047970
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -193,8 +200,14 @@ private[yarn] class AMDelegationTokenRenewer(
}
}
val nextSuffix = lastCredentialsFileSuffix + 1
+ // Time of next update should be late than renewal,
--- End diff --
later?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14065
Hi @tgravescs , thanks a lot for your comments. What I did to test is to run on security cluster with hive, hbase and hdfs installed and kerberized:
1. Verify credentials can be obtained in yarn/client properly.
2. Verify if principal and keytab is specified, credentials can be obtained in yarn/client correctly.
3. Verify if principal and keytab is specified, credentials can be renewed and updated in a specific time. I changed the renewal and update parameter from 75%, 80% to 1% and 1.2% to expect fast credential renewal. To see if:
* Credentials can be renewed in AM in a specific time.
* Renewed credentials can be written into hdfs with proper name: <credential>-<timestamp>-<integer suffix>.
* Credentials can be updated in driver and executors correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73930614
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -53,50 +53,58 @@ private[spark] class CredentialUpdater(
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
- @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
+ /** Start the credentail updater thread periodically to get the credentials */
+ def bootstrap(): Unit = {
+ val bootstrapTime = sparkConf.get(CREDENTIALS_RENEWAL_TIME)
--- End diff --
Similarly, I'd call this `startTime` or `runTime`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71102195
--- Diff: dev/.rat-excludes ---
@@ -99,3 +99,4 @@ spark-deps-.*
.*tsv
org.apache.spark.scheduler.ExternalClusterManager
.*\.sql
+org.apache.spark.deploy.yarn.security.ServiceCredentialProvider
--- End diff --
I followed the convention of other services like `DataSource` which also needs to be excluded from rat check.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62915/consoleFull)** for PR 14065 at commit [`dc3be43`](https://github.com/apache/spark/commit/dc3be432c5467d1d6f0fedbcb0cd172ab0fa8bba).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62467/consoleFull)** for PR 14065 at commit [`a66aed0`](https://github.com/apache/spark/commit/a66aed0d1f299b87f68f34f96f0ccc539507dfde).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #63516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63516/consoleFull)** for PR 14065 at commit [`bce8cd6`](https://github.com/apache/spark/commit/bce8cd6a5796fc95a432c169c8c40bea552382f0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71364134
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ServiceCredentialProvider.scala ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * ::DeveloperApi::
+ * A credential provider for this service, any service want to access by spark through security
+ * way must implement this interface.
+ */
+@DeveloperApi
+trait ServiceCredentialProvider {
+
+ /**
+ * Name of the service to provide credentials. This name should unique, Spark internally will
+ * use this name to differentiate credential provider.
+ */
+ def serviceName: String
+
+ /**
+ * To decide whether credential is required for this service. By default it based on whether
+ * security is enabled or not.
--- End diff --
whether Hadoop security is enabled
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62658/consoleFull)** for PR 14065 at commit [`bd04a2b`](https://github.com/apache/spark/commit/bd04a2b9590e59800e1206103d32779d190056ae).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69991485
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/token/ServiceTokenProvider.scala ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.token
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+import org.apache.hadoop.security.token.Token
+
+import org.apache.spark.SparkConf
+
+/**
+ * An interface to provide tokens for service, any service wants to communicate with Spark
+ * through token way needs to implement this interface and register into
+ * [[ConfigurableTokenManager]] through configurations.
+ */
+trait ServiceTokenProvider {
+
+ /**
+ * Name of the ServiceTokenProvider, should be unique. Using this to distinguish different
+ * service.
+ */
+ def serviceName: String
+
+ /**
+ * Used to indicate whether a token is required.
+ */
+ def isTokenRequired(conf: Configuration): Boolean = {
+ UserGroupInformation.isSecurityEnabled
+ }
+
+ /**
+ * Obtain tokens from this service, tokens will be added into Credentials and return as array.
+ */
+ def obtainTokensFromService(
+ sparkConf: SparkConf,
+ serviceConf: Configuration,
+ creds: Credentials): Array[Token[_]]
+}
+
+/**
+ * An interface for service in which token can be renewable, any [[ServiceTokenProvider]] in which
+ * token can be renewable should also implement this interface, Spark's internal time-based
+ * token renewal mechanism will invoke the methods to update the tokens periodically.
+ */
+trait ServiceTokenRenewable {
+
+ /**
+ * Get the token renewal interval from this service. This renewal interval will be used in
+ * periodical token renewal mechanism.
+ */
+ def getTokenRenewalInterval(sparkConf: SparkConf, serviceConf: Configuration): Long
+
+ /**
+ * Get the time length from now to next renewal.
+ */
+ def getTimeFromNowToRenewal(
--- End diff --
You only really need this method in the interface, right? The token provider should know what info it needs to calculate this value. It might not event need `getTokenRenewalInterval` for that (HDFS does, but then that logic should live inside the HDFS provider).
At that point, you could just merge both interfaces and have this method return an `Option` (None = no renewal necessary), or some magic value (e.g. `-1`) to indicate no renewal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73418187
--- Diff: docs/running-on-yarn.md ---
@@ -461,15 +461,24 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
</tr>
<tr>
- <td><code>spark.yarn.security.tokens.${service}.enabled</code></td>
+ <td><code>spark.yarn.security.credentials.${service}.enabled</code></td>
<td><code>true</code></td>
<td>
- Controls whether to retrieve delegation tokens for non-HDFS services when security is enabled.
- By default, delegation tokens for all supported services are retrieved when those services are
+ Controls whether to obtain credentials for services when security is enabled.
+ By default, credentials for all supported services are retrieved when those services are
configured, but it's possible to disable that behavior if it somehow conflicts with the
- application being run.
+ application being run. For the details please see
+ [Running in a Secure Cluster](running-on-yarn.html#running-in-a-secure-cluster)
<p/>
- Currently supported services are: <code>hive</code>, <code>hbase</code>
+ Currently supported services are: <code>hdfs</code>, <code>hive</code>, <code>hbase</code>
+ </td>
+</tr>
+<tr>
+ <td><code>spark.yarn.security.tokens.${service}.enabled</code></td>
--- End diff --
I'd just remove deprecated configs from the documentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73417691
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala ---
@@ -338,15 +312,15 @@ class SparkHadoopUtil extends Logging {
}
/**
- * Start a thread to periodically update the current user's credentials with new delegation
- * tokens so that writes to HDFS do not fail.
+ * Start a thread to periodically update the current user's credentials with new credentials so
+ * that access to security service do not fail.
--- End diff --
nit: s/security/secured; "does not fail"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72491990
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HDFSCredentialProvider.scala ---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.io.{ByteArrayInputStream, DataInputStream}
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier
+import org.apache.hadoop.mapred.Master
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.deploy.yarn.config._
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+
+private[security] class HDFSCredentialProvider extends ServiceCredentialProvider with Logging {
+
+ private val sparkConf = new SparkConf()
--- End diff --
Hmm, not sure I like this. Feels like this should be an argument to `obtainCredentials`. Or maybe you should have some kind of explicit initialization method. But code like this makes it hard to mock configuration during tests, for example.
One idea is to modify `isCredentialRequired` so that it's an actual initialization method that returns whether the provider should be active.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71347178
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala ---
@@ -319,6 +319,16 @@ package object config {
.stringConf
.createOptional
+ private[spark] val CREDENTIALS_RENEWAL_TIME = ConfigBuilder("spark.yarn.credentials.renewal")
+ .internal()
+ .timeConf(TimeUnit.MILLISECONDS)
+ .createWithDefault(Long.MaxValue)
+
+ private[spark] val CREDENTIALS_UPDATE_TIME = ConfigBuilder("spark.yarn.credentials.update")
--- End diff --
so this is really executors fetching credentials from AM correct? It took me a few minutes to figure out what this config was vs the one above, can we rename it to spark.yarn.credentials.executors.updateTime and CREDENTIALS_EXECUTOR_UPDATE_TIME.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71349021
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -400,8 +414,6 @@ private[spark] class Client(
// same name but different path files are added multiple time, YARN will fail to launch
// containers for the app with an internal error.
val distributedNames = new HashSet[String]
- YarnSparkHadoopUtil.get.obtainTokenForHiveMetastore(sparkConf, hadoopConf, credentials)
- YarnSparkHadoopUtil.get.obtainTokenForHBase(sparkConf, hadoopConf, credentials)
if (credentials != null) {
--- End diff --
this is now a bit out of place, the idea was to dump the tokens for debugging after they were filled. We should move this after the obtainCredentials call above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14065
@jerryshao do you plan to address the remaining feedback? (https://github.com/apache/spark/pull/14065#discussion_r73423485)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-16342][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61830/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71048093
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -193,8 +200,14 @@ private[yarn] class AMDelegationTokenRenewer(
}
}
val nextSuffix = lastCredentialsFileSuffix + 1
+ // Time of next update should be late than renewal,
+ // which is around new token issue date + 0.8 * renew interval
+ val timeOfNextUpdate =
--- End diff --
This is also part of the logic that could be consolidated in `credentialManager.obtainCredentials`. Also, comments above don't really match what the code is doing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-16342][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #61830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61830/consoleFull)** for PR 14065 at commit [`90f194e`](https://github.com/apache/spark/commit/90f194e34ffd198d2b1ee5b04f24afd8c4454d90).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14065
LGTM, just pointed out one thing that I think would make the code easier to follow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72492886
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ServiceCredentialProvider.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+
+/**
+ * A credential provider for a service. User must implement this if they need to access a
+ * secure service from Spark.
+ */
+trait ServiceCredentialProvider {
+
+ /**
+ * Name of the service to provide credentials. This name should unique, Spark internally will
+ * use this name to differentiate credential provider.
+ */
+ def serviceName: String
+
+ /**
+ * To decide whether credential is required for this service. By default it based on whether
+ * Hadoop security is enabled.
+ */
+ def isCredentialRequired(hadoopConf: Configuration): Boolean = {
--- End diff --
Aside from my previous comment to maybe re-purpose this method, it should be `credentialsRequired` (or `areCredentialsRequired` but that looks a bit weird in code).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #61896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61896/consoleFull)** for PR 14065 at commit [`2b17f2e`](https://github.com/apache/spark/commit/2b17f2e906b2fd67e1c2398fd8ba88b64b797550).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62149 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62149/consoleFull)** for PR 14065 at commit [`60a275f`](https://github.com/apache/spark/commit/60a275f9487eacab8ce8c70d6917d3aebe16d131).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62467/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73423485
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -41,16 +43,18 @@ private[spark] class ExecutorDelegationTokenUpdater(
SparkHadoopUtil.get.getConfBypassingFSCache(
hadoopConf, new Path(credentialsFile).toUri.getScheme)
- private val delegationTokenRenewer =
+ private val credentialUpdater =
Executors.newSingleThreadScheduledExecutor(
- ThreadUtils.namedThreadFactory("Delegation Token Refresh Thread"))
+ ThreadUtils.namedThreadFactory("Credential Refresh Thread"))
- // On the executor, this thread wakes up and picks up new tokens from HDFS, if any.
- private val executorUpdaterRunnable =
+ // This thread wakes up and picks up new credentials from HDFS, if any.
+ private val credentialUpdaterRunnable =
new Runnable {
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
+ @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
--- End diff --
Not sure I understand.
`timeOfNextUpdate` is set in L73, and is used in L82. So it doesn't seem like keeping this state across calls of that method is necessary.
I can see a case where, if the directory where credentials are expected to be stored is empty, then the code skips L73. That seems to be used to "bootstrap" the updater; the first invocation of `updateCredentialsIfRequired` will schedule an update based on the renewal time set in the conf.
I find that behavior a little odd and would prefer if the "bootstrapping" were more explicit, i.e., if there was code to explicitly schedule the first `updateCredentialsIfRequired` with the value of `CREDENTIALS_UPDATE_TIME`, and have the code treat an empty directory the same way as if the more recent file does not match the expected (i.e. L76).
I know this code is not being added as part of your change, but could you do that cleanup?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14065
Thanks @tgravescs for your comments, I will add the docs about it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71107536
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -69,6 +71,9 @@ private[yarn] class AMDelegationTokenRenewer(
private val freshHadoopConf =
hadoopUtil.getConfBypassingFSCache(hadoopConf, new Path(credentialsFile).toUri.getScheme)
+ @volatile private var timeOfNextRenewal =
+ sparkConf.get(CREDENTIALS_RENEWAL_TIME).getOrElse(Long.MaxValue)
--- End diff --
One possible scenario from current code is that if principal and keytab is configured, but all credential are non-renewable or disabled. In such case `CREDENTIALS_RENEWAL_TIME` may be empty but `AMDelegationTokenRenewer` will still be started because it only relies on principal configured or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69918095
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -390,8 +390,9 @@ private[spark] class Client(
// Upload Spark and the application JAR to the remote file system if necessary,
// and add them as local resources to the application master.
val fs = destDir.getFileSystem(hadoopConf)
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + destDir
- YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
+ hdfsTokenProvider(sparkConf).setNameNodesToAccess(sparkConf, Set(destDir))
--- End diff --
it seems a bit odd to me that we are doing these extra things for hdfs outside of the token provider.
what happens if a user needs to do something similar for some other service they want to implement their own class to handle?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63068/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r70020146
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -390,8 +390,9 @@ private[spark] class Client(
// Upload Spark and the application JAR to the remote file system if necessary,
// and add them as local resources to the application master.
val fs = destDir.getFileSystem(hadoopConf)
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + destDir
- YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
+ hdfsTokenProvider(sparkConf).setNameNodesToAccess(sparkConf, Set(destDir))
--- End diff --
Yes, I agreed that it is a bit odd for HDFS token provider to have special code path, let me find out a way to get rid of this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71048959
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManager.scala ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * A ConfigurableCredentialManager to manage all the registered credential providers and offer
+ * APIs for other modules to obtain credentials as well as renewal time. By default
+ * [[HDFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] will
+ * be loaded in, any plugged-in credential provider wants to be managed by
+ * ConfigurableCredentialManager needs to implement [[ServiceCredentialProvider]] interface and put
+ * into resources to be loaded by ServiceLoader.
+ *
+ * Also the specific credential provider is controlled by
+ * spark.yarn.security.credentials.{service}.enabled, it will not be loaded in if set to false.
+ */
+final class ConfigurableCredentialManager private[yarn] (sparkConf: SparkConf) extends Logging {
+ private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
+ private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"
+
+ // Maintain all the registered credential providers
+ private val credentialProviders = mutable.HashMap[String, ServiceCredentialProvider]()
+
+ // Default crendetial providers that will be loaded automatically, unless specifically disabled.
+ private val defaultCredentialProviders = Map(
+ "hdfs" -> "org.apache.spark.deploy.yarn.security.HDFSCredentialProvider",
+ "hive" -> "org.apache.spark.deploy.yarn.security.HiveCredentialProvider",
+ "hbase" -> "org.apache.spark.deploy.yarn.security.HBaseCredentialProvider"
+ )
+
+ // AMDelegationTokenRenewer, this will only be create and started in the AM
+ private var _delegationTokenRenewer: AMDelegationTokenRenewer = null
+
+ // ExecutorDelegationTokenUpdater, this will only be created and started in the driver and
+ // executor side.
+ private var _delegationTokenUpdater: ExecutorDelegationTokenUpdater = null
+
+ def initialize(): Unit = {
+ val providers = ServiceLoader.load(classOf[ServiceCredentialProvider],
+ Utils.getContextOrSparkClassLoader).asScala
+
+ // Filter out credentials in which spark.yarn.security.credentials.{service}.enabled is false.
+ providers.filter { p =>
+ sparkConf.getOption(providerEnabledConfig.format(p.serviceName))
+ .orElse {
+ sparkConf.getOption(deprecatedProviderEnabledConfig.format(p.serviceName)).map { c =>
+ logWarning(s"${deprecatedProviderEnabledConfig.format(p.serviceName)} is deprecated, " +
+ s"using ${providerEnabledConfig.format(p.serviceName)} instead")
+ c
+ }
+ }
+ .getOrElse(defaultCredentialProviders.keySet.find(_ == p.serviceName).isDefined.toString)
+ .toBoolean
+ }.foreach {
+ p => credentialProviders(p.serviceName) = p
+ }
+ }
+
+ /**
+ * Get credential provider for the specified service.
+ */
+ def getServiceCredentialProvider(service: String): Option[ServiceCredentialProvider] = {
+ credentialProviders.get(service)
+ }
+
+ /**
+ * Obtain credentials for specified service.
+ * @return time of next renewal if this service credential is renewable.
+ */
+ def obtainCredentialsFromService(
+ service: String,
+ hadoopConf: Configuration,
+ creds: Credentials): Option[Long] = {
+ getServiceCredentialProvider(service).flatMap { provider =>
+ if (provider.isCredentialRequired(hadoopConf)) {
+ provider.obtainCredentials(hadoopConf, creds)
+ } else {
+ None
+ }
+ }
+ }
+
+ /**
+ * Obtain credentials from all the registered providers.
+ * @return Array of time of next renewal if credential can be renewed.
+ * Otherwise will return None instead.
+ */
+ def obtainCredentials(hadoopConf: Configuration, creds: Credentials): Array[Option[Long]] = {
+ val timeOfNextRenewals = mutable.ArrayBuffer[Option[Long]]()
+ credentialProviders.values.foreach { provider =>
+ if (provider.isCredentialRequired(hadoopConf)) {
+ timeOfNextRenewals += provider.obtainCredentials(hadoopConf, creds)
+ } else {
+ logDebug(s"Service ${provider.serviceName} does not require a token." +
+ s" Check your configuration to see if security is disabled or not.")
+ }
+ }
+
+ timeOfNextRenewals.toArray
+ }
+
+ def delegationTokenRenewer(hadoopConf: Configuration): AMDelegationTokenRenewer = synchronized {
+ if (_delegationTokenRenewer == null) {
+ _delegationTokenRenewer = new AMDelegationTokenRenewer(sparkConf, hadoopConf, this)
+ _delegationTokenRenewer
+ } else {
+ _delegationTokenRenewer
+ }
+ }
+
+ def delegationTokenUpdater(hadoopConf: Configuration
--- End diff --
Same comment about the singleton-like interface.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN][WIP] Add a configurable token...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r69988867
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala ---
@@ -96,237 +87,19 @@ class YarnSparkHadoopUtil extends SparkHadoopUtil {
if (credentials != null) credentials.getSecretKey(new Text(key)) else null
}
- /**
- * Get the list of namenodes the user may access.
- */
- def getNameNodesToAccess(sparkConf: SparkConf): Set[Path] = {
- sparkConf.get(NAMENODES_TO_ACCESS)
- .map(new Path(_))
- .toSet
- }
-
- def getTokenRenewer(conf: Configuration): String = {
- val delegTokenRenewer = Master.getMasterPrincipal(conf)
- logDebug("delegation token renewer is: " + delegTokenRenewer)
- if (delegTokenRenewer == null || delegTokenRenewer.length() == 0) {
- val errorMessage = "Can't get Master Kerberos principal for use as renewer"
- logError(errorMessage)
- throw new SparkException(errorMessage)
- }
- delegTokenRenewer
- }
-
- /**
- * Obtains tokens for the namenodes passed in and adds them to the credentials.
- */
- def obtainTokensForNamenodes(
- paths: Set[Path],
- conf: Configuration,
- creds: Credentials,
- renewer: Option[String] = None
- ): Unit = {
- if (UserGroupInformation.isSecurityEnabled()) {
- val delegTokenRenewer = renewer.getOrElse(getTokenRenewer(conf))
- paths.foreach { dst =>
- val dstFs = dst.getFileSystem(conf)
- logInfo("getting token for namenode: " + dst)
- dstFs.addDelegationTokens(delegTokenRenewer, creds)
- }
- }
- }
-
- /**
- * Obtains token for the Hive metastore and adds them to the credentials.
- */
- def obtainTokenForHiveMetastore(
- sparkConf: SparkConf,
- conf: Configuration,
- credentials: Credentials) {
- if (shouldGetTokens(sparkConf, "hive") && UserGroupInformation.isSecurityEnabled) {
- YarnSparkHadoopUtil.get.obtainTokenForHiveMetastore(conf).foreach {
- credentials.addToken(new Text("hive.server2.delegation.token"), _)
- }
- }
- }
-
- /**
- * Obtain a security token for HBase.
- */
- def obtainTokenForHBase(
- sparkConf: SparkConf,
- conf: Configuration,
- credentials: Credentials): Unit = {
- if (shouldGetTokens(sparkConf, "hbase") && UserGroupInformation.isSecurityEnabled) {
- YarnSparkHadoopUtil.get.obtainTokenForHBase(conf).foreach { token =>
- credentials.addToken(token.getService, token)
- logInfo("Added HBase security token to credentials.")
- }
- }
- }
-
- /**
- * Return whether delegation tokens should be retrieved for the given service when security is
- * enabled. By default, tokens are retrieved, but that behavior can be changed by setting
- * a service-specific configuration.
- */
- private def shouldGetTokens(conf: SparkConf, service: String): Boolean = {
- conf.getBoolean(s"spark.yarn.security.tokens.${service}.enabled", true)
- }
-
private[spark] override def startExecutorDelegationTokenRenewer(sparkConf: SparkConf): Unit = {
- tokenRenewer = Some(new ExecutorDelegationTokenUpdater(sparkConf, conf))
- tokenRenewer.get.updateCredentialsIfRequired()
+ configurableTokenManager(sparkConf).delegationTokenUpdater(conf)
--- End diff --
I find this syntax a little confusing. You're calling `configurableTokenManager(sparkConf)` in a bunch of different places. To me that looks like either:
- each call is creating a new token manager
- there's some cache of token managers somewhere keyed by the spark configuration passed here
Neither sounds good to me. And the actual implementation is actually neither: there's a single token manager singleton that is instantiated in the first call to `configurableTokenManager`.
Why doesn't `Client` instantiate a token manager in its constructor instead? Another option is to have an explicit method in `ConfigurableTokenManager` to initialize the singleton, although I'm not a fan of singletons in general.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71480635
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -390,8 +391,21 @@ private[spark] class Client(
// Upload Spark and the application JAR to the remote file system if necessary,
// and add them as local resources to the application master.
val fs = destDir.getFileSystem(hadoopConf)
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + destDir
- YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
+
+ // Merge credentials obtained from registered providers
+ val nearestTimeOfNextRenewal = credentialManager.obtainCredentials(hadoopConf, credentials)
+
+ // If we use principal and keytab to login, also credentials can be renewed some time
+ // after current time, we should pass the next renewal and updating time to delegation token
+ // renewer and updater.
+ if (loginFromKeytab && nearestTimeOfNextRenewal > System.currentTimeMillis() &&
+ nearestTimeOfNextRenewal != Long.MaxValue) {
+ sparkConf.set(CREDENTIALS_RENEWAL_TIME, nearestTimeOfNextRenewal)
+ val nearestTimeOfNextUpdate =
+ (nearestTimeOfNextRenewal - System.currentTimeMillis()) * 1.1 + System.currentTimeMillis()
--- End diff --
Yes, this is a valid concern, I'm also thinking about this, I will try to improve this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #63200 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63200/consoleFull)** for PR 14065 at commit [`127d85e`](https://github.com/apache/spark/commit/127d85ed54f057581a35c88fc7f85e1b8e13de38).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62149/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62467/consoleFull)** for PR 14065 at commit [`a66aed0`](https://github.com/apache/spark/commit/a66aed0d1f299b87f68f34f96f0ccc539507dfde).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-16342][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #61830 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61830/consoleFull)** for PR 14065 at commit [`90f194e`](https://github.com/apache/spark/commit/90f194e34ffd198d2b1ee5b04f24afd8c4454d90).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62915/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72487536
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMCredentialRenewer.scala ---
@@ -27,39 +27,42 @@ import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.SparkConf
import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
import org.apache.spark.deploy.yarn.config._
import org.apache.spark.internal.Logging
import org.apache.spark.internal.config._
import org.apache.spark.util.ThreadUtils
-/*
+/**
* The following methods are primarily meant to make sure long-running apps like Spark
- * Streaming apps can run without interruption while writing to secure HDFS. The
- * scheduleLoginFromKeytab method is called on the driver when the
- * CoarseGrainedScheduledBackend starts up. This method wakes up a thread that logs into the KDC
- * once 75% of the renewal interval of the original delegation tokens used for the container
- * has elapsed. It then creates new delegation tokens and writes them to HDFS in a
+ * Streaming apps can run without interruption while access security services. The
--- End diff --
accessing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73931184
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -53,50 +53,58 @@ private[spark] class CredentialUpdater(
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
- @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
+ /** Start the credentail updater thread periodically to get the credentials */
+ def bootstrap(): Unit = {
+ val bootstrapTime = sparkConf.get(CREDENTIALS_RENEWAL_TIME)
+ val remainingTime = bootstrapTime - System.currentTimeMillis()
+ if (remainingTime <= 0) {
+ // We just checked for new credentials but none were there, wait a minute and retry.
+ // This handles the shutdown case where the staging directory may have been removed(see
+ // SPARK-12316 for more details).
+ credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.MINUTES)
+ } else {
+ logInfo(s"Scheduling credentials refresh from HDFS in $remainingTime millis.")
+ credentialUpdater.schedule(credentialUpdaterRunnable, remainingTime, TimeUnit.MILLISECONDS)
+ }
+ }
- def updateCredentialsIfRequired(): Unit = {
- try {
+ private def updateCredentialsIfRequired(): Unit = {
+ val timeToNextUpdate = try {
val credentialsFilePath = new Path(credentialsFile)
val remoteFs = FileSystem.get(freshHadoopConf)
SparkHadoopUtil.get.listFilesSorted(
remoteFs, credentialsFilePath.getParent,
credentialsFilePath.getName, SparkHadoopUtil.SPARK_YARN_CREDS_TEMP_EXTENSION)
- .lastOption.foreach { credentialsStatus =>
+ .lastOption.map { credentialsStatus =>
val suffix = SparkHadoopUtil.get.getSuffixForCredentialsPath(credentialsStatus.getPath)
if (suffix > lastCredentialsFileSuffix) {
logInfo("Reading new credentials from " + credentialsStatus.getPath)
val newCredentials = getCredentialsFromHDFSFile(remoteFs, credentialsStatus.getPath)
lastCredentialsFileSuffix = suffix
UserGroupInformation.getCurrentUser.addCredentials(newCredentials)
logInfo("Credentials updated from credentials file.")
- timeOfNextUpdate = getTimeOfNextUpdateFromFileName(credentialsStatus.getPath)
+
+ val remainingTime =
+ getTimeOfNextUpdateFromFileName(credentialsStatus.getPath) - System.currentTimeMillis()
+ if (remainingTime <= 0) 60L * 1000 else remainingTime
} else {
- // Check every hour to see if new credentials arrived.
- logInfo("Updated credentials were expected, but the AM has not updated the " +
- "credentials yet, will check again in an hour.")
- credentialUpdater.schedule(credentialUpdaterRunnable, 1, TimeUnit.HOURS)
- return
+ // If current credential file is elder than expected, sleep 1 hour and check again.
--- End diff --
"older"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable token mana...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r71047626
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala ---
@@ -171,10 +175,13 @@ private[yarn] class AMDelegationTokenRenewer(
keytabLoggedInUGI.doAs(new PrivilegedExceptionAction[Void] {
// Get a copy of the credentials
override def run(): Void = {
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + dst
- hadoopUtil.obtainTokensForNamenodes(nns, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHiveMetastore(sparkConf, freshHadoopConf, tempCreds)
- hadoopUtil.obtainTokenForHBase(sparkConf, freshHadoopConf, tempCreds)
+ var nearestNextTime = Long.MaxValue
+ credentialManager.obtainCredentials(freshHadoopConf, tempCreds).foreach { nextTime =>
--- End diff --
This logic looks identical to the one in `Client.scala`. Maybe `credentialManager.obtainCredentials` should do this calculation and return when to next invoke credential renewal?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73930244
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -53,50 +53,58 @@ private[spark] class CredentialUpdater(
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
- @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
+ /** Start the credentail updater thread periodically to get the credentials */
--- End diff --
"credential"
Also the whole comment sounds redundant. Just: "Start the credential updater task."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62506/consoleFull)** for PR 14065 at commit [`b8eeb28`](https://github.com/apache/spark/commit/b8eeb28b141b678cf4ccace36564f24536758132).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-16342][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72335383
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala ---
@@ -312,15 +312,15 @@ class SparkHadoopUtil extends Logging {
}
/**
- * Start a thread to periodically update the current user's credentials with new delegation
- * tokens so that writes to HDFS do not fail.
+ * Start a thread to periodically update the current user's credentials with new credentails so
--- End diff --
nit: credentials misspelled
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73930764
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -53,50 +53,58 @@ private[spark] class CredentialUpdater(
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
- @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
+ /** Start the credentail updater thread periodically to get the credentials */
+ def bootstrap(): Unit = {
+ val bootstrapTime = sparkConf.get(CREDENTIALS_RENEWAL_TIME)
+ val remainingTime = bootstrapTime - System.currentTimeMillis()
+ if (remainingTime <= 0) {
+ // We just checked for new credentials but none were there, wait a minute and retry.
--- End diff --
This comment is now wrong, since you haven't checked for anything yet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14065
Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable token manager for...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #62468 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62468/consoleFull)** for PR 14065 at commit [`9863ca4`](https://github.com/apache/spark/commit/9863ca42427479bf79731f204f606d1ccae20491).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r72491505
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HDFSCredentialProvider.scala ---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn.security
+
+import java.io.{ByteArrayInputStream, DataInputStream}
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier
+import org.apache.hadoop.mapred.Master
+import org.apache.hadoop.security.Credentials
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.deploy.yarn.config._
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+
+private[security] class HDFSCredentialProvider extends ServiceCredentialProvider with Logging {
+
+ private val sparkConf = new SparkConf()
+
+ // NameNode to access, used to get tokens from different FileSystems
+ private val unionNNsToAccess = {
+ val tmpConf = SparkHadoopUtil.get.newConfiguration(sparkConf)
+ val stageDir = sparkConf.get(STAGING_DIR).map(new Path(_))
+ .getOrElse(FileSystem.get(tmpConf).getHomeDirectory)
+ nnsToAccess(sparkConf) + stageDir
+ }
+
+ // Token renewal interval, this value will be set in the first call,
+ // if None means no token renewer specified, so cannot get token renewal interval.
+ private var tokenRenewalInterval: Option[Long] = null
+
+ override val serviceName: String = "hdfs"
--- End diff --
nit: too many spaces after `val`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14065
**[Test build #61912 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61912/consoleFull)** for PR 14065 at commit [`c9d9ed0`](https://github.com/apache/spark/commit/c9d9ed0cd0aef6b8017c04e635f27ef123a48887).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN] Add a configurable credential manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63405/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14065
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14065: [SPARK-14743][YARN][WIP] Add a configurable token manage...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14065
@tgravescs and @vanzin , these days I did some code refactoring work on this patch. Here listed changes I did compared to previous code:
1. Change the interface `ServiceTokenProvider` to `ServiceCredentialProvider` with the main method changed to `obtainCredentials` as suggested in comments.
Now since we're not limiting to tokens, so instead of obtaining tokens, here changed to obtain credentials. And the method `obtainCredentials` is defined as:
```scala
def obtainCredentials(hadoopConf: Configuration, creds: Credentials): Option[Long]
```
Here the return value `Option[Long]` means the time of next renewal, return `Some(Long)` if this credential is renewable, otherwise returns `None`.
Also remove several redundant methods like get token renewal interval and so on.
2. Change `ConfigurableTokenManager` to `ConfigurableCredentialManager` to manager all the credential providers.
3. Change the way to load credential providers to ServiceLoader as suggested in comments.
4. Change initialization way from singleton to normal way.
6. Change the mechanism of checking credentials in `AMDelegationRenewer` and `ExecutorDelegationTokenUpdate`. Since now we can get the time of next renewal, so we use this to decide when to wake up to check the new credentials.
Please help to review, thanks a lot for your time and greatly appreciate your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14065: [SPARK-14743][YARN] Add a configurable credential...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14065#discussion_r73930350
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala ---
@@ -53,50 +53,58 @@ private[spark] class CredentialUpdater(
override def run(): Unit = Utils.logUncaughtExceptions(updateCredentialsIfRequired())
}
- @volatile private var timeOfNextUpdate = sparkConf.get(CREDENTIALS_UPDATE_TIME)
+ /** Start the credentail updater thread periodically to get the credentials */
+ def bootstrap(): Unit = {
--- End diff --
I'd call this `startUpdater` or just `start`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org