You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Arthur Rand (JIRA)" <ji...@apache.org> on 2017/09/18 00:43:02 UTC
[jira] [Comment Edited] (SPARK-21842) Support Kerberos ticket
renewal and creation in Mesos
[ https://issues.apache.org/jira/browse/SPARK-21842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168278#comment-16168278 ]
Arthur Rand edited comment on SPARK-21842 at 9/18/17 12:42 AM:
---------------------------------------------------------------
Hey [~kalvinnchau]
I'm currently of the mind that using the RPC/broadcast approach is better (for Mesos) for a couple reasons.
1. We recently added Secret support to Spark on Mesos, this uses a temporary file system to put the keytab or TGT in the sandbox of the Spark driver. They are packed into the SparkAppConfig (CoarseGrainedSchedulerBackend.scala:L236) which is broadcast to the executors, so using RPC/broadcast is consistent with this.
2. Keeps all transfers of secure information within Spark.
3. Doesn't _require_ HDFS.
However I understand that there is a potential risk with executors falsely registering with the Driver and getting tokens. I know in the case of DC/OS this is less of a concern (we have some protections around this). But this could still happen today due to the code mentioned above. We could prevent this by keeping track of the executor IDs and only allowing executors to register when they have an expected ID..?
was (Author: arand):
Hey [~kalvinnchau]
I'm currently of the mind that using the RPC/broadcast approach is better (for Mesos) for a couple reasons.
1. We recently added Secret support to Spark on Mesos, this uses a temporary file system to put the keytab or TGT in the sandbox of the Spark driver. They are packed into the SparkAppConfig (CoarseGrainedSchedulerBackend.scala:L236) which is broadcast to the executors, so using RPC/broadcast is consistent with this.
2. Keeps all transfers of secure information within Spark.
3. Doesn't require HDFS. There is a little bit of a chicken-and-egg situation here w.r.t. YARN, but I'm obviously not familiar enough with how Spark-YARN-HDFS work together.
However I understand that there is a potential risk with executors falsely registering with the Driver and getting tokens. I know in the case of DC/OS this is less of a concern (we have some protections around this). But this could still happen today due to the code mentioned above. We could prevent this by keeping track of the executor IDs and only allowing executors to register when they have an expected ID..?
> Support Kerberos ticket renewal and creation in Mesos
> ------------------------------------------------------
>
> Key: SPARK-21842
> URL: https://issues.apache.org/jira/browse/SPARK-21842
> Project: Spark
> Issue Type: New Feature
> Components: Mesos
> Affects Versions: 2.3.0
> Reporter: Arthur Rand
>
> We at Mesosphere have written Kerberos support for Spark on Mesos. The code to use Kerberos on a Mesos cluster has been added to Apache Spark (SPARK-16742). This ticket is to complete the implementation and allow for ticket renewal and creation. Specifically for long running and streaming jobs.
> Mesosphere design doc (needs revision, wip): https://docs.google.com/document/d/1xyzICg7SIaugCEcB4w1vBWp24UDkyJ1Pyt2jtnREFqc/edit#heading=h.tdnq7wilqrj6
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org