You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Pierre Beauvois (JIRA)" <ji...@apache.org> on 2016/02/01 11:19:39 UTC
[jira] [Updated] (SPARK-13110) How to configure the access of the
Spark History Web UI with Kerberos authentication?
[ https://issues.apache.org/jira/browse/SPARK-13110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pierre Beauvois updated SPARK-13110:
------------------------------------
Description:
Hello,
Spark is installed on several machines of my cluster. These machines are used by the clients (we'll call these machines "CM"). Note that Spark is configured on Yarn and not on standalone mode.
I installed a Spark History server on a different machine which is unaccessible to the clients (we'll call this machine "SHS"). Now I'm trying to configure Spark History web UI access for the users of my cluster who are authenticated with Kerberos. For the moment I have been able to make Spark History working with its kerberos principal and keytab.
The spark-defaults.conf of the SHS is the following:
{code}
# Spark history server configurations
spark.history.provider = org.apache.spark.deploy.history.FsHistoryProvider
spark.history.fs.logDirectory = hdfs:///Products/SPARK/logs/
spark.history.fs.update.interval = 10s
spark.history.retainedApplications = 100
spark.history.ui.port = 18080
spark.history.kerberos.enabled = true
spark.history.kerberos.principal = sparkhistory/sparkhistoryserver.dns.fr@SANDBOX.HADOOP
spark.history.kerberos.keytab = /opt/application/Spark/current/keytabs/sparkhistory.keytab
spark.history.ui.acls.enable = true
spark.history.fs.cleaner.enabled = false
spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
{code}
The spark-defaults.conf of the CM is the following:
{code}
# Spark history server configurations
spark.history.provider =
spark.history.fs.logDirectory =
spark.history.fs.update.interval =
spark.history.retainedApplications =
spark.history.ui.port =
spark.history.kerberos.enabled =
spark.history.kerberos.principal =
spark.history.kerberos.keytab =
spark.history.ui.acls.enable = true
spark.history.fs.cleaner.enabled =
spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
{code}
First I would like to know if my configurations are good (both SHS and CM).
Secondly I would like to know how to restrict the web UI access for the users who are kerberos authenticated. Let me explain more what is the expected behaviour here:
- the user Obelix does a Spark job and finishes it properly
- Obelix can go to the ResourceManager web UI and click on "history". He's redirected to the Spark History web UI and he can have the details of its previous job. Note that Obelix is kerberos authenticated in order to be able to go to the ResourceManager web UI and the Spark History web UI.
- Asterix goes to the ResourceManager web UI and click on "history" of Obelix's job. Asterix is not redirected because he's not the user who launched the job.
was:
Hello,
Spark is installed on several machines of my cluster. These machines are used by the clients (we'll call these machines "CM"). Note that Spark is configured on Yarn and not on standalone mode.
I installed a Spark History server on a different machine which is unaccessible to the clients (we'll call this machine "SHS"). Now I'm trying to configure Spark History web UI access for the users of my cluster who are authenticated with Kerberos. For the moment I have been able to make Spark History working with its kerberos principal and keytab.
The spark-defaults.conf of the SHS is the following:
{code}
# Spark history server configurations
spark.history.provider = org.apache.spark.deploy.history.FsHistoryProvider
spark.history.fs.logDirectory = hdfs:///Products/SPARK/logs/
spark.history.fs.update.interval = 10s
spark.history.retainedApplications = 100
spark.history.ui.port = 18080
spark.history.kerberos.enabled = true
spark.history.kerberos.principal = sparkhistory/sparkhistoryserver.dns.fr@SANDBOX.HADOOP
spark.history.kerberos.keytab = /opt/application/Spark/current/keytabs/sparkhistory.keytab
spark.history.ui.acls.enable = true
spark.history.fs.cleaner.enabled = false
spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
{code}
The spark-defaults.conf of the CM is the following:
{code}
# Spark history server configurations
spark.history.provider =
spark.history.fs.logDirectory =
spark.history.fs.update.interval =
spark.history.retainedApplications =
spark.history.ui.port =
spark.history.kerberos.enabled =
spark.history.kerberos.principal =
spark.history.kerberos.keytab =
spark.history.ui.acls.enable = true
spark.history.fs.cleaner.enabled =
spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
{code}
First I would like to know if my configurations are good.
Secondly I would like to know how to restrict the web UI access for the users who are kerberos authenticated. Let me explain more what is the expected behaviour here:
- the user Obelix does a Spark job and finishes it properly
- Obelix can go to the ResourceManager web UI and click on "history". He's redirected to the Spark History web UI and he can have the details of its previous job. Note that Obelix is kerberos authenticated in order to be able to go to the ResourceManager web UI and the Spark History web UI.
- Asterix goes to the ResourceManager web UI and click on "history" of Obelix's job. Asterix is not redirected because he's not the user who launched the job.
> How to configure the access of the Spark History Web UI with Kerberos authentication?
> -------------------------------------------------------------------------------------
>
> Key: SPARK-13110
> URL: https://issues.apache.org/jira/browse/SPARK-13110
> Project: Spark
> Issue Type: Question
> Components: Web UI
> Affects Versions: 1.5.1, 1.5.2, 1.6.0
> Environment: Spark 1.6.0 / Hadoop 2.7.1 / Zookeeper 3.4.5 / Authentication done through Kerberos
> Reporter: Pierre Beauvois
>
> Hello,
> Spark is installed on several machines of my cluster. These machines are used by the clients (we'll call these machines "CM"). Note that Spark is configured on Yarn and not on standalone mode.
> I installed a Spark History server on a different machine which is unaccessible to the clients (we'll call this machine "SHS"). Now I'm trying to configure Spark History web UI access for the users of my cluster who are authenticated with Kerberos. For the moment I have been able to make Spark History working with its kerberos principal and keytab.
> The spark-defaults.conf of the SHS is the following:
> {code}
> # Spark history server configurations
> spark.history.provider = org.apache.spark.deploy.history.FsHistoryProvider
> spark.history.fs.logDirectory = hdfs:///Products/SPARK/logs/
> spark.history.fs.update.interval = 10s
> spark.history.retainedApplications = 100
> spark.history.ui.port = 18080
> spark.history.kerberos.enabled = true
> spark.history.kerberos.principal = sparkhistory/sparkhistoryserver.dns.fr@SANDBOX.HADOOP
> spark.history.kerberos.keytab = /opt/application/Spark/current/keytabs/sparkhistory.keytab
> spark.history.ui.acls.enable = true
> spark.history.fs.cleaner.enabled = false
> spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
> {code}
> The spark-defaults.conf of the CM is the following:
> {code}
> # Spark history server configurations
> spark.history.provider =
> spark.history.fs.logDirectory =
> spark.history.fs.update.interval =
> spark.history.retainedApplications =
> spark.history.ui.port =
> spark.history.kerberos.enabled =
> spark.history.kerberos.principal =
> spark.history.kerberos.keytab =
> spark.history.ui.acls.enable = true
> spark.history.fs.cleaner.enabled =
> spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
> {code}
> First I would like to know if my configurations are good (both SHS and CM).
> Secondly I would like to know how to restrict the web UI access for the users who are kerberos authenticated. Let me explain more what is the expected behaviour here:
> - the user Obelix does a Spark job and finishes it properly
> - Obelix can go to the ResourceManager web UI and click on "history". He's redirected to the Spark History web UI and he can have the details of its previous job. Note that Obelix is kerberos authenticated in order to be able to go to the ResourceManager web UI and the Spark History web UI.
> - Asterix goes to the ResourceManager web UI and click on "history" of Obelix's job. Asterix is not redirected because he's not the user who launched the job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org