You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by va...@apache.org on 2015/12/01 20:06:04 UTC
spark git commit: [SPARK-11821] Propagate Kerberos keytab for all
environments
Repository: spark
Updated Branches:
refs/heads/master 0a7bca2da -> 6a8cf80cc
[SPARK-11821] Propagate Kerberos keytab for all environments
andrewor14 the same PR as in branch 1.5
harishreedharan
Author: woj-i <wo...@gmail.com>
Closes #9859 from woj-i/master.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6a8cf80c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6a8cf80c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6a8cf80c
Branch: refs/heads/master
Commit: 6a8cf80cc8ef435ec46138fa57325bda5d68f3ce
Parents: 0a7bca2
Author: woj-i <wo...@gmail.com>
Authored: Tue Dec 1 11:05:45 2015 -0800
Committer: Marcelo Vanzin <va...@cloudera.com>
Committed: Tue Dec 1 11:05:45 2015 -0800
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 4 ++++
docs/running-on-yarn.md | 4 ++--
docs/sql-programming-guide.md | 7 ++++---
3 files changed, 10 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/6a8cf80c/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index 2e912b5..52d3ab3 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -545,6 +545,10 @@ object SparkSubmit {
if (args.isPython) {
sysProps.put("spark.yarn.isPython", "true")
}
+ }
+
+ // assure a keytab is available from any place in a JVM
+ if (clusterManager == YARN || clusterManager == LOCAL) {
if (args.principal != null) {
require(args.keytab != null, "Keytab must be specified when principal is specified")
if (!new File(args.keytab).exists()) {
http://git-wip-us.apache.org/repos/asf/spark/blob/6a8cf80c/docs/running-on-yarn.md
----------------------------------------------------------------------
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index 925a1e0..06413f8 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -358,14 +358,14 @@ If you need a reference to the proper location to put log files in the YARN so t
<td>
The full path to the file that contains the keytab for the principal specified above.
This keytab will be copied to the node running the YARN Application Master via the Secure Distributed Cache,
- for renewing the login tickets and the delegation tokens periodically.
+ for renewing the login tickets and the delegation tokens periodically. (Works also with the "local" master)
</td>
</tr>
<tr>
<td><code>spark.yarn.principal</code></td>
<td>(none)</td>
<td>
- Principal to be used to login to KDC, while running on secure HDFS.
+ Principal to be used to login to KDC, while running on secure HDFS. (Works also with the "local" master)
</td>
</tr>
<tr>
http://git-wip-us.apache.org/repos/asf/spark/blob/6a8cf80c/docs/sql-programming-guide.md
----------------------------------------------------------------------
diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md
index d7b205c..7b1d97b 100644
--- a/docs/sql-programming-guide.md
+++ b/docs/sql-programming-guide.md
@@ -1614,7 +1614,8 @@ This command builds a new assembly jar that includes Hive. Note that this Hive a
on all of the worker nodes, as they will need access to the Hive serialization and deserialization libraries
(SerDes) in order to access data stored in Hive.
-Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`. Please note when running
+Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` (for security configuration),
+ `hdfs-site.xml` (for HDFS configuration) file in `conf/`. Please note when running
the query on a YARN cluster (`cluster` mode), the `datanucleus` jars under the `lib_managed/jars` directory
and `hive-site.xml` under `conf/` directory need to be available on the driver and all executors launched by the
YARN cluster. The convenient way to do this is adding them through the `--jars` option and `--file` option of the
@@ -2028,7 +2029,7 @@ Beeline will ask you for a username and password. In non-secure mode, simply ent
your machine and a blank password. For secure mode, please follow the instructions given in the
[beeline documentation](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients).
-Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`.
+Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` and `hdfs-site.xml` files in `conf/`.
You may also use the beeline script that comes with Hive.
@@ -2053,7 +2054,7 @@ To start the Spark SQL CLI, run the following in the Spark directory:
./bin/spark-sql
-Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`.
+Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` and `hdfs-site.xml` files in `conf/`.
You may run `./bin/spark-sql --help` for a complete list of all available
options.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org