You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rxin <gi...@git.apache.org> on 2014/09/06 09:15:02 UTC
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/2298
[SPARK-938][doc] Add OpenStack Swift support
See compiled doc at
http://people.apache.org/~rxin/tmp/openstack-swift/_site/storage-openstack-swift.html
This is based on #1010.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark openstack-swift
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2298.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2298
----
commit b6c37ef5a28dfa0ed07f6ab8f154fcd43a2131af
Author: Gil Vernik <gi...@il.ibm.com>
Date: 2014-06-08T07:23:41Z
Openstack Swift support
commit ce483d76a1d524800859764b967c8b5a98fbd9ea
Author: Gil Vernik <gi...@il.ibm.com>
Date: 2014-06-08T07:34:04Z
SPARK-938 - Openstack Swift object storage support
This is initial documentation describing how to integrate Spark with
Swift. This commit contains documentation for stand alone cluster.
Next patches will contain details how to integrate Swift in other
deployment of Spark.
commit eff538dd8fb7e306c84874e9b4c7da68fa0fe5d0
Author: Gil Vernik <gi...@il.ibm.com>
Date: 2014-06-08T07:34:04Z
SPARK-938 - Openstack Swift object storage support
Documentation how to integrate Spark with Openstack Swift.
commit 9b625b569e3fc0024edd38b8fc33825c53be8151
Author: Gil Vernik <gi...@il.ibm.com>
Date: 2014-06-10T13:15:01Z
Merge branch 'master' of https://github.com/gilv/spark
Conflicts:
docs/openstack-integration.md
commit 2aba7631c98d881c031c233a39342b936749e161
Author: Gil Vernik <gi...@il.ibm.com>
Date: 2014-06-10T13:15:01Z
Fix to docs/openstack-integration.md
commit c977658580663c8ff0d003938ae833e4d9a19631
Author: Gil Vernik <gi...@il.ibm.com>
Date: 2014-06-10T15:14:05Z
Merge branch 'master' of https://github.com/gilv/spark
Conflicts:
docs/openstack-integration.md
commit 39a9737e16b27435f448030f1f7f7a6c506e08dc
Author: Gil Vernik <gi...@il.ibm.com>
Date: 2014-06-12T09:13:29Z
Spark integration with Openstack Swift
commit eb22295f1c3813d330abf1a4509cc2e8b62abcff
Author: Reynold Xin <rx...@apache.org>
Date: 2014-06-13T06:55:18Z
Merge pull request #1010 from gilv/master
SPARK-938 - Openstack Swift object storage support
commit 99f095d9577802912fa715495bd9aec3e3867d54
Author: Reynold Xin <rx...@apache.org>
Date: 2014-06-14T20:13:17Z
Pending openstack changes.
commit cca719227c828e790f7b2e3c94ef83f5fb55ceb3
Author: Gil Vernik <gi...@il.ibm.com>
Date: 2014-06-16T08:39:51Z
Removed white spases from pom.xml
commit 47ce99d10715e1097feba8690a10a38c09f69fd9
Author: Reynold Xin <rx...@apache.org>
Date: 2014-06-17T01:34:12Z
Merge branch 'master' into openstack
commit ac0679eb81389cef45c3b604581fb53274023f1c
Author: Reynold Xin <rx...@apache.org>
Date: 2014-06-17T01:46:14Z
Fixed an unclosed tr.
commit 69948278581a090cd635ead6672272809858d608
Author: Gil Vernik <gi...@il.ibm.com>
Date: 2014-06-17T06:16:17Z
Merge pull request #1 from rxin/openstack
Bring the branch up to date and fixed some documentation typos.
commit 9233fef3450846fc6ff1e7e7e3c75191a543a573
Author: Gil Vernik <gi...@il.ibm.com>
Date: 2014-06-18T05:19:48Z
Fixed typos
commit e9c376186e5ab4791a15fe3244687909a999b9bb
Author: Reynold Xin <rx...@apache.org>
Date: 2014-09-06T06:14:16Z
Merge pull request #1010 from gilv/master
SPARK-938 - Openstack Swift object storage support
commit 0447c9fc563ef4bc02f937bfea63ea1d62f252cf
Author: Reynold Xin <rx...@apache.org>
Date: 2014-09-06T06:59:34Z
Removed sample code.
commit 846f5cbbb605421646587cb2e065f070d83143ae
Author: Reynold Xin <rx...@apache.org>
Date: 2014-09-06T07:05:18Z
Added a link from overview page.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2298#issuecomment-54774345
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19959/consoleFull) for PR 2298 at commit [`dfb8fea`](https://github.com/apache/spark/commit/dfb8fea59caee1e9fed743d3df18075cba510172).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on a diff in the pull request:
https://github.com/apache/spark/pull/2298#discussion_r17220865
--- Diff: docs/storage-openstack-swift.md ---
@@ -0,0 +1,152 @@
+---
+layout: global
+title: Accessing OpenStack Swift from Spark
+---
+
+Spark's support for Hadoop InputFormat allows it to process data in OpenStack Swift using the
+same URI formats as in Hadoop. You can specify a path in Swift as input through a
+URI of the form <code>swift://container.PROVIDER/path</code>. You will also need to set your
+Swift security credentials, through <code>core-sites.xml</code> or via
+<code>SparkContext.hadoopConfiguration</code>.
+Current Swift driver requires Swift to use Keystone authentication method.
+
+# Configuring Swift for Better Data Locality
+
+Although not mandatory, it is recommended to configure the proxy server of Swift with
+<code>list_endpoints</code> to have better data locality. More information is
+[available here](https://github.com/openstack/swift/blob/master/swift/common/middleware/list_endpoints.py).
+
+
+# Dependencies
+
+The Spark application should include <code>hadoop-openstack</code> dependency.
+For example, for Maven support, add the following to the <code>pom.xml</code> file:
+
+{% highlight xml %}
+<dependencyManagement>
+ ...
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-openstack</artifactId>
+ <version>2.3.0</version>
+ </dependency>
+ ...
+</dependencyManagement>
+{% endhighlight %}
+
+
+# Configuration Parameters
+
+Create <code>core-sites.xml</code> and place it inside <code>/spark/conf</code> directory.
+There are two main categories of parameters that should to be configured: declaration of the
+Swift driver and the parameters that are required by Keystone.
+
+Configuration of Hadoop to use Swift File system achieved via
+
+<table class="table">
+<tr><th>Property Name</th><th>Value</th></tr>
+<tr>
+ <td>fs.swift.impl</td>
+ <td>org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem</td>
+</tr>
+</table>
+
+Additional parameters required by Keystone (v2.0) and should be provided to the Swift driver. Those
+parameters will be used to perform authentication in Keystone to access Swift. The following table
+contains a list of Keystone mandatory parameters. <code>PROVIDER</code> can be any name.
+
+<table class="table">
+<tr><th>Property Name</th><th>Meaning</th><th>Required</th></tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.auth.url</code></td>
+ <td>Keystone Authentication URL</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.auth.endpoint.prefix</code></td>
+ <td>Keystone endpoints prefix</td>
+ <td>Optional</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.tenant</code></td>
+ <td>Tenant</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.username</code></td>
+ <td>Username</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.password</code></td>
+ <td>Password</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.http.port</code></td>
+ <td>HTTP port</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.region</code></td>
+ <td>Keystone region</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.public</code></td>
+ <td>Indicates if all URLs are public</td>
+ <td>Mandatory</td>
+</tr>
+</table>
+
+For example, assume <code>PROVIDER=SparkTest</code> and Keystone contains user <code>tester</code> with password <code>testing</code>
+defined for tenant <code>test</code>. Than <code>core-sites.xml</code> should include:
--- End diff --
here again
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2298#discussion_r17220905
--- Diff: docs/storage-openstack-swift.md ---
@@ -0,0 +1,152 @@
+---
+layout: global
+title: Accessing OpenStack Swift from Spark
+---
+
+Spark's support for Hadoop InputFormat allows it to process data in OpenStack Swift using the
+same URI formats as in Hadoop. You can specify a path in Swift as input through a
+URI of the form <code>swift://container.PROVIDER/path</code>. You will also need to set your
+Swift security credentials, through <code>core-site.xml</code> or via
+<code>SparkContext.hadoopConfiguration</code>.
+Current Swift driver requires Swift to use Keystone authentication method.
+
+# Configuring Swift for Better Data Locality
+
+Although not mandatory, it is recommended to configure the proxy server of Swift with
+<code>list_endpoints</code> to have better data locality. More information is
+[available here](https://github.com/openstack/swift/blob/master/swift/common/middleware/list_endpoints.py).
+
+
+# Dependencies
+
+The Spark application should include <code>hadoop-openstack</code> dependency.
+For example, for Maven support, add the following to the <code>pom.xml</code> file:
+
+{% highlight xml %}
+<dependencyManagement>
+ ...
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-openstack</artifactId>
+ <version>2.3.0</version>
+ </dependency>
+ ...
+</dependencyManagement>
+{% endhighlight %}
+
+
+# Configuration Parameters
+
+Create <code>core-site.xml</code> and place it inside <code>/spark/conf</code> directory.
--- End diff --
i agree
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2298
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on a diff in the pull request:
https://github.com/apache/spark/pull/2298#discussion_r17220850
--- Diff: docs/storage-openstack-swift.md ---
@@ -0,0 +1,152 @@
+---
+layout: global
+title: Accessing OpenStack Swift from Spark
+---
+
+Spark's support for Hadoop InputFormat allows it to process data in OpenStack Swift using the
+same URI formats as in Hadoop. You can specify a path in Swift as input through a
+URI of the form <code>swift://container.PROVIDER/path</code>. You will also need to set your
+Swift security credentials, through <code>core-sites.xml</code> or via
--- End diff --
should this be `core-site.xml`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2298#discussion_r17220914
--- Diff: docs/storage-openstack-swift.md ---
@@ -0,0 +1,152 @@
+---
+layout: global
+title: Accessing OpenStack Swift from Spark
+---
+
+Spark's support for Hadoop InputFormat allows it to process data in OpenStack Swift using the
+same URI formats as in Hadoop. You can specify a path in Swift as input through a
+URI of the form <code>swift://container.PROVIDER/path</code>. You will also need to set your
+Swift security credentials, through <code>core-site.xml</code> or via
+<code>SparkContext.hadoopConfiguration</code>.
+Current Swift driver requires Swift to use Keystone authentication method.
+
+# Configuring Swift for Better Data Locality
+
+Although not mandatory, it is recommended to configure the proxy server of Swift with
+<code>list_endpoints</code> to have better data locality. More information is
+[available here](https://github.com/openstack/swift/blob/master/swift/common/middleware/list_endpoints.py).
+
+
+# Dependencies
+
+The Spark application should include <code>hadoop-openstack</code> dependency.
+For example, for Maven support, add the following to the <code>pom.xml</code> file:
+
+{% highlight xml %}
+<dependencyManagement>
+ ...
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-openstack</artifactId>
+ <version>2.3.0</version>
+ </dependency>
+ ...
+</dependencyManagement>
+{% endhighlight %}
+
+
+# Configuration Parameters
+
+Create <code>core-site.xml</code> and place it inside <code>/spark/conf</code> directory.
+There are two main categories of parameters that should to be configured: declaration of the
+Swift driver and the parameters that are required by Keystone.
+
+Configuration of Hadoop to use Swift File system achieved via
+
+<table class="table">
+<tr><th>Property Name</th><th>Value</th></tr>
+<tr>
+ <td>fs.swift.impl</td>
+ <td>org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem</td>
+</tr>
+</table>
+
+Additional parameters required by Keystone (v2.0) and should be provided to the Swift driver. Those
+parameters will be used to perform authentication in Keystone to access Swift. The following table
+contains a list of Keystone mandatory parameters. <code>PROVIDER</code> can be any name.
+
+<table class="table">
+<tr><th>Property Name</th><th>Meaning</th><th>Required</th></tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.auth.url</code></td>
+ <td>Keystone Authentication URL</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.auth.endpoint.prefix</code></td>
+ <td>Keystone endpoints prefix</td>
+ <td>Optional</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.tenant</code></td>
+ <td>Tenant</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.username</code></td>
+ <td>Username</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.password</code></td>
+ <td>Password</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.http.port</code></td>
+ <td>HTTP port</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.region</code></td>
+ <td>Keystone region</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.public</code></td>
+ <td>Indicates if all URLs are public</td>
+ <td>Mandatory</td>
+</tr>
+</table>
+
+For example, assume <code>PROVIDER=SparkTest</code> and Keystone contains user <code>tester</code> with password <code>testing</code>
+defined for tenant <code>test</code>. Than <code>core-site.xml</code> should include:
--- End diff --
Than --> Then
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on a diff in the pull request:
https://github.com/apache/spark/pull/2298#discussion_r17220885
--- Diff: docs/storage-openstack-swift.md ---
@@ -0,0 +1,152 @@
+---
+layout: global
+title: Accessing OpenStack Swift from Spark
+---
+
+Spark's support for Hadoop InputFormat allows it to process data in OpenStack Swift using the
+same URI formats as in Hadoop. You can specify a path in Swift as input through a
+URI of the form <code>swift://container.PROVIDER/path</code>. You will also need to set your
+Swift security credentials, through <code>core-site.xml</code> or via
+<code>SparkContext.hadoopConfiguration</code>.
+Current Swift driver requires Swift to use Keystone authentication method.
+
+# Configuring Swift for Better Data Locality
+
+Although not mandatory, it is recommended to configure the proxy server of Swift with
+<code>list_endpoints</code> to have better data locality. More information is
+[available here](https://github.com/openstack/swift/blob/master/swift/common/middleware/list_endpoints.py).
+
+
+# Dependencies
+
+The Spark application should include <code>hadoop-openstack</code> dependency.
+For example, for Maven support, add the following to the <code>pom.xml</code> file:
+
+{% highlight xml %}
+<dependencyManagement>
+ ...
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-openstack</artifactId>
+ <version>2.3.0</version>
+ </dependency>
+ ...
+</dependencyManagement>
+{% endhighlight %}
+
+
+# Configuration Parameters
+
+Create <code>core-site.xml</code> and place it inside <code>/spark/conf</code> directory.
--- End diff --
minor - but could this say `conf/` rather than `/spark/conf`... we usually don't assume the name of the root folder (in many cases it e.g. has a version)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2298#issuecomment-54774597
Great thanks LGTM Reynold. I'll pull this in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on a diff in the pull request:
https://github.com/apache/spark/pull/2298#discussion_r17220855
--- Diff: docs/storage-openstack-swift.md ---
@@ -0,0 +1,152 @@
+---
+layout: global
+title: Accessing OpenStack Swift from Spark
+---
+
+Spark's support for Hadoop InputFormat allows it to process data in OpenStack Swift using the
+same URI formats as in Hadoop. You can specify a path in Swift as input through a
+URI of the form <code>swift://container.PROVIDER/path</code>. You will also need to set your
+Swift security credentials, through <code>core-sites.xml</code> or via
+<code>SparkContext.hadoopConfiguration</code>.
+Current Swift driver requires Swift to use Keystone authentication method.
+
+# Configuring Swift for Better Data Locality
+
+Although not mandatory, it is recommended to configure the proxy server of Swift with
+<code>list_endpoints</code> to have better data locality. More information is
+[available here](https://github.com/openstack/swift/blob/master/swift/common/middleware/list_endpoints.py).
+
+
+# Dependencies
+
+The Spark application should include <code>hadoop-openstack</code> dependency.
+For example, for Maven support, add the following to the <code>pom.xml</code> file:
+
+{% highlight xml %}
+<dependencyManagement>
+ ...
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-openstack</artifactId>
+ <version>2.3.0</version>
+ </dependency>
+ ...
+</dependencyManagement>
+{% endhighlight %}
+
+
+# Configuration Parameters
+
+Create <code>core-sites.xml</code> and place it inside <code>/spark/conf</code> directory.
+There are two main categories of parameters that should to be configured: declaration of the
+Swift driver and the parameters that are required by Keystone.
+
+Configuration of Hadoop to use Swift File system achieved via
+
+<table class="table">
+<tr><th>Property Name</th><th>Value</th></tr>
+<tr>
+ <td>fs.swift.impl</td>
+ <td>org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem</td>
+</tr>
+</table>
+
+Additional parameters required by Keystone (v2.0) and should be provided to the Swift driver. Those
+parameters will be used to perform authentication in Keystone to access Swift. The following table
+contains a list of Keystone mandatory parameters. <code>PROVIDER</code> can be any name.
+
+<table class="table">
+<tr><th>Property Name</th><th>Meaning</th><th>Required</th></tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.auth.url</code></td>
+ <td>Keystone Authentication URL</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.auth.endpoint.prefix</code></td>
+ <td>Keystone endpoints prefix</td>
+ <td>Optional</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.tenant</code></td>
+ <td>Tenant</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.username</code></td>
+ <td>Username</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.password</code></td>
+ <td>Password</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.http.port</code></td>
+ <td>HTTP port</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.region</code></td>
+ <td>Keystone region</td>
+ <td>Mandatory</td>
+</tr>
+<tr>
+ <td><code>fs.swift.service.PROVIDER.public</code></td>
+ <td>Indicates if all URLs are public</td>
+ <td>Mandatory</td>
+</tr>
+</table>
+
+For example, assume <code>PROVIDER=SparkTest</code> and Keystone contains user <code>tester</code> with password <code>testing</code>
+defined for tenant <code>test</code>. Than <code>core-sites.xml</code> should include:
+
+{% highlight xml %}
+<configuration>
+ <property>
+ <name>fs.swift.impl</name>
+ <value>org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem</value>
+ </property>
+ <property>
+ <name>fs.swift.service.SparkTest.auth.url</name>
+ <value>http://127.0.0.1:5000/v2.0/tokens</value>
+ </property>
+ <property>
+ <name>fs.swift.service.SparkTest.auth.endpoint.prefix</name>
+ <value>endpoints</value>
+ </property>
+ <name>fs.swift.service.SparkTest.http.port</name>
+ <value>8080</value>
+ </property>
+ <property>
+ <name>fs.swift.service.SparkTest.region</name>
+ <value>RegionOne</value>
+ </property>
+ <property>
+ <name>fs.swift.service.SparkTest.public</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>fs.swift.service.SparkTest.tenant</name>
+ <value>test</value>
+ </property>
+ <property>
+ <name>fs.swift.service.SparkTest.username</name>
+ <value>tester</value>
+ </property>
+ <property>
+ <name>fs.swift.service.SparkTest.password</name>
+ <value>testing</value>
+ </property>
+</configuration>
+{% endhighlight %}
+
+Notice that
+<code>fs.swift.service.PROVIDER.tenant</code>,
+<code>fs.swift.service.PROVIDER.username</code>,
+<code>fs.swift.service.PROVIDER.password</code> contains sensitive information and keeping them in
+<code>core-sites.xml</code> is not always a good approach.
--- End diff --
same here...,
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2298#issuecomment-54771815
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19959/consoleFull) for PR 2298 at commit [`dfb8fea`](https://github.com/apache/spark/commit/dfb8fea59caee1e9fed743d3df18075cba510172).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org