[GitHub] spark pull request #12004: [SPARK-7481] [build] Add spark-cloud module to pu...

[steveloughran <gi...@git.apache.org>]

Github user steveloughran commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12004#discussion_r114330941
  
    --- Diff: docs/cloud-integration.md ---
    @@ -0,0 +1,512 @@
    +---
    +layout: global
    +displayTitle: Integration with Cloud Infrastructures
    +title: Integration with Cloud Infrastructures
    +description: Introduction to cloud storage support in Apache Spark SPARK_VERSION_SHORT
    +---
    +<!---
    +  Licensed under the Apache License, Version 2.0 (the "License");
    +  you may not use this file except in compliance with the License.
    +  You may obtain a copy of the License at
    +
    +   http://www.apache.org/licenses/LICENSE-2.0
    +
    +  Unless required by applicable law or agreed to in writing, software
    +  distributed under the License is distributed on an "AS IS" BASIS,
    +  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  See the License for the specific language governing permissions and
    +  limitations under the License. See accompanying LICENSE file.
    +-->
    +
    +* This will become a table of contents (this text will be scraped).
    +{:toc}
    +
    +## <a name="introduction"></a>Introduction
    +
    +
    +All the public cloud infrastructures, Amazon AWS, Microsoft Azure, Google GCS and others offer
    +persistent data storage systems, "object stores". These are not quite the same as classic file
    +systems: in order to scale to hundreds of Petabytes, without any single points of failure
    +or size limits, object stores, "blobstores", have a simpler model of `name => data`.
    +
    +Apache Spark can read or write data in object stores for data access.
    +through filesystem connectors implemented in Apache Hadoop or provided by third-parties.
    +These libraries make the object stores look *almost* like filesystems, with directories and
    +operations on files (rename) and directories (create, rename, delete) which mimic
    +those of a classic filesystem. Because of this, Spark and Spark-based applications
    +can work with object stores, generally treating them as as if they were slower-but-larger filesystems.
    +
    +With these connectors, Apache Spark supports object stores as the source
    +of data for analysis, including Spark Streaming and DataFrames.
    +
    +
    +## <a name="quick_start"></a>Quick Start
    +
    +Provided the relevant libraries are on the classpath, and Spark is configured with your credentials,
    +objects in an object store can be can be read or written through URLs which uses the name of the
    +object store client as the schema and the bucket/container as the hostname.
    +
    +
    +### Dependencies
    +
    +The Spark application neeeds the relevant Hadoop libraries, which can
    +be done by including the `spark-hadoop-cloud` module for the specific version of spark used.
    +
    +The Spark application should include <code>hadoop-openstack</code> dependency, which can
    +be done by including the `spark-hadoop-cloud` module for the specific version of spark used.
    +For example, for Maven support, add the following to the <code>pom.xml</code> file:
    +
    +{% highlight xml %}
    +<dependencyManagement>
    +  ...
    +  <dependency>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-hadoop-cloud_2.11</artifactId>
    +    <version>${spark.version}</version>
    +  </dependency>
    +  ...
    +</dependencyManagement>
    +{% endhighlight %}
    +
    +If using the Scala 2.10-compatible version of Spark, the artifact is of course `spark-hadoop-cloud_2.10`.
    +
    +### Basic Use
    +
    +You can refer to data in an object store just as you would data in a filesystem, by
    +using a URL to the data in methods like `SparkContext.textFile()` to read data, 
    +`saveAsTextFile()` to write it back.
    +
    +
    +Because object stores are viewed by Spark as filesystems, object stores can
    +be used as the source or destination of any spark work —be it batch, SQL, DataFrame,
    --- End diff --
    
    sentence removed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org