You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by an...@apache.org on 2013/08/23 19:17:55 UTC
svn commit: r1516937 [2/3] - in /incubator/spark: ./ _includes/ _layouts/
_plugins/ css/ images/ news/ news/_posts/ releases/ releases/_posts/
screencasts/ screencasts/_posts/ site/ talks/
Added: incubator/spark/documentation.md
URL: http://svn.apache.org/viewvc/incubator/spark/documentation.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/documentation.md (added)
+++ incubator/spark/documentation.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,84 @@
+---
+layout: global
+title: Documentation
+type: "page singular"
+navigation:
+ weight: 3
+ show: true
+---
+
+<h2>Spark Documentation</h2>
+
+<p>Setup instructions, programming guides, and other documentation are available for each version of Spark below:</p>
+
+<ul>
+ <li><a href="{{site.url}}docs/latest/">Spark 0.7.3 (latest release)</a></li>
+ <li><a href="{{site.url}}docs/0.6.2/">Spark 0.6.2</a></li>
+ <li><a href="https://github.com/mesos/spark/wiki/Spark-0.5-Documentation">Spark 0.5.x</a> (hosted on GitHub)</li>
+</ul>
+
+<p>Read these documents to get started with Spark. In addition, this page lists a few other external resources for learning Spark.</p>
+
+<h3>Video Tutorials</h3>
+
+<ul>
+ <li><a href="{{site.url}}screencasts/1-first-steps-with-spark.html">Screencast 1: First Steps with Spark</a></li>
+ <li><a href="{{site.url}}screencasts/2-spark-documentation-overview.html">Screencast 2: Spark Documentation Overview</a></li>
+</ul>
+
+<h3>Hands-On Exercises</h3>
+
+<ul>
+ <li><a href="http://ampcamp.berkeley.edu/big-data-mini-course/">Hands-on exercises from Strata 2013</a> are available online. These exercises let you launch a small EC2 cluster, load a dataset, and query it with Spark, Shark, and Spark Streaming.</li>
+</ul>
+
+<h3>AMP Camp Slides and Videos</h3>
+
+<ul>
+ <li>The AMP Lab hosted a <a href="http://ampcamp.berkeley.edu/agenda-2012/">two-day training camp</a> on Spark, Shark and "big data" analysis in August 2012. <a href="http://ampcamp.berkeley.edu/agenda-2012/">Slides and videos</a> are available online for free.
+ </li>
+</ul>
+
+<h3>Presentations</h3>
+
+<ul>
+ <li>
+ <a href="{{site.url}}talks/overview.pptx">Overview of Spark</a> (<a href="{{site.url}}talks/overview.pptx">pptx</a>) (<a href="/talks/overview.pdf">pdf</a>)
+ </li>
+ <li>Programming Spark tutorial at AMP Camp: <a href="http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/matei-zaharia-part-1-amp-camp-2012-spark-intro.pdf">part 1 (pdf)</a>, <a href="http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/matei-zaharia-part-2-amp-camp-2012-standalone-programs.pdf">part 2 (pdf)</a>, <a href="http://www.youtube.com/watch?v=7k4yDKBYOcw">video</a>.
+ </li>
+ <li><a href="{{site.url}}talks/strata_spark_streaming.ppt">Overview of Spark Streaming</a> (<a href="{{site.url}}talks/strata_spark_streaming.ppt">ppt</a>) (<a href="{{site.url}}talks/strata_spark_streaming.pdf">pdf</a>)
+</ul>
+
+<h3>External Tutorials, Development Blogs, and Talks</h3>
+
+<ul>
+ <li><a href="http://www.slideshare.net/EvanChan2/cassandra2013-spark-talk-final">Real-time Analytics with Cassandra, Spark, and Shark</a> -- presentation by Evan Chan from Ooyala at the 2013 Cassandra Summit</li>
+ <li><a href="http://syndeticlogic.net/?p=311">Getting Spark Setup in Eclipse</a> -- Developer blog post by Spark community member James Percent</li>
+ <li><a href="http://aws.amazon.com/articles/Elastic-MapReduce/4926593393724923">Run Spark and Shark on Amazon Elastic MapReduce</a> -- Article by Amazon AWS Elastic MapReduce team member Parviz Deyhim</li>
+ <li><a href="http://blog.quantifind.com/posts/spark-unit-test/">Unit testing with Spark</a> -- Quantifind tech blog post by Imran Rashid</li>
+ <li><a href="http://blog.quantifind.com/posts/logging-post/">Configuring Spark logs</a> -- Quantifind tech blog by Imran Rashid</li>
+ <li><a href="http://www.ibm.com/developerworks/library/os-spark/">Spark, an alternative for fast data analytics</a> -- IBM Developer Works article by M. Tim Jones</li>
+</ul>
+
+<h3>Research Papers</h3>
+
+<ul>
+ <li>
+ <a href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-214.pdf">Shark: SQL and Rich Analytics at Scale</a>. Reynold Xin, Joshua Rosen, Matei Zaharia, Michael J. Franklin, Scott Shenker, Ion Stoica. <em>Technical Report UCB/EECS-2012-214</em>. November 2012.
+ </li>
+ <li>
+ <a href="http://www.cs.berkeley.edu/~matei/papers/2012/hotcloud_spark_streaming.pdf">Discretized Streams: An Efficient and Fault-Tolerant Model for Stream Processing on Large Clusters</a>. Matei Zaharia, Tathagata Das, Haoyuan Li, Scott Shenker, Ion Stoica. <em>HotCloud 2012</em>. June 2012.
+ </li>
+ <li>
+ <a href="http://www.cs.berkeley.edu/~matei/papers/2012/sigmod_shark_demo.pdf">Shark: Fast Data Analysis Using Coarse-grained Distributed Memory</a> (demo). Cliff Engle, Antonio Lupher, Reynold Xin, Matei Zaharia, Haoyuan Li, Scott Shenker, Ion Stoica. <em>SIGMOD 2012</em>. May 2012. <b>Best Demo Award</b>.
+ </li>
+ <li>
+ <a href="http://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf">Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing</a>. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion Stoica. <em>NSDI 2012</em>. April 2012. <b>Best Paper Award</b> and <b>Honorable Mention for Community Award</b>.
+ </li>
+ <li>
+ <a href="http://www.cs.berkeley.edu/~matei/papers/2011/tr_spark.pdf">Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing</a>. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion Stoica. <em>Technical Report UCB/EECS-2011-82</em>. July 2011.</li>
+ <li>
+ <a href="http://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf">Spark: Cluster Computing with Working Sets</a>. Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, Ion Stoica. <em>HotCloud 2010</em>. June 2010.
+ </li>
+</ul>
\ No newline at end of file
Added: incubator/spark/downloads.md
URL: http://svn.apache.org/viewvc/incubator/spark/downloads.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/downloads.md (added)
+++ incubator/spark/downloads.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,31 @@
+---
+layout: global
+title: Downloads
+type: singular
+navigation:
+ weight: 3
+ show: true
+---
+
+<h2>Download Spark</h2>
+The latest release of Spark is 0.7.3. You can either download it as a <a href="/download/spark-0.7.3-sources.tgz">source package</a> (4 MB tar.gz) or get prebuilt packages for <a href="/download/spark-0.7.3-prebuilt-hadoop1.tgz">Hadoop 1 / CDH3</a> or <a href="/download/spark-0.7.3-prebuilt-cdh4.tgz">CDH 4</a> (61 MB tar.gz).
+
+If you are interested in working with the newest under-development code or contributing to Spark development, you can also check out the master branch from Git: <tt>git clone git://github.com/mesos/spark.git</tt>.
+
+Once you've downloaded Spark, you can find instructions for installing and building it on the <a href="{{site.url}}documentation.html">documentation page</a>.
+<h3>Previous Releases</h3>
+<ul>
+ <li><a href="http://spark-project.org/download/spark-0.7.3-sources.tgz">Spark 0.7.3</a> (July 16, 2013) <a href="{{site.url}}releases/spark-release-0-7-3.html">(release notes)</a> (prebuilt: <a href="/download/spark-0.7.3-prebuilt-hadoop1.tgz">Hadoop 1 / CDH3</a>, <a href="{{site.url}}download/spark-0.7.3-prebuilt-cdh4.tgz">CDH 4</a>)</li>
+ <li><a href="http://spark-project.org/download/spark-0.7.2-sources.tgz">Spark 0.7.2</a> (June 2, 2013) <a href="{{site.url}}releases/spark-release-0-7-2.html">(release notes)</a> (prebuilt: <a href="/download/spark-0.7.2-prebuilt-hadoop1.tgz">Hadoop 1 / CDH3</a>, <a href="/download/spark-0.7.2-prebuilt-cdh4.tgz">CDH 4</a>)</li>
+ <li><a href="http://spark-project.org/download/spark-0.7.0-sources.tgz">Spark 0.7.0</a> (February 27, 2013) <a href="{{site.url}}releases/spark-release-0-7-0.html">(release notes)</a></li>
+ <li><a href="http://spark-project.org/download/spark-0.6.2-sources.tgz">Spark 0.6.2</a> (February 7, 2013) <a href="{{site.url}}releases/spark-release-0-6-2.html">(release notes)</a></li>
+ <li><a href="http://spark-project.org/download-spark-0.6.1-sources-tgz">Spark 0.6.1</a> (November 22, 2012) <a href="{{site.url}}releases/spark-release-0-6-1.html">(release notes)</a></li>
+ <li><a href="http://spark-project.org/download-spark-0.5.2-sources-tgz">Spark 0.5.2</a> (November 22, 2012) <a href="{{site.url}}releases/spark-release-0-5-2.html">(release notes)</a></li>
+ <li><a href="http://spark-project.org/download-spark-0.6.0-sources-tgz">Spark 0.6.0</a> (October 15, 2012) <a href="{{site.url}}releases/spark-release-0-6-0.html">(release notes)</a></li>
+ <li><a href="http://spark-project.org/download-spark-0.5.1-sources-tgz">Spark 0.5.1</a> (October 11, 2012) <a href="{{site.url}}releases/spark-release-0-5-1.html">(release notes)</a></li>
+ <li><a href="http://spark-project.org/download-spark-0.5.0-sources-tgz">Spark 0.5.0</a> (June 12, 2012) <a href="{{site.url}}releases/spark-release-0-5-0.html">(release notes)</a></li>
+ <li><a href="http://spark-project.org/download-spark-0.3-for-scala-2-9-sources-tgz">Spark 0.3 for Scala 2.9</a> (July 14, 2011) <a href="{{site.url}}releases/spark-release-0-3.html">(release notes)</a></li>
+ <li><a href="http://spark-project.org/download-spark-0.3-for-scala-2-8-sources-tgz">Spark 0.3 for Scala 2.8</a> (July 14, 2011) <a href="{{site.url}}releases/spark-release-0-3.html">(release notes)</a></li>
+ <li><a href="http://spark-project.org/download-spark-0.2-sources-tgz">Spark 0.2</a> (May 24, 2011)</li>
+</ul>
+
Added: incubator/spark/examples.md
URL: http://svn.apache.org/viewvc/incubator/spark/examples.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/examples.md (added)
+++ incubator/spark/examples.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,99 @@
+---
+layout: global
+title: Examples
+type: "page singular"
+navigation:
+ weight: 4
+ show: true
+---
+<h2>Spark Examples</h2>
+
+Spark is built around <em>distributed datasets</em> that support types of parallel operations: transformations, which are lazy and yield another distributed dataset (e.g., <code>map</code>, <code>filter</code>, and <code>join</code>), and actions, which force the computation of a dataset and return a result (e.g., <code>count</code>). The following examples show off some of the available operations and features.
+
+<h3>Text Search</h3>
+
+In this example, we search through the error messages in a log file:
+
+<p>
+</p><div class="code">
+<span class="keyword">val</span> file = spark.textFile(<span class="string">"hdfs://..."</span>)<br>
+<span class="keyword">val</span> errors = file.<span class="sparkop">filter</span>(<span class="closure">line => line.contains("ERROR")</span>)<br>
+<span class="comment">// Count all the errors</span><br>
+errors.<span class="sparkop">count</span>()<br>
+<span class="comment">// Count errors mentioning MySQL</span><br>
+errors.<span class="sparkop">filter</span>(<span class="closure">line => line.contains("MySQL")</span>).<span class="sparkop">count</span>()<br>
+<span class="comment">// Fetch the MySQL errors as an array of strings</span><br>
+errors.<span class="sparkop">filter</span>(<span class="closure">line => line.contains("MySQL")</span>).<span class="sparkop">collect</span>()<br>
+</div>
+<p></p>
+
+<p>The red code fragments are Scala function literals (closures) that get passed automatically to the cluster. The blue ones are Spark operations.</p>
+
+<h3>In-Memory Text Search</h3>
+
+<p>Spark can <em>cache</em> datasets in memory to speed up reuse. In the example above, we can load just the error messages in RAM using:</p>
+
+<p>
+</p><div class="code">
+errors.<span class="sparkop">cache</span>()
+</div>
+<p></p>
+
+<p>After the first action that uses <code>errors</code>, later ones will be much faster.</p>
+
+
+<h3>Word Count</h3>
+
+<p>In this example, we use a few more transformations to build a dataset of (String, Int) pairs called <code>counts</code> and then save it to a file.</p>
+
+<p>
+</p><div class="code">
+<span class="keyword">val</span> file = spark.textFile(<span class="string">"hdfs://..."</span>)<br>
+<span class="keyword">val</span> counts = file.<span class="sparkop">flatMap</span>(<span class="closure">line => line.split(" ")</span>)<br>
+ .<span class="sparkop">map</span>(<span class="closure">word => (word, 1)</span>)<br>
+ .<span class="sparkop">reduceByKey</span>(<span class="closure">_ + _</span>)<br>
+counts.<span class="sparkop">saveAsTextFile</span>(<span class="string">"hdfs://..."</span>)
+</div>
+<p></p>
+
+<h3>Estimating Pi</h3>
+
+<p>Spark can also be used for compute-intensive tasks. This code estimates <span style="font-family: serif; font-size: 120%;">Ï</span> by "throwing darts" at a circle. We pick random points in the unit square ((0, 0) to (1,1)) and see how many fall in the unit circle. The fraction should be <span style="font-family: serif; font-size: 120%;">Ï / 4</span>, so we use this to get our estimate.</p>
+
+<p>
+</p><div class="code">
+<span class="keyword">val</span> count = spark.parallelize(1 to NUM_SAMPLES).<span class="sparkop">map</span>(<span class="closure">i =><br>
+ <span class="keyword">val</span> x = Math.random<br>
+ <span class="keyword">val</span> y = Math.random<br>
+ <span class="keyword">if</span> (x*x + y*y < 1) 1.0 <span class="keyword">else</span> 0.0<br>
+</span>).<span class="sparkop">reduce</span>(<span class="closure">_ + _</span>)<br>
+println(<span class="string">"Pi is roughly "</span> + 4 * count / NUM_SAMPLES)<br>
+</div>
+<p></p>
+
+<h3>Logistic Regression</h3>
+
+<p>This is an iterative machine learning algorithm that seeks to find the best hyperplane that separates two sets of points in a multi-dimensional feature space. It can be used to classify messages into spam vs non-spam, for example. Because the algorithm applies the same MapReduce operation repeatedly to the same dataset, it benefits greatly from caching the input data in RAM across iterations.</p>
+
+<p>
+</p><div class="code">
+<span class="keyword">val</span> points = spark.textFile(...).<span class="sparkop">map</span>(parsePoint).<span class="sparkop">cache</span>()<br>
+<span class="keyword">var</span> w = Vector.random(D) <span class="comment">// current separating plane</span><br>
+<span class="keyword">for</span> (i <- 1 to ITERATIONS) {<br>
+ <span class="keyword">val</span> gradient = points.<span class="sparkop">map</span>(<span class="closure">p =><br>
+ (1 / (1 + exp(-p.y*(w dot p.x))) - 1) * p.y * p.x<br>
+ </span>).<span class="sparkop">reduce</span>(<span class="closure">_ + _</span>)<br>
+ w -= gradient<br>
+}<br>
+println(<span class="string">"Final separating plane: "</span> + w)<br>
+</div>
+<p></p>
+
+<p>Note that <code>w</code> gets shipped automatically to the cluster with every <code>map</code> call.</p>
+
+<p>The graph below compares the performance of this Spark program against a Hadoop implementation on 30 GB of data on an 80-core cluster, showing the benefit of in-memory caching:</p>
+
+<p align="center">
+<img src="{{site.url}}images/spark-lr.png" alt="Logistic regression performance in Spark vs Hadoop">
+</p>
+
Added: incubator/spark/faq.md
URL: http://svn.apache.org/viewvc/incubator/spark/faq.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/faq.md (added)
+++ incubator/spark/faq.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,39 @@
+---
+layout: global
+title: FAQ
+type: "page singular"
+navigation:
+ weight: 7
+ show: true
+---
+<h2>Spark FAQ</h2>
+
+<p class="question">Is Spark a modified version of Hadoop?</p>
+<p class="answer">No. Spark is a completely separate codebase optimized for low latency, although it can load data from any Hadoop input source (InputFormat).</p>
+
+<p class="question">Which languages does Spark support?</p>
+<p class="answer">Starting in version 0.7, Spark supports Scala, Java and Python.</p>
+
+<p class="question">Does Spark require modified versions of Scala or Python?</p>
+<p class="answer">No. Spark requires no changes to Scala or compiler plugins. The Python API uses the standard CPython implementation, and can call into existing C libraries for Python such as NumPy.</p>
+
+<p class="question">What happens when a cached dataset does not fit in memory?</p>
+<p class="answer">Spark can either spill it to disk or recompute the partitions that don't fit in RAM each time they are requested. By default, it uses recomputation, but you can set a dataset's <a href="{{site.url}}docs/latest/scala-programming-guide.html#rdd-persistence">storage level</a> to <tt>MEMORY_AND_DISK</tt> to avoid this. </p>
+
+<p class="question">How can I run Spark on a cluster?</p>
+<p class="answer">You can use either the <a href="{{site.url}}docs/latest/spark-standalone.html">standalone deploy mode</a>, which depends only on Java, or the <a href="{{site.url}}docs/latest/running-on-mesos.html">Apache Mesos</a> cluster manager.</p>
+<p>Note that you can also run Spark locally (possibly on multiple cores) without any special setup by just passing <tt>local[N]</tt> as the master URL, where <tt>N</tt> is the number of parallel threads you want.</p>
+
+<p class="question">I don't know Scala; how hard is it to pick it up to use Spark?</p>
+<p class="answer">Scala itself is pretty easy to pick up if you have Java experience. Check out <a href="http://www.artima.com/scalazine/articles/steps.html">First Steps to Scala</a> for a quick introduction, the <a href="http://www.scala-lang.org/docu/files/ScalaTutorial.pdf">Scala tutorial for Java programmers</a>, or the free online book <a href="http://www.artima.com/pins1ed/">Programming in Scala</a>.</p>
+<p>Spark 0.6 also added a <a href="{{site.url}}docs/latest/java-programming-guide.html">Java API</a>, letting you use Spark from Java, and Spark 0.7 added a <a href="{{site.url}}docs/latest/python-programming-guide.html">Python API</a>.</p>
+
+<p class="question">What license is Spark under?</p>
+<p class="answer">Spark is open source under the liberal <a href="https://github.com/mesos/spark/blob/master/LICENSE">BSD license</a>.</p>
+
+<p class="question">How can I contribute to Spark?</p>
+<p class="answer">Contact the <a href="http://groups.google.com/group/spark-users">mailing list</a> or send us a pull request on GitHub. We're glad to hear about your experience using Spark and to accept patches </p>
+<p>If you would like to report an issue, post it to the <a href="https://spark-project.atlassian.net/browse/SPARK">Spark issue tracker</a>.</p>
+
+<p class="question">Where can I get more help?</p>
+<p class="answer">Please post on the <a href="http://groups.google.com/group/spark-users">spark-users</a> mailing list. We'll be glad to help!</p>
\ No newline at end of file
Added: incubator/spark/favicon.ico
URL: http://svn.apache.org/viewvc/incubator/spark/favicon.ico?rev=1516937&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/spark/favicon.ico
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/spark/images/amplab-small.png
URL: http://svn.apache.org/viewvc/incubator/spark/images/amplab-small.png?rev=1516937&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/spark/images/amplab-small.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/spark/images/download.png
URL: http://svn.apache.org/viewvc/incubator/spark/images/download.png?rev=1516937&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/spark/images/download.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/spark/images/spark-lr.png
URL: http://svn.apache.org/viewvc/incubator/spark/images/spark-lr.png?rev=1516937&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/spark/images/spark-lr.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/spark/images/spark-project-header1.png
URL: http://svn.apache.org/viewvc/incubator/spark/images/spark-project-header1.png?rev=1516937&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/spark/images/spark-project-header1.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/spark/index.md
URL: http://svn.apache.org/viewvc/incubator/spark/index.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/index.md (added)
+++ incubator/spark/index.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,52 @@
+---
+layout: global
+title: Home
+type: page
+navigation:
+ weight: 1
+ show: true
+---
+## What is Spark?
+Spark is an open source cluster computing system that aims to make data analytics <em>fast</em> â both fast to run and fast to write.
+To run programs faster, Spark provides primitives for in-memory cluster computing: your job can load data into memory and query it repeatedly much more quickly than with disk-based systems like Hadoop MapReduce.
+To make programming faster, Spark provides clean, concise APIs in <a href="http://www.scala-lang.org" onclick="javascript:_gaq.push(['_trackEvent','outbound-article','http://www.scala-lang.org']);">Scala</a>, <a href="{{site.url}}docs/latest/quick-start.html#a-standalone-job-in-java" >Java</a> and <a href="{{site.url}}docs/latest/quick-start.html#a-standalone-job-in-python" >Python</a>. You can also use Spark interactively from the Scala and Python shells to rapidly query big datasets.
+
+## What can it do?
+Spark was initially developed for two applications where keeping data in memory helps: <em>iterative</em> algorithms, which are common in machine learning, and <em>interactive</em> data mining. In both cases, Spark can run up to <b>100x</b> faster than Hadoop MapReduce. However, you can use Spark for general data processing too. Check out our <a href="{{site.url}}examples.html" >example jobs</a>.
+Spark is also the engine behind <a href="http://shark.cs.berkeley.edu" onclick="javascript:_gaq.push(['_trackEvent','outbound-article','http://shark.cs.berkeley.edu']);">Shark</a>, a fully <a href="http://hive.apache.org" onclick="javascript:_gaq.push(['_trackEvent','outbound-article','http://hive.apache.org']);">Apache Hive</a>-compatible data warehousing system that can run 100x faster than Hive.
+While Spark is a new engine, it can access any data source supported by Hadoop, making it easy to run over existing data.
+
+## Who uses it?
+Spark was developed in the <a href="https://amplab.cs.berkeley.edu" onclick="javascript:_gaq.push(['_trackEvent','outbound-article','http://amplab.cs.berkeley.edu']);">UC Berkeley AMPLab</a>. It’s used by several groups of researchers at Berkeley to run large-scale applications such as spam filtering and traffic prediction. It’s also used to accelerate data analytics at <a href="http://www.yahoo.com" onclick="javascript:_gaq.push(['_trackEvent','outbound-article','http://www.yahoo.com']);">Yahoo!</a>, <a href="http://www.conviva.com" onclick="javascript:_gaq.push(['_trackEvent','outbound-article','http://www.conviva.com']);">Conviva</a>, <a href="http://www.quantifind.com" onclick="javascript:_gaq.push(['_trackEvent','outbound-article','http://www.quantifind.com']);">Quantifind</a>, and other companies — in total, 17 companies have contributed to Spark! Spark is <a href="https://github.com/mesos/spark" onclick="javascript:_gaq.push(['_trackEvent','outbound-article'
,'http://github.com']);">open source</a> under a BSD license, so <a href="{{site.url}}downloads.html" >download</a> it to check it out.
+
+## Apache Incubator notice
+Apache Spark is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+
+{% sidebar %}
+ <h3 class="widget-title">Latest News</h3>
+ <div class="latestnewswidget">
+ {% for post in site.categories.news limit:4 %}
+ <div><a href="{{ post.url }}">{{ post.title }}</a> <span class="post-info">({{post.date| date:"%B %d, %Y"}})</span></div>
+ {% endfor %}
+ </div>
+
+ <div style="text-align:right"><a href="{{site.url}}news/index.html">News Archive</a></div>
+
+ <!-- Not porting the following to Pygments since it becomes a lot less colorful -->
+
+ <div class="code" style="margin-top: 20px;">
+ file = spark.textFile(<span class="string">"hdfs://..."</span>)<br>
+ <br>
+ file.<span class="sparkop">flatMap</span>(<span class="closure">line => line.split(" ")</span>)<br>
+ .<span class="sparkop">map</span>(<span class="closure">word => (word, 1)</span>)<br>
+ .<span class="sparkop">reduceByKey</span>(<span class="closure">_ + _</span>)
+ </div>
+ <div class="caption">Word Count implemented in Spark</div>
+
+ <div align="center" style="margin-top: 20px;">
+ <img src="{{site.url}}images/spark-lr.png" alt="Logistic regression performance in Spark vs Hadoop">
+ </div>
+ <div class="caption">Logistic regression in Spark vs Hadoop</div>
+ <h2 style="text-align:center"><a href="{{site.url}}downloads"><img src="{{site.url}}images/download.png" alt="Download" style="vertical-align: middle"> Download Spark</a></h2>
+{% endsidebar %}
+
Added: incubator/spark/mailing-lists.md
URL: http://svn.apache.org/viewvc/incubator/spark/mailing-lists.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/mailing-lists.md (added)
+++ incubator/spark/mailing-lists.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,39 @@
+---
+layout: global
+title: Mailing Lists
+type: "page singular"
+navigation:
+ weight: 5
+ show: true
+---
+<h2>Spark Mailing Lists</h2>
+
+<p>Get help using Spark or contribute to the project on our mailing lists:</p>
+<ul>
+ <li>
+ <a href="http://mail-archives.apache.org/mod_mbox/incubator-spark-user/">user@spark.incubator.apache.org</a> is for usage questions, help, and announcements. <a href="mailto:user-subscribe@spark.incubator.apache.org?subject=(send%20this%20email%20to%20subscribe)">(subscribe)</a> <a href="http://mail-archives.apache.org/mod_mbox/incubator-spark-user/">(archives)</a>
+ </li>
+ <li>
+ <a href="http://mail-archives.apache.org/mod_mbox/incubator-spark-dev/">dev@spark.incubator.apache.org</a> is for people who want to contribute code to Spark. <a href="mailto:dev-subscribe@spark.incubator.apache.org?subject=(send%20this%20email%20to%20subscribe)">(subscribe)</a> <a href="http://mail-archives.apache.org/mod_mbox/incubator-spark-dev/">(archives)</a>
+ </li>
+</ul>
+
+<h2>User Meetups</h2>
+
+<p>We host a regular <a href="http://www.meetup.com/spark-users/">Bay Area Spark User Meetup</a> where you can meet the community in person. The meetup.com page also contains an <a href="http://www.meetup.com/spark-users/events/past/">archive of past meetups</a>, and <a href="http://www.meetup.com/spark-users/files/">slides</a> from them.</p>
+
+<h2>Issue Tracker</h2>
+
+<p>The project tracks bugs and new features on <a href="https://spark-project.atlassian.net/browse/SPARK">JIRA</a>. If you'd like, you can also subscribe to the <a href="http://groups.google.com/group/spark-issues">spark-issues</a> group to receive emails about new or updated issues.</p>
+
+<h2>Old Mailing Lists</h2>
+
+<p>The following Google Groups contain Spark usage and developer discussions from before the transition to Apache. They will become read-only in September 2013:</p>
+<ul>
+ <li>
+ <a href="http://groups.google.com/group/spark-users">spark-users</a> -- usage questions, help, and announcements.
+ </li>
+ <li>
+ <a href="http://groups.google.com/group/spark-developers">spark-developers</a> -- code contributor discussion.
+ </li>
+</ul>
Added: incubator/spark/news/_posts/2012-01-10-spark-meetups.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2012-01-10-spark-meetups.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2012-01-10-spark-meetups.html (added)
+++ incubator/spark/news/_posts/2012-01-10-spark-meetups.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,11 @@
+---
+layout: post
+title: We've started hosting a Bay Area Spark User Meetup
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+---
+We've started hosting a regular <a href="http://www.meetup.com/spark-users/">Bay Area Spark User Meetup</a>. Sign up on the meetup.com page to be notified about events and meet other Spark developers and users.
Added: incubator/spark/news/_posts/2012-04-25-nsdi-paper.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2012-04-25-nsdi-paper.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2012-04-25-nsdi-paper.html (added)
+++ incubator/spark/news/_posts/2012-04-25-nsdi-paper.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,14 @@
+---
+layout: post
+title: Spark wins Best Paper Award at USENIX NSDI
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '1'
+ _wp_old_slug: hello-world
+---
+Our <a href="http://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf">paper on Spark</a> won the Best Paper Award at the <a href="http://www.usenix.org/nsdi12/">USENIX NSDI conference</a>. You can see a video of the talk, as well as slides, online on the <a href="https://www.usenix.org/conference/nsdi12/resilient-distributed-datasets-fault-tolerant-abstraction-memory-cluster-computing">NSDI website</a>.
Added: incubator/spark/news/_posts/2012-10-15-spark-version-0-6-0-released.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2012-10-15-spark-version-0-6-0-released.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2012-10-15-spark-version-0-6-0-released.html (added)
+++ incubator/spark/news/_posts/2012-10-15-spark-version-0-6-0-released.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,13 @@
+---
+layout: post
+title: Spark version 0.6.0 released
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '1'
+---
+<a href="/spark-release-0-6-0">Spark version 0.6.0</a> was released today, a major release that brings a wide range of performance improvements and new features, including a simpler standalone deploy mode and a Java API. Read more about it in the <a href="/spark-release-0-6-0">release notes</a>.
Added: incubator/spark/news/_posts/2012-11-22-spark-0-6-1-and-0-5-2-released.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2012-11-22-spark-0-6-1-and-0-5-2-released.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2012-11-22-spark-0-6-1-and-0-5-2-released.html (added)
+++ incubator/spark/news/_posts/2012-11-22-spark-0-6-1-and-0-5-2-released.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,13 @@
+---
+layout: post
+title: Spark 0.6.1 and 0.5.2 out
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '1'
+---
+Today we've made available two maintenance releases for Spark: <a href="http://spark-project.org/spark-release-0-6-1/" title="Spark Release 0.6.1">0.6.1</a> and <a href="http://spark-project.org/spark-release-0-5-2/" title="Spark Release 0.5.2">0.5.2</a>. They both contain important bug fixes as well as some new features, such as the ability to build against Hadoop 2 distributions. We recommend that users update to the latest version for their branch; for new users, we recommend <a href="http://spark-project.org/spark-release-0-6-1/" title="Spark Release 0.6.1">0.6.1</a>.
Added: incubator/spark/news/_posts/2012-12-21-spark-and-shark-in-the-news.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2012-12-21-spark-and-shark-in-the-news.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2012-12-21-spark-and-shark-in-the-news.html (added)
+++ incubator/spark/news/_posts/2012-12-21-spark-and-shark-in-the-news.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,25 @@
+---
+layout: post
+title: Spark and Shark in the news
+categories:
+- In-The-News
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+Recently, we've seen quite a bit of coverage of both Spark and <a href="http://shark.cs.berkeley.edu">Shark</a> in the news. I wanted to list some of the more recent articles, for readers interested in learning more.
+
+<ul>
+<li>Curt Monash, editor of the popular DBMS2 blog, wrote a great <a href="http://www.dbms2.com/2012/12/13/introduction-to-spark-shark-bdas-and-amplab/">introduction to Spark and Shark</a>, as well as a more detailed <a href="http://www.dbms2.com/2012/12/13/spark-shark-and-rdds-technology-notes/">technical overview</a>.</li>
+<li><a href="http://siliconangle.com/blog/2012/11/29/big-data-up-to-100x-faster-researchers-crank-up-the-speed-dial/">Silicon Angle</a> covered Spark and Shark after our <a href="https://amplab.cs.berkeley.edu/news/sparkshark-a-big-hit-at-aws-reinvent/">presentation at Amazon re:Invent</a>.</li>
+<li><a href="http://www.datanami.com/datanami/2012-12-01/the_week_in_big_data_research.html?page=2">Datanami</a> highlighted Shark in its survey of big data research projects.</li>
+<li>O'Reilly's Strata blog recently covered <a href="http://strata.oreilly.com/2012/08/seven-reasons-why-i-like-spark.html">Spark</a>, <a href="http://strata.oreilly.com/2012/11/shark-real-time-queries-and-analytics-for-big-data.html">Shark</a>, and the <a href="http://strata.oreilly.com/2012/10/spark-0-6-improves-performance-and-accessibility.html">Spark 0.6 release</a>.</li>
+<li><a href="http://data-informed.com/spark-an-open-source-engine-for-iterative-data-mining/">DataInformed</a> interviewed two Spark users and wrote about their applications in anomaly detection, predictive analytics and data mining.</li>
+</ul>
+
+In other news, there will be a full day of tutorials on Spark and Shark at the <a href="http://strataconf.com/strata2013">O'Reilly Strata conference</a> in February. They include a three-hour <a href="http://strataconf.com/strata2013/public/schedule/detail/27438">introduction to Spark, Shark and BDAS</a> Tuesday morning, and a three-hour <a href="http://strataconf.com/strata2013/public/schedule/detail/27440">hands-on exercise session</a>.
Added: incubator/spark/news/_posts/2012-12-21-video-from-first-spark-development-meetup.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2012-12-21-video-from-first-spark-development-meetup.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2012-12-21-video-from-first-spark-development-meetup.html (added)
+++ incubator/spark/news/_posts/2012-12-21-video-from-first-spark-development-meetup.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,14 @@
+---
+layout: post
+title: Video up from first Spark development meetup
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+On December 18th, we held the first of a series of Spark development meetups, for people interested in learning the Spark codebase and contributing to the project. There was quite a bit more demand than we anticipated, with over 80 people signing up and 64 attending. The first meetup was an <a href="http://www.meetup.com/spark-users/events/94101942/">introduction to Spark internals</a>. Thanks to one of the attendees, there's now a <a href="http://www.youtube.com/watch?v=49Hr5xZyTEA">video of the meetup</a> on YouTube. We've also posted the <a href="http://files.meetup.com/3138542/dev-meetup-dec-2012.pptx">slides</a>. Look to see more development meetups on Spark and Shark in the future.
Added: incubator/spark/news/_posts/2013-01-12-spark-tips-from-quantifind.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-01-12-spark-tips-from-quantifind.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-01-12-spark-tips-from-quantifind.html (added)
+++ incubator/spark/news/_posts/2013-01-12-spark-tips-from-quantifind.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,19 @@
+---
+layout: post
+title: Spark tips from Quantifind
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+<a href="http://www.quantfind.com">Quantfind</a>, one of the Bay Area companies that has been using Spark for predictive analytics, recently posted two useful entries on working with Spark in their tech blog:
+<ul>
+<li><a href="http://blog.quantifind.com/posts/spark-unit-test/">Unit testing with Spark</a></li>
+<li><a href="http://blog.quantifind.com/posts/logging-post/">Configuring Spark's logs</a></li>
+</ul>
+Thanks for sharing this, and looking forward to see others!
Added: incubator/spark/news/_posts/2013-02-07-spark-0-6-2-released.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-02-07-spark-0-6-2-released.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-02-07-spark-0-6-2-released.html (added)
+++ incubator/spark/news/_posts/2013-02-07-spark-0-6-2-released.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,14 @@
+---
+layout: post
+title: Spark 0.6.2 released
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+We recently released <a href="http://spark-project.org/spark-release-0-6-2/" title="Spark Release 0.6.2">Spark 0.6.2</a>, a new version of Spark. This is a maintenance release that includes several bug fixes and usability improvements (see the <a href="http://spark-project.org/spark-release-0-6-2/" title="Spark Release 0.6.2">release notes</a>). We recommend that all users upgrade to this release.
Added: incubator/spark/news/_posts/2013-02-24-run-spark-and-shark-on-amazon-emr.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-02-24-run-spark-and-shark-on-amazon-emr.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-02-24-run-spark-and-shark-on-amazon-emr.html (added)
+++ incubator/spark/news/_posts/2013-02-24-run-spark-and-shark-on-amazon-emr.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,14 @@
+---
+layout: post
+title: Spark/Shark Tutorial for Amazon EMR
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+This weekend, Amazon posted an <a href="http://aws.amazon.com/articles/Elastic-MapReduce/4926593393724923">article</a> and code that make it easy to launch Spark and Shark on Elastic MapReduce. The article includes examples of how to run both interactive Scala commands and SQL queries from Shark on data in S3. Head over to the <a href="http://aws.amazon.com/articles/Elastic-MapReduce/4926593393724923">Amazon article</a> for details. We're very excited because, to our knowledge, this makes Spark the first non-Hadoop engine that you can launch with EMR.
Added: incubator/spark/news/_posts/2013-02-27-spark-0-7-0-released.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-02-27-spark-0-7-0-released.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-02-27-spark-0-7-0-released.html (added)
+++ incubator/spark/news/_posts/2013-02-27-spark-0-7-0-released.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,14 @@
+---
+layout: post
+title: Spark 0.7.0 released
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+We're proud to announce the release of <a href="http://spark-project.org/spark-release-0-7-0/" title="Spark Release 0.7.0">Spark 0.7.0</a>, a new major version of Spark that adds several key features, including a <a href="/docs/latest/python-programming-guide.html">Python API</a> for Spark and an <a href="/docs/latest/streaming-programming-guide.html">alpha of Spark Streaming</a>. This release is the result of the largest group of contributors yet behind a Spark release -- 31 contributors from inside and outside Berkeley. Head over to the <a href="http://spark-project.org/spark-release-0-7-0/" title="Spark Release 0.7.0">release notes</a> to read more about the new features, or <a href="/downloads/">download</a> the release today.
Added: incubator/spark/news/_posts/2013-03-17-strata-exercises-now-available-online.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-03-17-strata-exercises-now-available-online.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-03-17-strata-exercises-now-available-online.html (added)
+++ incubator/spark/news/_posts/2013-03-17-strata-exercises-now-available-online.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,14 @@
+---
+layout: post
+title: Strata exercises now available online
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+At this year's <a href="http://strataconf.com/strata2013">Strata</a> conference, the AMP Lab hosted a full day of tutorials on Spark, Shark, and Spark Streaming, including online exercises on Amazon EC2. Those exercises are now <a href="http://ampcamp.berkeley.edu/big-data-mini-course/">available online</a>, letting you learn Spark and Shark at your own pace on an EC2 cluster with real data. They are a great resource for learning the systems. You can also find <a href="http://ampcamp.berkeley.edu/amp-camp-two-strata-2013/">slides</a> from the Strata tutorials online, as well as <a href="http://ampcamp.berkeley.edu/amp-camp-one-berkeley-2012/">videos</a> from the AMP Camp workshop we held at Berkeley in August.
Added: incubator/spark/news/_posts/2013-04-16-spark-screencasts-published.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-04-16-spark-screencasts-published.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-04-16-spark-screencasts-published.html (added)
+++ incubator/spark/news/_posts/2013-04-16-spark-screencasts-published.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,20 @@
+---
+layout: post
+title: Spark screencasts published
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '2'
+ _wpas_done_all: '1'
+---
+We have released the first two screencasts in a series of short hands-on video training courses we will be publishing to help new users get up and running with Spark in minutes.
+
+The first Spark screencast is called <a href="http://spark-project.org/screencast-1-first-steps-with-spark/">First Steps With Spark</a> and walks you through downloading and building Spark, as well as using the Spark shell, all in less than 10 minutes!
+
+The second screencast is a 2 minute <a href="http://spark-project.org/spark-documentation-overview-screencast-2/">overview of the Spark documentation</a>.
+
+We hope you find these screencasts useful.
Added: incubator/spark/news/_posts/2013-06-02-spark-0-7-2-released.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-06-02-spark-0-7-2-released.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-06-02-spark-0-7-2-released.html (added)
+++ incubator/spark/news/_posts/2013-06-02-spark-0-7-2-released.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,14 @@
+---
+layout: post
+title: Spark 0.7.2 released
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+We're happy to announce the release of <a href="http://spark-project.org/spark-release-0-7-2/" title="Spark Release 0.7.2">Spark 0.7.2</a>, a new maintenance release that includes several bug fixes and improvements, as well as new code examples and API features. We recommend that all users update to this release. Head over to the <a href="http://spark-project.org/spark-release-0-7-2/" title="Spark Release 0.7.2">release notes</a> to read about the new features, or <a href="/downloads/">download</a> the release today.
Added: incubator/spark/news/_posts/2013-06-21-spark-accepted-into-apache-incubator.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-06-21-spark-accepted-into-apache-incubator.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-06-21-spark-accepted-into-apache-incubator.html (added)
+++ incubator/spark/news/_posts/2013-06-21-spark-accepted-into-apache-incubator.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,14 @@
+---
+layout: post
+title: Spark accepted into Apache Incubator
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+Spark was recently <a href="http://mail-archives.apache.org/mod_mbox/incubator-general/201306.mbox/%3CCDE7B773.E9A48%25chris.a.mattmann%40jpl.nasa.gov%3E">accepted</a> into the <a href="http://incubator.apache.org">Apache Incubator</a>, which will serve as the long-term home for the project. While moving the source code and issue tracking to Apache will take some time, we are excited to be joining the community at Apache. Stay tuned on this site for updates on how the project hosting will change.
Added: incubator/spark/news/_posts/2013-06-21-spark-featured-in-wired.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-06-21-spark-featured-in-wired.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-06-21-spark-featured-in-wired.html (added)
+++ incubator/spark/news/_posts/2013-06-21-spark-featured-in-wired.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,15 @@
+---
+layout: post
+title: Spark featured in Wired
+categories:
+- In-The-News
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+Spark, its creators at the AMP Lab, and some of its users were featured in a <a href="http://www.wired.com/wiredenterprise/2013/06/yahoo-amazon-amplab-spark/all/">Wired Enterprise article</a> a few days ago. Read on to learn a little about how Spark is being used in industry.
Added: incubator/spark/news/_posts/2013-07-16-spark-0-7-3-released.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-07-16-spark-0-7-3-released.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-07-16-spark-0-7-3-released.html (added)
+++ incubator/spark/news/_posts/2013-07-16-spark-0-7-3-released.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,14 @@
+---
+layout: post
+title: Spark 0.7.3 released
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+We've just posted <a href="http://spark-project.org/spark-release-0-7-3/" title="Spark Release 0.7.3">Spark Release 0.7.3</a>, a maintenance release that contains several fixes, including streaming API updates and new functionality for adding JARs to a <code>spark-shell</code> session. We recommend that all users update to this release. Visit the <a href="http://spark-project.org/spark-release-0-7-3/" title="Spark Release 0.7.3">release notes</a> to read about the new features, or <a href="/downloads/">download</a> the release today.
Added: incubator/spark/news/_posts/2013-07-22-spark-mailing-lists-moving-to-apache.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-07-22-spark-mailing-lists-moving-to-apache.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-07-22-spark-mailing-lists-moving-to-apache.html (added)
+++ incubator/spark/news/_posts/2013-07-22-spark-mailing-lists-moving-to-apache.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,25 @@
+---
+layout: post
+title: Spark mailing lists moving to Apache
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+<p>As part of the Spark project's recent move to Apache, we are planning to migrate the mailing lists to Apache infrastructure this month, so that <strong>the existing Google groups will become read-only on September 1, 2013</strong>. To keep receiving updates about Spark or to participate in development discussions, please subscribe to the following lists:</p>
+
+<ul>
+ <li>
+ <a href="http://mail-archives.apache.org/mod_mbox/incubator-spark-user/">user@spark.incubator.apache.org</a> -- for usage questions, help, and announcements. <a href="mailto:user-subscribe@spark.incubator.apache.org?subject=(send%20this%20email%20to%20subscribe)">(subscribe)</a> <a href="http://mail-archives.apache.org/mod_mbox/incubator-spark-user/">(archives)</a>
+ </li>
+ <li>
+ <a href="http://mail-archives.apache.org/mod_mbox/incubator-spark-dev/">dev@spark.incubator.apache.org</a> -- for people who want to contribute code to Spark. <a href="mailto:dev-subscribe@spark.incubator.apache.org?subject=(send%20this%20email%20to%20subscribe)">(subscribe)</a> <a href="http://mail-archives.apache.org/mod_mbox/incubator-spark-dev/">(archives)</a>
+ </li>
+</ul>
+
+<p>Most users will probably want the User list, but individuals interested in contributing code to the project should also subscribe to the Dev list.</p>
Added: incubator/spark/news/_posts/2013-07-23-amp-camp-2013-registration-ope.html
URL: http://svn.apache.org/viewvc/incubator/spark/news/_posts/2013-07-23-amp-camp-2013-registration-ope.html?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/_posts/2013-07-23-amp-camp-2013-registration-ope.html (added)
+++ incubator/spark/news/_posts/2013-07-23-amp-camp-2013-registration-ope.html Fri Aug 23 17:17:53 2013
@@ -0,0 +1,14 @@
+---
+layout: post
+title: Registration open for AMP Camp training camp in Berkeley
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+Want to learn how to use Spark, Shark, GraphX, and related technologies in person? The AMP Lab is hosting a two-day training workshop for them on August 29th and 30th in Berkeley. The workshop will include tutorials, talks from users, and over four hours of hands-on exercises. <a href="http://ampcamp.berkeley.edu/amp-camp-three-berkeley-2013/">Registration is now open on the AMP Camp website</a>, for a price of $250 per person. We recommend signing up early because last year's workshop was sold out.
Added: incubator/spark/news/index.md
URL: http://svn.apache.org/viewvc/incubator/spark/news/index.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/news/index.md (added)
+++ incubator/spark/news/index.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,15 @@
+---
+layout: global
+type: archive
+title: Spark News Archive
+---
+## Spark News Archive
+{% for post in site.categories.news %}
+ <article class="hentry">
+ <header class="entry-header">
+ <h1 class="entry-title"><a href="{{ post.url }}">{{ post.title }}</a></h1>
+ <div class="entry-meta">{{post.date | date: "%B %d, %Y"}}</div>
+ </header>
+ <div class="entry-content">{{post.excerpt}}</div>
+ </article>
+{% endfor %}
Added: incubator/spark/releases/_posts/2011-07-14-spark-release-0-3.md
URL: http://svn.apache.org/viewvc/incubator/spark/releases/_posts/2011-07-14-spark-release-0-3.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/releases/_posts/2011-07-14-spark-release-0-3.md (added)
+++ incubator/spark/releases/_posts/2011-07-14-spark-release-0-3.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,62 @@
+---
+layout: post
+title: Spark Release 0.3
+categories:
+- Releases
+tags: []
+status: publish
+type: post
+published: true
+---
+Spark 0.3 brings a variety of new features. You can download it for either <a href="https://github.com/mesos/spark/tarball/0.3-scala-2.9">Scala 2.9</a> or <a href="https://github.com/mesos/spark/tarball/0.3-scala-2.8">Scala 2.8</a>.
+
+<h3>Scala 2.9 Support</h3>
+
+This is the first release to support Scala 2.9 in addition to 2.8. Future releases are likely to be 2.9-only unless there is high demand for 2.8.
+
+<h3>Save Operations</h3>
+
+You can now save distributed datasets to the Hadoop filesystem (HDFS), Amazon S3, Hypertable, and any other storage system supported by Hadoop. There are convenience methods for several common formats, like text files and SequenceFiles. For example, to save a dataset as text:
+
+<div class="code">
+<span class="keyword">val</span> numbers = spark.parallelize(1 to 100)<br> numbers.<span class="sparkop">saveAsTextFile</span>(<span class="string">"hdfs://..."</span>)
+</div>
+
+<h3>Native Types for SequenceFiles</h3>
+
+In working with SequenceFiles, which store objects that implement Hadoop's Writable interface, Spark will now let you use native types for certain common Writable types, like IntWritable and Text. For example:
+
+<div class="code">
+<span class="comment">// Will read a SequenceFile of (IntWritable, Text)</span><br>
+<span class="keyword">val</span> data = spark.sequenceFile[Int, String](<span class="string">"hdfs://..."</span>)
+</div>
+
+Similarly, you can save datasets of basic types directly as SequenceFiles:
+
+<div class="code">
+<span class="comment">// Will write a SequenceFile of (IntWritable, IntWritable)</span><br>
+<span class="keyword">val</span> squares = spark.parallelize(1 to 100).<span class="sparkop">map</span>(<span class="closure">n => (n, n*n)</span>)<br>
+squares.saveAsSequenceFile(<span class="string">"hdfs://..."</span>)
+</div>
+
+<h3>Maven Integration</h3>
+
+Spark now fetches dependencies via Maven and can publish Maven artifacts for easier dependency management.
+
+<h3>Faster Broadcast & Shuffle</h3>
+
+This release includes broadcast and shuffle algorithms from <a href="http://www.mosharaf.com/wp-content/uploads/orchestra-sigcomm11.pdf">this paper</a> to better support applications that communicate large amounts of data.
+
+<h3>Support for Non-Filesystem Hadoop Input Formats</h3>
+
+The new <tt>SparkContext.hadoopRDD</tt> method allows reading data from Hadoop-compatible storage systems other than file systems, such as HBase, Hypertable, etc.
+
+<h3>Other Features</h3>
+
+<ul>
+ <li>Outer join operators (<tt>leftOuterJoin</tt>, <tt>rightOuterJoin</tt>, etc).</li>
+ <li>Support for Scala 2.9 interpreter features (history search, Ctrl-C current line, etc) in the 2.9 version.</li>
+ <li>Better default levels of parallelism for various operations.</li>
+ <li>Ability to control number of splits in a file.</li>
+ <li>Various bug fixes.</li>
+</ul>
Added: incubator/spark/releases/_posts/2012-06-12-spark-release-0-5-0.md
URL: http://svn.apache.org/viewvc/incubator/spark/releases/_posts/2012-06-12-spark-release-0-5-0.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/releases/_posts/2012-06-12-spark-release-0-5-0.md (added)
+++ incubator/spark/releases/_posts/2012-06-12-spark-release-0-5-0.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,36 @@
+---
+layout: post
+title: Spark Release 0.5.0
+categories:
+- Releases
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '1'
+---
+Spark 0.5.0 brings several new features and sets the stage for some big changes coming this summer as we incorporate code from the <a href="http://www.cs.berkeley.edu/~matei/papers/2012/hotcloud_spark_streaming.pdf">Spark Streaming</a> project. You can download it as a <a href="https://github.com/mesos/spark/zipball/v0.5.0">zip</a> or <a href="https://github.com/mesos/spark/tarball/v0.5.0">tar.gz</a>.
+
+<h3>Mesos 0.9 Support</h3>
+
+This release runs on <a href="http://www.mesosproject.org/">Apache Mesos 0.9</a>, the first Apache Incubator release of Mesos, which contains significant usability and stability improvements. Most notable are better memory accounting for applications with long-term memory use, easier access of old jobs' traces and logs (by keeping a history of executed tasks on the web UI), and simpler installation.
+
+<h3>Performance Improvements</h3>
+Spark's scheduling is more communication-efficient when sending out operations on RDDs with large lineage graphs. In addition, the cache replacement policy has been improved to more smartly replace data when an RDD does not fit in the cache, shuffles are more efficient, and the serializer used for shipping closures is now configurable, making it possible to use faster libraries than Java serialization there.
+
+<h3>Debug Improvements</h3>
+
+Spark now reports exceptions on the worker nodes back to the master, so you can see them all in one log file. It also automatically marks and filters duplicate errors.
+
+<h3>New Operators</h3>
+
+These include <tt>sortByKey</tt> for parallel sorting, <tt>takeSample</tt>, and more efficient <tt>fold</tt> and <tt>aggregate</tt> operators. In addition, more of the old operators make use of, and retain, RDD partitioning information to reduce communication cost. For example, if you <tt>join</tt> two hash-partitioned RDDs that were partitioned in the same way, Spark will not shuffle any data across the network.
+
+<h3>EC2 Launch Script Improvements</h3>
+
+Spark's EC2 launch scripts are now included in the main package, and have the ability to discover and use the latest Spark AMI automatically instead of launching a hardcoded machine image ID.
+
+<h3>New Hadoop API Support</h3>
+
+You can now use Spark to read and write data to storage formats in the new <tt>org.apache.mapreduce</tt> packages (the "new Hadoop" API). In addition, this release fixes an issue caused by a HDFS initialization bug in some recent versions of HDFS.
Added: incubator/spark/releases/_posts/2012-10-11-spark-release-0-5-1.md
URL: http://svn.apache.org/viewvc/incubator/spark/releases/_posts/2012-10-11-spark-release-0-5-1.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/releases/_posts/2012-10-11-spark-release-0-5-1.md (added)
+++ incubator/spark/releases/_posts/2012-10-11-spark-release-0-5-1.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,46 @@
+---
+layout: post
+title: Spark Release 0.5.1
+categories:
+- Releases
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '1'
+---
+Spark 0.5.1 is a maintenance release that adds several important bug fixes and usability features. You can download it as a <a href="http://github.com/downloads/mesos/spark/spark-0.5.1.tgz">tar.gz file</a>.
+
+<h3>Maven Publishing</h3>
+
+Spark is now available in Maven Central, making it easier to link into your programs without having to build it as a JAR. Use the following Maven identifiers to add it to a project:
+<ul>
+ <li>groupId: org.spark-project</li>
+ <li>artifactId: spark-core_2.9.2</li>
+ <li>version: 0.5.1</li>
+</ul>
+
+<h3>Scala 2.9.2</h3>
+
+Spark now builds against Scala 2.9.2 by default.
+
+<h3>Improved Accumulators</h3>
+
+The new Accumulable class generalizes Accumulators for the case when the type being accumulated is not the same as the types of elements being added (e.g. you wish to accumulate a collection, such as a Set, by adding individual elements). This interface is also more efficient in avoiding the creation of temporary objects. (Contributed by Imran Rashid.)
+
+<h3>Bug Fixes</h3>
+
+<ul>
+ <li>Spark's algorithm for estimating the sizes of objects (in order to manage memory correctly) has been improved
+ to handle JVMs with 32- vs 64-bit pointers and to measure objects more accurately. (Contributed by Shivaram Venkataraman.)</li>
+ <li>Improved algorithms for taking random samples out of datasets to avoid biases that could occur in the previous ones. (Suggested by Henry Milner.)</li>
+ <li>Improved load balancing across nodes in sort operations.</li>
+ <li>Fixed a shuffle bug that could cause reduce tasks to fail to receive a map task's full output.</li>
+ <li>Fixed a bug with locating custom KryoSerializers.</li>
+ <li>Reduced memory consumption of <tt>saveAsObjectFile</tt> when objects are large.</li>
+</ul>
+
+<h3>EC2 Improvements</h3>
+
+Spark's EC2 launch script now configures Spark's memory limit automatically based on the machine's available RAM.
Added: incubator/spark/releases/_posts/2012-10-15-spark-release-0-6-0.md
URL: http://svn.apache.org/viewvc/incubator/spark/releases/_posts/2012-10-15-spark-release-0-6-0.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/releases/_posts/2012-10-15-spark-release-0-6-0.md (added)
+++ incubator/spark/releases/_posts/2012-10-15-spark-release-0-6-0.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,90 @@
+---
+layout: post
+title: Spark Release 0.6.0
+categories:
+- Releases
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+---
+Spark 0.6.0 is a major release that brings several new features, architectural changes, and performance enhancements. The most visible additions are a standalone deploy mode, a Java API, and expanded documentation; but there are also numerous other changes under the hood, which improve performance in some cases by as much as 2x.
+
+You can download this release as either a <a href="http://github.com/downloads/mesos/spark/spark-0.6.0-sources.tar.gz">source package</a> (2 MB tar.gz) or <a href="http://github.com/downloads/mesos/spark/spark-0.6.0-prebuilt.tar.gz">prebuilt package</a> (48 MB tar.gz)
+
+<h3>Simpler Deployment</h3>
+
+In addition to running on Mesos, Spark now has a <a href="/docs/0.6.0/spark-standalone.html">standalone deploy mode</a> that lets you quickly launch a cluster without installing an external cluster manager. The standalone mode only needs Java installed on each machine, and Spark deployed to it.
+
+In addition, there is <a href="/docs/0.6.0/running-on-yarn.html">experimental support for running on YARN</a> (Hadoop NextGen), currently in a separate branch.
+
+<h3>Java API</h3>
+
+Java programmers can now use Spark through a new <a href="/docs/0.6.0/java-programming-guide.html">Java API layer</a>. This layer makes available all of Spark's features, including parallel transformations, distributed datasets, broadcast variables, and accumulators, in a Java-friendly manner.
+
+<h3>Expanded Documentation</h3>
+
+Spark's <a href="/docs/0.6.0/">documentation</a> has been expanded with a new <a href="/docs/0.6.0/quick-start.html">quick start guide</a>, additional deployment instructions, configuration guide, tuning guide, and improved <a href="/docs/0.6.0/api/core">Scaladoc</a> API documentation.
+
+<h3>Engine Changes</h3>
+
+Under the hood, Spark 0.6 has new, custom storage and communication layers brought in from the upcoming <a href="http://www.cs.berkeley.edu/~matei/papers/2012/hotcloud_spark_streaming.pdf">Spark Streaming</a> project. These can improve performance over past versions by as much as 2x. Specifically:
+
+<ul>
+ <li>A new communication manager using asynchronous Java NIO lets shuffle operations run faster, especially when sending large amounts of data or when jobs have many tasks.</li>
+ <li>A new storage manager supports per-dataset storage level settings (e.g. whether to keep the dataset in memory, deserialized, on disk, etc, or even replicated across nodes).</li>
+ <li>Spark's scheduler and control plane have been optimized to better support ultra-low-latency jobs (under 500ms) and high-throughput scheduling decisions.</li>
+</ul>
+
+<h3>New APIs</h3>
+
+<ul>
+ <li>This release adds the ability to control caching strategies on a per-RDD level, so that different RDDs may be stored in memory, on disk, as serialized bytes, etc. You can choose your storage level using the <a href="/docs/0.6.0/scala-programming-guide.html#rdd-persistence"><tt>persist()</tt> method</a> on RDD.</li>
+ <li>A new Accumulable class generalizes Accumulators for the case when the type being accumulated is not the same as the types of elements being added (e.g. you wish to accumulate a collection, such as a Set, by adding individual elements).</li>
+ <li>You can now dynamically add files or JARs that should be shipped to your workers with <tt>SparkContext.addFile/Jar</tt>.</li>
+ <li>More Spark operators (e.g. joins) support custom partitioners.</li>
+</ul>
+
+<h3>Enhanced Debugging</h3>
+
+Spark's log now prints which operation in your program each RDD and job described in your logs belongs to, making it easier to tie back to which parts of your code experience problems.
+
+<h3>Maven Artifacts</h3>
+
+Spark is now available in Maven Central, making it easier to link into your programs without having to build it as a JAR. Use the following Maven identifiers to add it to a project:
+
+<ul>
+ <li>groupId: org.spark-project</li>
+ <li>artifactId: spark-core_2.9.2</li>
+ <li>version: 0.6.0</li>
+</ul>
+
+<h3>Compatibility</h3>
+
+This release is source-compatible with Spark 0.5 programs, but you will need to recompile them against 0.6. In addition, the configuration for caching has changed: instead of having a <tt>spark.cache.class</tt> parameter that sets one caching strategy for all RDDs, you can now set a <a href="/docs/0.6.0/scala-programming-guide.html#rdd-persistence">per-RDD storage level</a>. Spark will warn if you try to set <tt>spark.cache.class</tt>.
+
+<h3>Credits</h3>
+
+Spark 0.6 was the work of a large set of new contributors from Berkeley and outside.
+
+<ul>
+ <li>Tathagata Das contributed the new communication layer, and parts of the storage layer.</li>
+ <li>Haoyuan Li contributed the new storage manager.</li>
+ <li>Denny Britz contributed the YARN deploy mode, key aspects of the standalone one, and several other features.</li>
+ <li>Andy Konwinski contributed the revamped documentation site, Maven publishing, and several API docs.</li>
+ <li>Josh Rosen contributed the Java API, and several bug fixes.</li>
+ <li>Patrick Wendell contributed the enhanced debugging feature and helped with testing and documentation.</li>
+ <li>Reynold Xin contributed numerous bug and performance fixes.</li>
+ <li>Imran Rashid contributed the new Accumulable class.</li>
+ <li>Harvey Feng contributed improvements to shuffle operations.</li>
+ <li>Shivaram Venkataraman improved Spark's memory estimation and wrote a memory tuning guide.</li>
+ <li>Ravi Pandya contributed Spark run scripts for Windows.
+ </li><li>Mosharaf Chowdhury provided several fixes to broadcast.</li>
+ <li>Henry Milner pointed out several bugs in sampling algorithms.</li>
+ <li>Ray Racine provided improvements to the EC2 scripts.</li>
+ <li>Paul Ruan and Bill Zhao helped with testing.</li>
+</ul>
+
+<p style="padding-top:5px;">Thanks also to all the Spark users who have diligently suggested features or reported bugs.</p>
Added: incubator/spark/releases/_posts/2012-11-22-spark-release-0-5-2.md
URL: http://svn.apache.org/viewvc/incubator/spark/releases/_posts/2012-11-22-spark-release-0-5-2.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/releases/_posts/2012-11-22-spark-release-0-5-2.md (added)
+++ incubator/spark/releases/_posts/2012-11-22-spark-release-0-5-2.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,15 @@
+---
+layout: post
+title: Spark Release 0.5.2
+categories:
+- Releases
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '1'
+---
+Spark 0.5.2 is a minor release, whose main addition is to allow Spark to compile against Hadoop 2 distributions. To do this, edit <code>project/SparkBuild.scala</code> and change both the <code>HADOOP_VERSION</code> and <code>HADOOP_MAJOR_VERSION</code> variables, then recompile Spark. This change was contributed by Thomas Dudziak.
+
+You can download Spark 0.5.2 as a <a href="https://github.com/downloads/mesos/spark/spark-0.5.2-sources.tgz">tar.gz file</a> (2 MB).
Added: incubator/spark/releases/_posts/2012-11-22-spark-release-0-6-1.md
URL: http://svn.apache.org/viewvc/incubator/spark/releases/_posts/2012-11-22-spark-release-0-6-1.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/releases/_posts/2012-11-22-spark-release-0-6-1.md (added)
+++ incubator/spark/releases/_posts/2012-11-22-spark-release-0-6-1.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,30 @@
+---
+layout: post
+title: Spark Release 0.6.1
+categories:
+- Releases
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+---
+Spark 0.6.1 is a maintenance release that contains several important bug fixes and performance improvements. You can download it as a <a href="https://github.com/downloads/mesos/spark/spark-0.6.1-sources.tgz">source package</a> (2.4 MB tar.gz) or <a href="https://github.com/downloads/mesos/spark/spark-0.6.1-prebuilt.tgz">prebuilt package</a> (48 MB tar.gz).
+
+The fixes and improvements in this version include:
+<ul>
+ <li>Fixed overly aggressive message timeouts that could cause workers to disconnect from the cluster</li>
+ <li>Fixed a bug in the standalone deploy mode that did not expose hostnames to scheduler, affecting HDFS locality</li>
+ <li>Improved connection reuse in shuffle, which can greatly speed up small shuffles (contributed by Reynold Xin)</li>
+ <li>Fixed some potential deadlocks in the block manager (contributed by Tathagata Das)</li>
+ <li>Fixed a bug getting IDs of failed hosts from Mesos (contributed by Imran Rashid)</li>
+ <li>Several EC2 script improvements, like better handling of spot instances (contributed by Josh Rosen)</li>
+ <li>Made the local IP address that Spark binds to customizable (contributed by Mikhail Bautin)</li>
+ <li>Support for Hadoop 2 distributions (contributed by Thomas Dudziak)</li>
+ <li>Support for locating Scala on Debian distributions (contributed by Thomas Dudziak)</li>
+ <li>Improved standalone cluster web UI to show more information about jobs</li>
+ <li>Added an option to spread out jobs over the standalone cluster instead of concentrating them on a small number of nodes (<code>spark.deploy.spreadOut</code>)</li>
+</ul>
+
+We recommend that all Spark 0.6 users update to this maintenance release.
Added: incubator/spark/releases/_posts/2013-02-07-spark-release-0-6-2.md
URL: http://svn.apache.org/viewvc/incubator/spark/releases/_posts/2013-02-07-spark-release-0-6-2.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/releases/_posts/2013-02-07-spark-release-0-6-2.md (added)
+++ incubator/spark/releases/_posts/2013-02-07-spark-release-0-6-2.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,43 @@
+---
+layout: post
+title: Spark Release 0.6.2
+categories: []
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+Spark 0.6.2 is a maintenance release that contains several bug fixes and usability improvements. You can download it as a <a href="http://spark-project.org/files/spark-0.6.2-sources.tgz">source package</a> (2.5 MB tar.gz) or <a href="http://spark-project.org/files/spark-0.6.2-prebuilt.tgz">prebuilt package</a> (48 MB tar.gz).
+
+We recommend that all Spark 0.6 users update to this maintenance release.
+
+The fixes and improvements in this version include:
+<ul>
+ <li>A number of fault tolerance fixes regarding detecting dead nodes, handling missing map output fetches, and allowing failed nodes to rejoin the cluster</li>
+ <li>Documentation fixes that clarify the configuration for the standalone mode and improve the quick start instructions</li>
+ <li>A connection reuse bug fix that improves shuffle performance</li>
+ <li>Support for launching a cluster across multiple availability zones in the EC2 scripts</li>
+ <li>Support for deleting security groups when an EC2 cluster is terminated</li>
+ <li>Improved memory configuration for the standalone deploy cluster daemons: instead of using <code>SPARK_MEM</code> for their memory, which often led people to give them much more memory than they intended, they now use a separate variable, <code>SPARK_DAEMON_MEMORY</code>, with a reasonable default of 512 MB
+ <li>Fixes to the Windows run scripts for Spark</li>
+ <li>Better detection of a machine's external IP address</li>
+ <li>Several small optimizations and bug fixes</li>
+</ul>
+
+In total, eleven people contributed to this release:
+<ul>
+ <li>Stephen Haberman (bug fix)</li>
+ <li>Shane Huang (shuffle fix)</li>
+ <li>Fernand Pajot (bug fix)</li>
+ <li>Andrew Psaltis (bug fix)</li>
+ <li>Imran Rashid (standalone cluster, bug fix)</li>
+ <li>Charles Reiss (fault recovery fixes, node re-registration, tests)</li>
+ <li>Josh Rosen (fault recovery, Java API fixes, deploy scripts)</li>
+ <li>Peter Sankauskas (EC2 scripts)</li>
+ <li>Lee Moon Soo (bug fix)</li>
+ <li>Patrick Wendell (bugs, docs)</li>
+ <li>Matei Zaharia (fault recovery, UI, docs, bug fixes)</li>
+</ul>
Added: incubator/spark/releases/_posts/2013-02-27-spark-release-0-7-0.md
URL: http://svn.apache.org/viewvc/incubator/spark/releases/_posts/2013-02-27-spark-release-0-7-0.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/releases/_posts/2013-02-27-spark-release-0-7-0.md (added)
+++ incubator/spark/releases/_posts/2013-02-27-spark-release-0-7-0.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,112 @@
+---
+layout: post
+title: Spark Release 0.7.0
+categories: []
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+The Spark team is proud to release version 0.7.0, a new major release that brings several new features. Most notable are a <a href="/docs/0.7.0/python-programming-guide.html">Python API for Spark</a> and an <a href="/docs/0.7.0/streaming-programming-guide.html">alpha of Spark Streaming</a>. (Details on Spark Streaming can also be found in this <a href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-259.pdf">technical report</a>.) The release also adds numerous other improvements across the board. Overall, this is our biggest release to date, with 31 contributors, of which 20 were external to Berkeley.
+
+You can download Spark 0.7.0 as either a <a href="/files/spark-0.7.0-sources.tgz">source package</a> (4 MB tar.gz) or <a href="/files/spark-0.7.0-prebuilt.tgz">prebuilt package</a> (60 MB tar.gz).
+
+<h3>Python API</h3>
+
+Spark 0.7 adds a <a href="/docs/0.7.0/python-programming-guide.html">Python API</a> called PySpark that makes it possible to use Spark from Python, both in standalone programs and in interactive Python shells. It uses the standard CPython runtime, so your programs can call into native libraries like NumPy and SciPy. Like the Scala and Java APIs, PySpark will automatically ship functions from your main program, along with the variables they depend on, to the cluster. PySpark supports most Spark features, including RDDs, accumulators, broadcast variables, and HDFS input and output.
+
+<h3>Spark Streaming Alpha</h3>
+
+Spark Streaming is a new extension of Spark that adds near-real-time processing capability. It offers a simple and high-level API, where users can transform streams using parallel operations like <tt>map</tt>, <tt>filter</tt>, <tt>reduce</tt>, and new sliding window functions. It automatically distributes work over a cluster and provides efficient fault recovery with exactly-once semantics for transformations, without relying on costly transactions to an external system. Spark Streaming is described in more detail in <a href="/talks/strata_spark_streaming.ppt">these slides</a> and <a href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-259.pdf">our technical report</a>. This release is our first alpha of Spark Streaming, with most of the functionality implemented and APIs in Java and Scala.
+
+<h3>Memory Dashboard</h3>
+
+Spark jobs now launch a web dashboard for monitoring the memory usage of each distributed dataset (RDD) in the program. Look for lines like this in your log:
+
+<tt>15:08:44 INFO BlockManagerUI: Started BlockManager web UI at http://mbk.local:63814</tt>
+
+You can also control which port to use through the <tt>spark.ui.port</tt> property.
+
+<h3>Maven Build</h3>
+
+Spark can now be built using Maven in addition to SBT. The Maven build enables easier publishing to repositories of your choice, easy selection of Hadoop versions using the Maven profile (<tt>-Phadoop1</tt> or <tt>-Phadoop2</tt>), as well as Debian packaging using <tt>mvn -Phadoop1,deb install</tt>.
+
+<h3>New Operations</h3>
+
+This release adds several RDD transformations, including <tt>keys</tt>, <tt>values</tt>, <tt>keyBy</tt>, <tt>subtract</tt>, <tt>coalesce</tt>, <tt>zip</tt>. It also adds <tt>SparkContext.hadoopConfiguration</tt> to allow programs to configure Hadoop input/output settings globally across operations. Finally, it adds the <tt>RDD.toDebugString()</tt> method, which can be used to print an RDD's lineage graph for troubleshooting.
+
+<h3>EC2 Improvements</h3>
+
+<ul>
+ <li>Spark will now read S3 credentials from the <tt>AWS_ACCESS_KEY_ID</tt> and <tt>AWS_SECRET_ACCESS_KEY</tt> environment variables, if set, making it easier to access Amazon S3.</li>
+ <li>This release fixes a bug with S3 access that would leave streams open when they are not fully read (e.g. when calling <tt>RDD.first()</tt> or a SQL query with a limit), causing nodes to hang.</li>
+ <li>The EC2 scripts now support both standalone and Mesos clusters, and launch Ganglia on the cluster.</li>
+ <li>Spark EC2 clusters can now be spread across multiple availability zones.</li>
+</ul>
+
+<h3>Other Improvements</h3>
+
+<ul>
+ <li>Shuffle operations like <tt>groupByKey</tt> and <tt>reduceByKey</tt> now try to infer parallelism from the size of the parent RDD (unless <tt>spark.default.parallelism</tt> is set).</li>
+ <li>Several performance improvements to shuffles.</li>
+ <li>Standalone deploy cluster now spreads jobs out across machines by default, leading to better data locality.</li>
+ <li>Better error reporting when jobs aren't being launched due to not enough resources.</li>
+ <li>Standalone deploy web UI now includes JSON endpoints for querying cluster state.</li>
+ <li>Better support for IBM JVM.</li>
+ <li>Default Hadoop version dependency updated to 1.0.4.</li>
+ <li>Improved failure handling and reporting of error messages.</li>
+ <li>Separate configuration for standalone cluster daemons and user applications.</li>
+ <li>Significant refactoring of the scheduler codebase to enable richer unit testing.</li>
+ <li>Several bug and performance fixes throughout.</li>
+</ul>
+
+<h3>Compatibility</h3>
+
+This release is API-compatible with Spark 0.6 programs, but the following features changed slightly:
+<ul>
+ <li>Parallel shuffle operations where you don't specify a level of parallelism use the number of partitions of the parent RDD instead of a constant default. However, if you set <tt>spark.default.parallelism</tt>, they will use that.</li>
+ <li><tt>SparkContext.addFile</tt>, which distributes a file to worker nodes, is no longer guaranteed to put it in the executor's working directory---instead, you can find the directory it used using <tt>SparkFiles.getRootDirectory</tt>, or get a particular file using <tt>SparkFiles.get</tt>. This was done to avoid cluttering the local directory when running in local mode.</li>
+</ul>
+
+<h3>Credits</h3>
+
+Spark 0.7 was the work of many contributors from Berkeley and outside---in total, 31 different contributors, of which 20 were from outside Berkeley. Here are the people who contributed, along with areas they worked on:
+
+<ul>
+ <li>Mikhail Bautin -- Maven build</li>
+ <li>Denny Britz -- memory dashboard, streaming, bug fixes</li>
+ <li>Paul Cavallaro -- error reporting</li>
+ <li>Tathagata Das -- streaming (lead developer), 24/7 operation, bug fixes, docs</li>
+ <li>Thomas Dudziak -- Maven build, Hadoop 2 support</li>
+ <li>Harvey Feng -- bug fix</li>
+ <li>Stephen Haberman -- new RDD operations, configuration, S3 improvements, code cleanup, bug fixes</li>
+ <li>Tyson Hamilton -- JSON status endpoints</li>
+ <li>Mark Hamstra -- API improvements, docs</li>
+ <li>Michael Heuer -- docs</li>
+ <li>Shane Huang -- shuffle performance fixes</li>
+ <li>Andy Konwinski -- docs</li>
+ <li>Ryan LeCompte -- streaming</li>
+ <li>Haoyuan Li -- streaming</li>
+ <li>Richard McKinley -- build</li>
+ <li>Sean McNamara -- streaming</li>
+ <li>Lee Moon Soo -- bug fix</li>
+ <li>Fernand Pajot -- bug fix</li>
+ <li>Nick Pentreath -- Python API, examples</li>
+ <li>Andrew Psaltis -- bug fixes</li>
+ <li>Imran Rashid -- memory dashboard, bug fixes</li>
+ <li>Charles Reiss -- fault recovery fixes, code cleanup, testability, error reporting</li>
+ <li>Josh Rosen -- Python API (lead developer), EC2 scripts, bug fixes</li>
+ <li>Peter Sankauskas -- EC2 scripts</li>
+ <li>Prashant Sharma -- streaming</li>
+ <li>Shivaram Venkataraman -- EC2 scripts, optimizations</li>
+ <li>Patrick Wendell -- streaming, bug fixes, examples, docs</li>
+ <li>Reynold Xin -- optimizations, UI</li>
+ <li>Haitao Yao -- run scripts</li>
+ <li>Matei Zaharia -- streaming, fault recovery, Python API, code cleanup, bug fixes, docs</li>
+ <li>Eric Zhang -- examples</li>
+</ul>
+
+Thanks to everyone who contributed!
Added: incubator/spark/releases/_posts/2013-06-02-spark-release-0-7-2.md
URL: http://svn.apache.org/viewvc/incubator/spark/releases/_posts/2013-06-02-spark-release-0-7-2.md?rev=1516937&view=auto
==============================================================================
--- incubator/spark/releases/_posts/2013-06-02-spark-release-0-7-2.md (added)
+++ incubator/spark/releases/_posts/2013-06-02-spark-release-0-7-2.md Fri Aug 23 17:17:53 2013
@@ -0,0 +1,56 @@
+---
+layout: post
+title: Spark Release 0.7.2
+categories: []
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+Spark 0.7.2 is a maintenance release that contains multiple bug fixes and improvements. You can download it as a <a href="http://spark-project.org/download-spark-0.7.2-sources">source package</a> (4 MB tar.gz) or get prebuilt packages for <a href="http://spark-project.org/download-spark-0.7.2-prebuilt-hadoop1">Hadoop 1 / CDH3</a> or <a href="http://spark-project.org/download-spark-0.7.2-prebuilt-cdh4">CDH 4</a> (61 MB tar.gz).
+
+
+We recommend that all users update to this maintenance release.
+
+
+The fixes and improvements in this version include:
+<ul>
+ <li>Scala version updated to 2.9.3.</li>
+ <li>Several improvements to Bagel, including performance fixes and a configurable storage level.</li>
+ <li>New API methods: subtractByKey, foldByKey, mapWith, filterWith, foreachPartition, and others.</li>
+ <li>A new metrics reporting interface, SparkListener, to collect information about each computation stage: task lengths, bytes shuffled, etc.</li>
+ <li>Several new examples using the Java API, including K-means and computing pi.</li>
+ <li>Support for launching multiple worker instances per host in the standalone mode.</li>
+ <li>Various bug fixes across the board.</li>
+</ul>
+
+The following people contributed to this release:
+<ul>
+ <li>Jey Kottalam (Maven build, bug fixes, EC2 scripts, packaging the release)</li>
+ <li>Andrew Ash (bug fixes, docs)</li>
+ <li>Andrey Kouznetsov (bug fixes)</li>
+ <li>Andy Konwinski (docs)</li>
+ <li>Charles Reiss (bug fixes)</li>
+ <li>Christoph Grothaus (bug fixes)</li>
+ <li>Erik van Oosten (bug fixes)</li>
+ <li>Giovanni Delussu (bug fixes)</li>
+ <li>Hiral Patel (bug fixes)</li>
+ <li>Holden Karau (error reporting, EC2 scripts)</li>
+ <li>Imran Rashid (metrics reporting system)</li>
+ <li>Josh Rosen (EC2 scripts)</li>
+ <li>Mark Hamstra (new API methods, tests)</li>
+ <li>Mikhail Bautin (build)</li>
+ <li>Mosharaf Chowdhury (bug fixes)</li>
+ <li>Nick Pentreath (Bagel, examples)</li>
+ <li>Patrick Wendell (bug fixes)</li>
+ <li>Reynold Xin (bug fixes)</li>
+ <li>Stephen Haberman (bug fixes, tests, subtractByKey)</li>
+ <li>Kalpit Shah (build, multiple workers per host)</li>
+ <li>Mike Potts (run scripts)</li>
+ <li>Matei Zaharia (Bagel, bug fixes, build)</li>
+</ul>
+
+We thank everyone who helped with this release, and hope to see more contributions from you in the future!