You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2016/01/14 19:30:13 UTC
svn commit: r1724669 [3/3] - in /spark: news/_posts/ site/ site/graphx/
site/mllib/ site/news/ site/releases/ site/screencasts/ site/sql/
site/streaming/
Modified: spark/site/releases/spark-release-1-3-0.html
URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-1-3-0.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/releases/spark-release-1-3-0.html (original)
+++ spark/site/releases/spark-release-1-3-0.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
@@ -175,7 +175,7 @@
<p>To download Spark 1.3 visit the <a href="/downloads.html">downloads</a> page.</p>
<h3 id="spark-core">Spark Core</h3>
-<p>Spark 1.3 sees a handful of usability improvements in the core engine. The core API now supports <a href="https://issues.apache.org/jira/browse/SPARK-5430">multi level aggregation trees</a> to help speed up expensive reduce operations. <a href="https://issues.apache.org/jira/browse/SPARK-5063">Improved error reporting</a> has been added for certain gotcha operations. Spark’s Jetty dependency is <a href="https://issues.apache.org/jira/browse/SPARK-3996">now shaded</a> to help avoid conflicts with user programs. Spark now supports <a href="https://issues.apache.org/jira/browse/SPARK-3883">SSL encryption</a> for some communication endpoints. Finaly, realtime <a href="https://issues.apache.org/jira/browse/SPARK-3428">GC metrics</a> and <a href="https://issues.apache.org/jira/browse/SPARK-4874">record counts</a> have been added to the UI. </p>
+<p>Spark 1.3 sees a handful of usability improvements in the core engine. The core API now supports <a href="https://issues.apache.org/jira/browse/SPARK-5430">multi level aggregation trees</a> to help speed up expensive reduce operations. <a href="https://issues.apache.org/jira/browse/SPARK-5063">Improved error reporting</a> has been added for certain gotcha operations. Spark’s Jetty dependency is <a href="https://issues.apache.org/jira/browse/SPARK-3996">now shaded</a> to help avoid conflicts with user programs. Spark now supports <a href="https://issues.apache.org/jira/browse/SPARK-3883">SSL encryption</a> for some communication endpoints. Finaly, realtime <a href="https://issues.apache.org/jira/browse/SPARK-3428">GC metrics</a> and <a href="https://issues.apache.org/jira/browse/SPARK-4874">record counts</a> have been added to the UI.</p>
<h3 id="dataframe-api">DataFrame API</h3>
<p>Spark 1.3 adds a new <a href="/docs/1.3.0/sql-programming-guide.html#dataframes">DataFrames API</a> that provides powerful and convenient operators when working with structured datasets. The DataFrame is an evolution of the base RDD API that includes named fields along with schema information. Itâs easy to construct a DataFrame from sources such as Hive tables, JSON data, a JDBC database, or any implementation of Sparkâs new data source API. Data frames will become a common interchange format between Spark components and when importing and exporting data to other systems. Data frames are supported in Python, Scala, and Java.</p>
@@ -187,7 +187,7 @@
<p>In this release Spark MLlib introduces several new algorithms: latent Dirichlet allocation (LDA) for <a href="https://issues.apache.org/jira/browse/SPARK-1405">topic modeling</a>, <a href="https://issues.apache.org/jira/browse/SPARK-2309">multinomial logistic regression</a> for multiclass classification, <a href="https://issues.apache.org/jira/browse/SPARK-5012">Gaussian mixture model (GMM)</a> and <a href="https://issues.apache.org/jira/browse/SPARK-4259">power iteration clustering</a> for clustering, <a href="https://issues.apache.org/jira/browse/SPARK-4001">FP-growth</a> for frequent pattern mining, and <a href="https://issues.apache.org/jira/browse/SPARK-4409">block matrix abstraction</a> for distributed linear algebra. Initial support has been added for <a href="https://issues.apache.org/jira/browse/SPARK-4587">model import/export</a> in exchangeable format, which will be expanded in future versions to cover more model types in Java/Python/Scala. The implementations of k-mea
ns and ALS receive <a href="https://issues.apache.org/jira/browse/SPARK-3424, https://issues.apache.org/jira/browse/SPARK-3541">updates</a> that lead to significant performance gain. PySpark now supports the <a href="https://issues.apache.org/jira/browse/SPARK-4586">ML pipeline API</a> added in Spark 1.2, and <a href="https://issues.apache.org/jira/browse/SPARK-5094">gradient boosted trees</a> and <a href="https://issues.apache.org/jira/browse/SPARK-5012">Gaussian mixture model</a>. Finally, the ML pipeline API has been ported to support the new DataFrames abstraction.</p>
<h3 id="spark-streaming">Spark Streaming</h3>
-<p>Spark 1.3 introduces a new <a href="https://issues.apache.org/jira/browse/SPARK-4964"><em>direct</em> Kafka API</a> (<a href="http://spark.apache.org/docs/1.3.0/streaming-kafka-integration.html">docs</a>) which enables exactly-once delivery without the use of write ahead logs. It also adds a <a href="https://issues.apache.org/jira/browse/SPARK-5047">Python Kafka API</a> along with infrastructure for additional Python APIâs in future releases. An online version of <a href="https://issues.apache.org/jira/browse/SPARK-4979">logistic regression</a> and the ability to read <a href="https://issues.apache.org/jira/browse/SPARK-4969">binary records</a> have also been added. For stateful operations, support has been added for loading of an <a href="https://issues.apache.org/jira/browse/SPARK-3660">initial state RDD</a>. Finally, the streaming programming guide has been updated to include information about SQL and DataFrame operations within streaming applications, and important clari
fications to the fault-tolerance semantics. </p>
+<p>Spark 1.3 introduces a new <a href="https://issues.apache.org/jira/browse/SPARK-4964"><em>direct</em> Kafka API</a> (<a href="http://spark.apache.org/docs/1.3.0/streaming-kafka-integration.html">docs</a>) which enables exactly-once delivery without the use of write ahead logs. It also adds a <a href="https://issues.apache.org/jira/browse/SPARK-5047">Python Kafka API</a> along with infrastructure for additional Python APIâs in future releases. An online version of <a href="https://issues.apache.org/jira/browse/SPARK-4979">logistic regression</a> and the ability to read <a href="https://issues.apache.org/jira/browse/SPARK-4969">binary records</a> have also been added. For stateful operations, support has been added for loading of an <a href="https://issues.apache.org/jira/browse/SPARK-3660">initial state RDD</a>. Finally, the streaming programming guide has been updated to include information about SQL and DataFrame operations within streaming applications, and important clari
fications to the fault-tolerance semantics.</p>
<h3 id="graphx">GraphX</h3>
<p>GraphX adds a handful of utility functions in this release, including conversion into a <a href="https://issues.apache.org/jira/browse/SPARK-4917">canonical edge graph</a>.</p>
@@ -203,7 +203,7 @@
<ul>
<li><a href="https://issues.apache.org/jira/browse/SPARK-6194">SPARK-6194</a>: A memory leak in PySPark’s <code>collect()</code>.</li>
<li><a href="https://issues.apache.org/jira/browse/SPARK-6222">SPARK-6222</a>: An issue with failure recovery in Spark Streaming.</li>
- <li><a href="https://issues.apache.org/jira/browse/SPARK-6315">SPARK-6315</a>: Spark SQL can’t read parquet data generated with Spark 1.1. </li>
+ <li><a href="https://issues.apache.org/jira/browse/SPARK-6315">SPARK-6315</a>: Spark SQL can’t read parquet data generated with Spark 1.1.</li>
<li><a href="https://issues.apache.org/jira/browse/SPARK-6247">SPARK-6247</a>: Errors analyzing certain join types in Spark SQL.</li>
</ul>
Modified: spark/site/releases/spark-release-1-3-1.html
URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-1-3-1.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/releases/spark-release-1-3-1.html (original)
+++ spark/site/releases/spark-release-1-3-1.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
@@ -180,10 +180,10 @@
<h4 id="spark-sql">Spark SQL</h4>
<ul>
<li>Unable to use reserved words in DDL (<a href="http://issues.apache.org/jira/browse/SPARK-6250">SPARK-6250</a>)</li>
- <li>Parquet no longer caches metadata (<a href="http://issues.apache.org/jira/browse/SPARK-6575">SPARK-6575</a>) </li>
+ <li>Parquet no longer caches metadata (<a href="http://issues.apache.org/jira/browse/SPARK-6575">SPARK-6575</a>)</li>
<li>Bug when joining two Parquet tables (<a href="http://issues.apache.org/jira/browse/SPARK-6851">SPARK-6851</a>)</li>
- <li>Unable to read parquet data generated by Spark 1.1.1 (<a href="http://issues.apache.org/jira/browse/SPARK-6315">SPARK-6315</a>) </li>
- <li>Parquet data source may use wrong Hadoop FileSystem (<a href="http://issues.apache.org/jira/browse/SPARK-6330">SPARK-6330</a>) </li>
+ <li>Unable to read parquet data generated by Spark 1.1.1 (<a href="http://issues.apache.org/jira/browse/SPARK-6315">SPARK-6315</a>)</li>
+ <li>Parquet data source may use wrong Hadoop FileSystem (<a href="http://issues.apache.org/jira/browse/SPARK-6330">SPARK-6330</a>)</li>
</ul>
<h4 id="spark-streaming">Spark Streaming</h4>
Modified: spark/site/releases/spark-release-1-4-0.html
URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-1-4-0.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/releases/spark-release-1-4-0.html (original)
+++ spark/site/releases/spark-release-1-4-0.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
@@ -234,7 +234,7 @@ Python coverage. MLlib also adds several
</ul>
<h3 id="spark-streaming">Spark Streaming</h3>
-<p>Spark streaming adds visual instrumentation graphs and significantly improved debugging information in the UI. It also enhances support for both Kafka and Kinesis. </p>
+<p>Spark streaming adds visual instrumentation graphs and significantly improved debugging information in the UI. It also enhances support for both Kafka and Kinesis.</p>
<ul>
<li><a href="https://issues.apache.org/jira/browse/SPARK-7602">SPARK-7602</a>: Visualization and monitoring in the streaming UI including batch drill down (<a href="https://issues.apache.org/jira/browse/SPARK-6796">SPARK-6796</a>, <a href="https://issues.apache.org/jira/browse/SPARK-6862">SPARK-6862</a>)</li>
@@ -260,7 +260,7 @@ Python coverage. MLlib also adds several
<h4 id="test-partners">Test Partners</h4>
-<p>Thanks to The following organizations, who helped benchmark or integration test release candidates: <br /> Intel, Palantir, Cloudera, Mesosphere, Huawei, Shopify, Netflix, Yahoo, UC Berkeley and Databricks. </p>
+<p>Thanks to The following organizations, who helped benchmark or integration test release candidates: <br /> Intel, Palantir, Cloudera, Mesosphere, Huawei, Shopify, Netflix, Yahoo, UC Berkeley and Databricks.</p>
<h4 id="contributors">Contributors</h4>
<ul>
Modified: spark/site/releases/spark-release-1-4-1.html
URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-1-4-1.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/releases/spark-release-1-4-1.html (original)
+++ spark/site/releases/spark-release-1-4-1.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
Modified: spark/site/releases/spark-release-1-5-0.html
URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-1-5-0.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/releases/spark-release-1-5-0.html (original)
+++ spark/site/releases/spark-release-1-5-0.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
@@ -175,25 +175,25 @@
<p>You can consult JIRA for the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12332078">detailed changes</a>. We have curated a list of high level changes here:</p>
<ul id="markdown-toc">
- <li><a href="#apis-rdd-dataframe-and-sql">APIs: RDD, DataFrame and SQL</a></li>
- <li><a href="#backend-execution-dataframe-and-sql">Backend Execution: DataFrame and SQL</a></li>
- <li><a href="#integrations-data-sources-hive-hadoop-mesos-and-cluster-management">Integrations: Data Sources, Hive, Hadoop, Mesos and Cluster Management</a></li>
- <li><a href="#r-language">R Language</a></li>
- <li><a href="#machine-learning-and-advanced-analytics">Machine Learning and Advanced Analytics</a></li>
- <li><a href="#spark-streaming">Spark Streaming</a></li>
- <li><a href="#deprecations-removals-configs-and-behavior-changes">Deprecations, Removals, Configs, and Behavior Changes</a> <ul>
- <li><a href="#spark-core">Spark Core</a></li>
- <li><a href="#spark-sql--dataframes">Spark SQL & DataFrames</a></li>
- <li><a href="#spark-streaming-1">Spark Streaming</a></li>
- <li><a href="#mllib">MLlib</a></li>
+ <li><a href="#apis-rdd-dataframe-and-sql" id="markdown-toc-apis-rdd-dataframe-and-sql">APIs: RDD, DataFrame and SQL</a></li>
+ <li><a href="#backend-execution-dataframe-and-sql" id="markdown-toc-backend-execution-dataframe-and-sql">Backend Execution: DataFrame and SQL</a></li>
+ <li><a href="#integrations-data-sources-hive-hadoop-mesos-and-cluster-management" id="markdown-toc-integrations-data-sources-hive-hadoop-mesos-and-cluster-management">Integrations: Data Sources, Hive, Hadoop, Mesos and Cluster Management</a></li>
+ <li><a href="#r-language" id="markdown-toc-r-language">R Language</a></li>
+ <li><a href="#machine-learning-and-advanced-analytics" id="markdown-toc-machine-learning-and-advanced-analytics">Machine Learning and Advanced Analytics</a></li>
+ <li><a href="#spark-streaming" id="markdown-toc-spark-streaming">Spark Streaming</a></li>
+ <li><a href="#deprecations-removals-configs-and-behavior-changes" id="markdown-toc-deprecations-removals-configs-and-behavior-changes">Deprecations, Removals, Configs, and Behavior Changes</a> <ul>
+ <li><a href="#spark-core" id="markdown-toc-spark-core">Spark Core</a></li>
+ <li><a href="#spark-sql--dataframes" id="markdown-toc-spark-sql--dataframes">Spark SQL & DataFrames</a></li>
+ <li><a href="#spark-streaming-1" id="markdown-toc-spark-streaming-1">Spark Streaming</a></li>
+ <li><a href="#mllib" id="markdown-toc-mllib">MLlib</a></li>
</ul>
</li>
- <li><a href="#known-issues">Known Issues</a> <ul>
- <li><a href="#sqldataframe">SQL/DataFrame</a></li>
- <li><a href="#streaming">Streaming</a></li>
+ <li><a href="#known-issues" id="markdown-toc-known-issues">Known Issues</a> <ul>
+ <li><a href="#sqldataframe" id="markdown-toc-sqldataframe">SQL/DataFrame</a></li>
+ <li><a href="#streaming" id="markdown-toc-streaming">Streaming</a></li>
</ul>
</li>
- <li><a href="#credits">Credits</a></li>
+ <li><a href="#credits" id="markdown-toc-credits">Credits</a></li>
</ul>
<h3 id="apis-rdd-dataframe-and-sql">APIs: RDD, DataFrame and SQL</h3>
Modified: spark/site/releases/spark-release-1-5-1.html
URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-1-5-1.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/releases/spark-release-1-5-1.html (original)
+++ spark/site/releases/spark-release-1-5-1.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
Modified: spark/site/releases/spark-release-1-5-2.html
URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-1-5-2.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/releases/spark-release-1-5-2.html (original)
+++ spark/site/releases/spark-release-1-5-2.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
Modified: spark/site/releases/spark-release-1-6-0.html
URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-1-6-0.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/releases/spark-release-1-6-0.html (original)
+++ spark/site/releases/spark-release-1-6-0.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
@@ -175,13 +175,13 @@
<p>You can consult JIRA for the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12333083&projectId=12315420">detailed changes</a>. We have curated a list of high level changes here:</p>
<ul id="markdown-toc">
- <li><a href="#spark-coresql">Spark Core/SQL</a></li>
- <li><a href="#spark-streaming">Spark Streaming</a></li>
- <li><a href="#mllib">MLlib</a></li>
- <li><a href="#deprecations">Deprecations</a></li>
- <li><a href="#changes-of-behavior">Changes of behavior</a></li>
- <li><a href="#known-issues">Known issues</a></li>
- <li><a href="#credits">Credits</a></li>
+ <li><a href="#spark-coresql" id="markdown-toc-spark-coresql">Spark Core/SQL</a></li>
+ <li><a href="#spark-streaming" id="markdown-toc-spark-streaming">Spark Streaming</a></li>
+ <li><a href="#mllib" id="markdown-toc-mllib">MLlib</a></li>
+ <li><a href="#deprecations" id="markdown-toc-deprecations">Deprecations</a></li>
+ <li><a href="#changes-of-behavior" id="markdown-toc-changes-of-behavior">Changes of behavior</a></li>
+ <li><a href="#known-issues" id="markdown-toc-known-issues">Known issues</a></li>
+ <li><a href="#credits" id="markdown-toc-credits">Credits</a></li>
</ul>
<h3 id="spark-coresql">Spark Core/SQL</h3>
@@ -204,7 +204,7 @@
<ul>
<li><a href="https://issues.apache.org/jira/browse/SPARK-10000">SPARK-10000</a> <strong>Unified Memory Management</strong> - Shared memory for execution and caching instead of exclusive division of the regions.</li>
<li><a href="https://issues.apache.org/jira/browse/SPARK-11787">SPARK-11787</a> <strong>Parquet Performance</strong> - Improve Parquet scan performance when using flat schemas.</li>
- <li><a href="https://issues.apache.org/jira/browse/SPARK-9241">SPARK-9241 </a> <strong>Improved query planner for queries having distinct aggregations</strong> - Query plans of distinct aggregations are more robust when distinct columns have high cardinality. </li>
+ <li><a href="https://issues.apache.org/jira/browse/SPARK-9241">SPARK-9241 </a> <strong>Improved query planner for queries having distinct aggregations</strong> - Query plans of distinct aggregations are more robust when distinct columns have high cardinality.</li>
<li><a href="https://issues.apache.org/jira/browse/SPARK-9858">SPARK-9858 </a> <strong>Adaptive query execution</strong> - Initial support for automatically selecting the number of reducers for joins and aggregations.</li>
<li><a href="https://issues.apache.org/jira/browse/SPARK-10978">SPARK-10978</a> <strong>Avoiding double filters in Data Source API</strong> - When implementing a data source with filter pushdown, developers can now tell Spark SQL to avoid double evaluating a pushed-down filter.</li>
<li><a href="https://issues.apache.org/jira/browse/SPARK-11111">SPARK-11111</a> <strong>Fast null-safe joins</strong> - Joins using null-safe equality (<code><=></code>) will now execute using SortMergeJoin instead of computing a cartisian product.</li>
@@ -217,7 +217,7 @@
<h3 id="spark-streaming">Spark Streaming</h3>
<ul>
- <li><strong>API Updates</strong>
+ <li><strong>API Updates</strong>
<ul>
<li><a href="https://issues.apache.org/jira/browse/SPARK-2629">SPARK-2629 </a> <strong>New improved state management</strong> - <code>mapWithState</code> - a DStream transformation for stateful stream processing, supercedes <code>updateStateByKey</code> in functionality and performance.</li>
<li><a href="https://issues.apache.org/jira/browse/SPARK-11198">SPARK-11198</a> <strong>Kinesis record deaggregation</strong> - Kinesis streams have been upgraded to use KCL 1.4.0 and supports transparent deaggregation of KPL-aggregated records.</li>
@@ -228,7 +228,7 @@
<li><strong>UI Improvements</strong>
<ul>
<li>Made failures visible in the streaming tab, in the timelines, batch list, and batch details page.</li>
- <li>Made output operations visible in the streaming tab as progress bars. </li>
+ <li>Made output operations visible in the streaming tab as progress bars.</li>
</ul>
</li>
</ul>
Modified: spark/site/research.html
URL: http://svn.apache.org/viewvc/spark/site/research.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/research.html (original)
+++ spark/site/research.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
Modified: spark/site/screencasts/1-first-steps-with-spark.html
URL: http://svn.apache.org/viewvc/spark/site/screencasts/1-first-steps-with-spark.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/screencasts/1-first-steps-with-spark.html (original)
+++ spark/site/screencasts/1-first-steps-with-spark.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
Modified: spark/site/screencasts/2-spark-documentation-overview.html
URL: http://svn.apache.org/viewvc/spark/site/screencasts/2-spark-documentation-overview.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/screencasts/2-spark-documentation-overview.html (original)
+++ spark/site/screencasts/2-spark-documentation-overview.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
Modified: spark/site/screencasts/3-transformations-and-caching.html
URL: http://svn.apache.org/viewvc/spark/site/screencasts/3-transformations-and-caching.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/screencasts/3-transformations-and-caching.html (original)
+++ spark/site/screencasts/3-transformations-and-caching.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
Modified: spark/site/screencasts/4-a-standalone-job-in-spark.html
URL: http://svn.apache.org/viewvc/spark/site/screencasts/4-a-standalone-job-in-spark.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/screencasts/4-a-standalone-job-in-spark.html (original)
+++ spark/site/screencasts/4-a-standalone-job-in-spark.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
Modified: spark/site/screencasts/index.html
URL: http://svn.apache.org/viewvc/spark/site/screencasts/index.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/screencasts/index.html (original)
+++ spark/site/screencasts/index.html Thu Jan 14 18:30:11 2016
@@ -134,6 +134,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -143,9 +146,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
Modified: spark/site/sql/index.html
URL: http://svn.apache.org/viewvc/spark/site/sql/index.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/sql/index.html (original)
+++ spark/site/sql/index.html Thu Jan 14 18:30:11 2016
@@ -137,6 +137,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -146,9 +149,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
Modified: spark/site/streaming/index.html
URL: http://svn.apache.org/viewvc/spark/site/streaming/index.html?rev=1724669&r1=1724668&r2=1724669&view=diff
==============================================================================
--- spark/site/streaming/index.html (original)
+++ spark/site/streaming/index.html Thu Jan 14 18:30:11 2016
@@ -137,6 +137,9 @@
<h5>Latest News</h5>
<ul class="list-unstyled">
+ <li><a href="/news/spark-summit-east-agenda-posted.html">Spark Summit East (Feb 16, 2016, New York) agenda posted</a>
+ <span class="small">(Jan 14, 2016)</span></li>
+
<li><a href="/news/spark-1-6-0-released.html">Spark 1.6.0 released</a>
<span class="small">(Jan 04, 2016)</span></li>
@@ -146,9 +149,6 @@
<li><a href="/news/spark-1-5-2-released.html">Spark 1.5.2 released</a>
<span class="small">(Nov 09, 2015)</span></li>
- <li><a href="/news/submit-talks-to-spark-summit-east-2016.html">Submission is open for Spark Summit East 2016</a>
- <span class="small">(Oct 14, 2015)</span></li>
-
</ul>
<p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
</div>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org