You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2013/09/21 00:06:21 UTC
[01/50] git commit: Fix some review comments
Updated Branches:
refs/heads/master a106ed8b9 -> 119de8029
Fix some review comments
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/b4588549
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/b4588549
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/b4588549
Branch: refs/heads/master
Commit: b458854977c437e85fd89056e5d40383c8fa962e
Parents: 170b386
Author: Matei Zaharia <ma...@eecs.berkeley.edu>
Authored: Sun Sep 8 21:25:49 2013 -0700
Committer: Matei Zaharia <ma...@eecs.berkeley.edu>
Committed: Sun Sep 8 21:25:49 2013 -0700
----------------------------------------------------------------------
docs/cluster-overview.md | 2 +-
docs/quick-start.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b4588549/docs/cluster-overview.md
----------------------------------------------------------------------
diff --git a/docs/cluster-overview.md b/docs/cluster-overview.md
index cf6b48c..7025c23 100644
--- a/docs/cluster-overview.md
+++ b/docs/cluster-overview.md
@@ -80,7 +80,7 @@ The following table summarizes terms you'll see used to refer to cluster concept
<tbody>
<tr>
<td>Application</td>
- <td>Any user program invoking Spark</td>
+ <td>User program built on Spark. Consists of a <em>driver program</em> and <em>executors</em> on the cluster.</td>
</tr>
<tr>
<td>Driver program</td>
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b4588549/docs/quick-start.md
----------------------------------------------------------------------
diff --git a/docs/quick-start.md b/docs/quick-start.md
index 1b069ce..8f782db 100644
--- a/docs/quick-start.md
+++ b/docs/quick-start.md
@@ -36,7 +36,7 @@ scala> textFile.count() // Number of items in this RDD
res0: Long = 74
scala> textFile.first() // First item in this RDD
-res1: String = Welcome to the Spark documentation!
+res1: String = # Apache Spark
{% endhighlight %}
Now let's use a transformation. We will use the [`filter`](scala-programming-guide.html#transformations) transformation to return a new RDD with a subset of the items in the file.
[38/50] git commit: Merge pull request #926 from kayousterhout/dynamic
Posted by rx...@apache.org.
Merge pull request #926 from kayousterhout/dynamic
Changed localProperties to use ThreadLocal (not DynamicVariable).
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/a310de69
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/a310de69
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/a310de69
Branch: refs/heads/master
Commit: a310de69e05c30fc86a7ea4bab503b7fe59cada9
Parents: 58c7d8b 93c4253
Author: Patrick Wendell <pw...@gmail.com>
Authored: Wed Sep 11 19:36:11 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Wed Sep 11 19:36:11 2013 -0700
----------------------------------------------------------------------
.../scala/org/apache/spark/SparkContext.scala | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
[24/50] git commit: Fix HDFS access bug with assembly build.
Posted by rx...@apache.org.
Fix HDFS access bug with assembly build.
Due to this change in HDFS:
https://issues.apache.org/jira/browse/HADOOP-7549
there is a bug when using the new assembly builds. The symptom is that any HDFS access
results in an exception saying "No filesystem for scheme 'hdfs'". This adds a merge
strategy in the assembly build which fixes the problem.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/0c1985b1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/0c1985b1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/0c1985b1
Branch: refs/heads/master
Commit: 0c1985b153a2dc2c891ae61c1ee67506926384ae
Parents: 2425eb8
Author: Patrick Wendell <pw...@gmail.com>
Authored: Tue Sep 10 21:38:22 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Tue Sep 10 22:05:13 2013 -0700
----------------------------------------------------------------------
assembly/pom.xml | 6 ++++++
project/SparkBuild.scala | 1 +
2 files changed, 7 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/0c1985b1/assembly/pom.xml
----------------------------------------------------------------------
diff --git a/assembly/pom.xml b/assembly/pom.xml
index d19f44d..808a829 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -106,6 +106,12 @@
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
+ <resource>META-INF/services/org.apache.hadoop.fs.FileSystem</resource>
+ </transformer>
+ </transformers>
+ <transformers>
+ <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
+ <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
</transformers>
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/0c1985b1/project/SparkBuild.scala
----------------------------------------------------------------------
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index a60b553..0736ec3 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -303,6 +303,7 @@ object SparkBuild extends Build {
mergeStrategy in assembly := {
case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
case m if m.toLowerCase.matches("meta-inf.*\\.sf$") => MergeStrategy.discard
+ case "META-INF/services/org.apache.hadoop.fs.FileSystem" => MergeStrategy.concat
case "reference.conf" => MergeStrategy.concat
case _ => MergeStrategy.first
}
[05/50] git commit: Merge pull request #910 from mateiz/ml-doc-tweaks
Posted by rx...@apache.org.
Merge pull request #910 from mateiz/ml-doc-tweaks
Small tweaks to MLlib docs
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/2447b1c4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/2447b1c4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/2447b1c4
Branch: refs/heads/master
Commit: 2447b1c4e6118a1f11901b36e8f9459d30fdc5c7
Parents: 7d3204b 7a5c4b6
Author: Matei Zaharia <ma...@gmail.com>
Authored: Sun Sep 8 22:27:49 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Sun Sep 8 22:27:49 2013 -0700
----------------------------------------------------------------------
docs/mllib-guide.md | 18 ++++++++----------
.../apache/spark/mllib/classification/SVM.scala | 13 +++++++------
.../spark/mllib/regression/LinearRegression.scala | 4 ++--
3 files changed, 17 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
[16/50] git commit: Fix copy issue in
https://github.com/mesos/spark/pull/899
Posted by rx...@apache.org.
Fix copy issue in https://github.com/mesos/spark/pull/899
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/f5a8afa6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/f5a8afa6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/f5a8afa6
Branch: refs/heads/master
Commit: f5a8afa6c38c6d3a3bf170e609468ff1d610f3ea
Parents: a85758c
Author: Matei Zaharia <ma...@eecs.berkeley.edu>
Authored: Mon Sep 9 13:47:56 2013 -0700
Committer: Matei Zaharia <ma...@eecs.berkeley.edu>
Committed: Mon Sep 9 13:47:56 2013 -0700
----------------------------------------------------------------------
make-distribution.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/f5a8afa6/make-distribution.sh
----------------------------------------------------------------------
diff --git a/make-distribution.sh b/make-distribution.sh
index 91f6278..bffb198 100755
--- a/make-distribution.sh
+++ b/make-distribution.sh
@@ -95,7 +95,7 @@ cp $FWDIR/assembly/target/scala*/*assembly*hadoop*.jar "$DISTDIR/jars/"
# Copy other things
mkdir "$DISTDIR"/conf
-cp -r "$FWDIR/conf/*.template" "$DISTDIR"
+cp "$FWDIR/conf/*.template" "$DISTDIR"/conf
cp -r "$FWDIR/bin" "$DISTDIR"
cp -r "$FWDIR/python" "$DISTDIR"
cp "$FWDIR/spark-class" "$DISTDIR"
[23/50] git commit: Update Python API features
Posted by rx...@apache.org.
Update Python API features
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/2425eb85
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/2425eb85
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/2425eb85
Branch: refs/heads/master
Commit: 2425eb85ca709273c48958f81a81c8a04657ea1f
Parents: 8c14f4b
Author: Matei Zaharia <ma...@eecs.berkeley.edu>
Authored: Tue Sep 10 11:12:59 2013 -0700
Committer: Matei Zaharia <ma...@eecs.berkeley.edu>
Committed: Tue Sep 10 11:12:59 2013 -0700
----------------------------------------------------------------------
docs/python-programming-guide.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/2425eb85/docs/python-programming-guide.md
----------------------------------------------------------------------
diff --git a/docs/python-programming-guide.md b/docs/python-programming-guide.md
index 5662e7d..f67a1cc 100644
--- a/docs/python-programming-guide.md
+++ b/docs/python-programming-guide.md
@@ -16,7 +16,7 @@ This guide will show how to use the Spark features described there in Python.
There are a few key differences between the Python and Scala APIs:
* Python is dynamically typed, so RDDs can hold objects of multiple types.
-* PySpark does not yet support a few API calls, such as `lookup`, `sort`, and `persist` at custom storage levels. See the [API docs](api/pyspark/index.html) for details.
+* PySpark does not yet support a few API calls, such as `lookup`, `sort`, and non-text input files, though these will be added in future releases.
In PySpark, RDDs support the same methods as their Scala counterparts but take Python functions and return Python collection types.
Short functions can be passed to RDD methods using Python's [`lambda`](http://www.diveintopython.net/power_of_introspection/lambda_functions.html) syntax:
[37/50] git commit: Merge pull request #927 from benh/mesos-docs
Posted by rx...@apache.org.
Merge pull request #927 from benh/mesos-docs
Updated Spark on Mesos documentation.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/58c7d8b1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/58c7d8b1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/58c7d8b1
Branch: refs/heads/master
Commit: 58c7d8b13875536f1b091a8fdc462ab52d075e36
Parents: 91a59e6 8e2602d
Author: Matei Zaharia <ma...@gmail.com>
Authored: Wed Sep 11 17:33:42 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Wed Sep 11 17:33:42 2013 -0700
----------------------------------------------------------------------
docs/running-on-mesos.md | 33 ++++++++++++++++-----------------
1 file changed, 16 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
[39/50] git commit: Add git scm url for publishing
Posted by rx...@apache.org.
Add git scm url for publishing
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/45ec1cc9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/45ec1cc9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/45ec1cc9
Branch: refs/heads/master
Commit: 45ec1cc90e25a82ffab587feeb419132efd1b9a1
Parents: a310de6
Author: Patrick Wendell <pw...@gmail.com>
Authored: Thu Sep 12 13:47:09 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Thu Sep 12 13:47:31 2013 -0700
----------------------------------------------------------------------
pom.xml | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/45ec1cc9/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index bb03a92..7bacbb1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -38,6 +38,7 @@
</licenses>
<scm>
<connection>scm:git:git@github.com:apache/incubator-spark.git</connection>
+ <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-spark.git</developerConnection>
<url>scm:git:git@github.com:apache/incubator-spark.git</url>
</scm>
<developers>
[02/50] git commit: Merge pull request #896 from atalwalkar/master
Posted by rx...@apache.org.
Merge pull request #896 from atalwalkar/master
updated content
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/f1f83712
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/f1f83712
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/f1f83712
Branch: refs/heads/master
Commit: f1f83712f4e166894e8de855545fa3f4309648f7
Parents: f68848d 81a8bd4
Author: Matei Zaharia <ma...@gmail.com>
Authored: Sun Sep 8 21:26:11 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Sun Sep 8 21:26:11 2013 -0700
----------------------------------------------------------------------
docs/mllib-guide.md | 187 ++++++++++++++++++++-
mllib/data/sample_svm_data.txt | 322 ++++++++++++++++++++++++++++++++++++
2 files changed, 508 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
[22/50] git commit: Add explicit jets3t dependency,
which is excluded in hadoop-client
Posted by rx...@apache.org.
Add explicit jets3t dependency, which is excluded in hadoop-client
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/f117dc6d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/f117dc6d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/f117dc6d
Branch: refs/heads/master
Commit: f117dc6d0d1349c28b7e1c8cf7729d1aafde5d25
Parents: 8c14f4b
Author: Matei Zaharia <ma...@gmail.com>
Authored: Tue Sep 10 06:39:25 2013 +0000
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Tue Sep 10 06:39:25 2013 +0000
----------------------------------------------------------------------
core/pom.xml | 5 +++++
pom.xml | 5 +++++
project/SparkBuild.scala | 1 +
3 files changed, 11 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/f117dc6d/core/pom.xml
----------------------------------------------------------------------
diff --git a/core/pom.xml b/core/pom.xml
index b5c5194..14cd520 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -37,6 +37,11 @@
<artifactId>hadoop-client</artifactId>
</dependency>
<dependency>
+ <groupId>net.java.dev.jets3t</groupId>
+ <artifactId>jets3t</artifactId>
+ <version>0.7.1</version>
+ </dependency>
+ <dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/f117dc6d/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index d05449e..bb03a92 100644
--- a/pom.xml
+++ b/pom.xml
@@ -374,6 +374,11 @@
</exclusions>
</dependency>
<dependency>
+ <groupId>net.java.dev.jets3t</groupId>
+ <artifactId>jets3t</artifactId>
+ <version>0.7.1</version>
+ </dependency>
+ <dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-api</artifactId>
<version>${hadoop.version}</version>
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/f117dc6d/project/SparkBuild.scala
----------------------------------------------------------------------
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index a60b553..087c6d8 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -208,6 +208,7 @@ object SparkBuild extends Build {
"io.netty" % "netty-all" % "4.0.0.Beta2",
"org.apache.derby" % "derby" % "10.4.2.0" % "test",
"org.apache.hadoop" % "hadoop-client" % hadoopVersion excludeAll(excludeJackson, excludeNetty, excludeAsm),
+ "net.java.dev.jets3t" % "jets3t" % "0.7.1",
"org.apache.avro" % "avro" % "1.7.4",
"org.apache.avro" % "avro-ipc" % "1.7.4" excludeAll(excludeNetty),
"com.codahale.metrics" % "metrics-core" % "3.0.0",
[30/50] git commit: Merge pull request #925 from davidmccauley/master
Posted by rx...@apache.org.
Merge pull request #925 from davidmccauley/master
SPARK-894 - Not all WebUI fields delivered VIA JSON
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/e07eef83
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/e07eef83
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/e07eef83
Branch: refs/heads/master
Commit: e07eef83af47fd2c3b8db7be4d19dce4b1dc3609
Parents: 8432f27 5dd875c
Author: Reynold Xin <re...@gmail.com>
Authored: Wed Sep 11 07:35:39 2013 -0700
Committer: Reynold Xin <re...@gmail.com>
Committed: Wed Sep 11 07:35:39 2013 -0700
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
[49/50] git commit: Merge pull request #938 from ilikerps/master
Posted by rx...@apache.org.
Merge pull request #938 from ilikerps/master
Fix issue with spark_ec2 seeing empty security groups
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/cd7222c3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/cd7222c3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/cd7222c3
Branch: refs/heads/master
Commit: cd7222c3dd2211ce790fa52110db911b862bb63b
Parents: 2aff798 f589ce7
Author: Patrick Wendell <pw...@gmail.com>
Authored: Thu Sep 19 14:21:24 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Thu Sep 19 14:21:24 2013 -0700
----------------------------------------------------------------------
ec2/spark_ec2.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
[47/50] git commit: Merge pull request #933 from jey/yarn-typo-fix
Posted by rx...@apache.org.
Merge pull request #933 from jey/yarn-typo-fix
Fix typo in Maven build docs
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/2aff7989
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/2aff7989
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/2aff7989
Branch: refs/heads/master
Commit: 2aff7989ab617f33052098498119886c40794774
Parents: dbd2c4f ac0dd99
Author: Patrick Wendell <pw...@gmail.com>
Authored: Sun Sep 15 14:05:04 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Sun Sep 15 14:05:04 2013 -0700
----------------------------------------------------------------------
docs/building-with-maven.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
[20/50] git commit: Document fortran dependency for MLBase
Posted by rx...@apache.org.
Document fortran dependency for MLBase
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/cefee1ed
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/cefee1ed
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/cefee1ed
Branch: refs/heads/master
Commit: cefee1ed1aa0fca086c54c7d0aa87f98571180c1
Parents: c81377b
Author: Patrick Wendell <pw...@gmail.com>
Authored: Mon Sep 9 21:45:04 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Mon Sep 9 21:45:04 2013 -0700
----------------------------------------------------------------------
docs/mllib-guide.md | 7 +++++++
1 file changed, 7 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/cefee1ed/docs/mllib-guide.md
----------------------------------------------------------------------
diff --git a/docs/mllib-guide.md b/docs/mllib-guide.md
index 1583ce4..f991d86 100644
--- a/docs/mllib-guide.md
+++ b/docs/mllib-guide.md
@@ -11,6 +11,13 @@ filtering, as well as an underlying gradient descent optimization primitive.
This guide will outline the functionality supported in MLlib and also provides
an example of invoking MLlib.
+# Dependencies
+MLlib uses the [jblas](https://github.com/mikiobraun/jblas) linear algebra library, which itself
+depends on native Fortran routines. You may need to install the
+[gfortran runtime library](https://github.com/mikiobraun/jblas/wiki/Missing-Libraries)
+if it is not already present on your nodes. MLlib will throw a linking error if it cannot
+detect these libraries automatically.
+
# Binary Classification
Binary classification is a supervised learning problem in which we want to
[15/50] git commit: Merge pull request #907 from
stephenh/document_coalesce_shuffle
Posted by rx...@apache.org.
Merge pull request #907 from stephenh/document_coalesce_shuffle
Add better docs for coalesce.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/a85758c2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/a85758c2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/a85758c2
Branch: refs/heads/master
Commit: a85758c20006537e7863cf30bc3c4b6b6efd4981
Parents: 084fc36 59003d3
Author: Matei Zaharia <ma...@gmail.com>
Authored: Mon Sep 9 13:45:40 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Mon Sep 9 13:45:40 2013 -0700
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/rdd/RDD.scala | 17 +++++++++++++++++
.../test/scala/org/apache/spark/rdd/RDDSuite.scala | 14 ++++++++++----
2 files changed, 27 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
[43/50] git commit: Bumping Mesos version to 0.13.0
Posted by rx...@apache.org.
Bumping Mesos version to 0.13.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/c856860c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/c856860c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/c856860c
Branch: refs/heads/master
Commit: c856860c5b4b0e3ef6f8e23d72d1d0069e3a56bf
Parents: e9eba8c
Author: Patrick Wendell <pw...@gmail.com>
Authored: Sun Sep 15 12:46:26 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Sun Sep 15 12:46:26 2013 -0700
----------------------------------------------------------------------
docs/_config.yml | 2 +-
pom.xml | 2 +-
project/SparkBuild.scala | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/c856860c/docs/_config.yml
----------------------------------------------------------------------
diff --git a/docs/_config.yml b/docs/_config.yml
index 5c135a0..b061764 100644
--- a/docs/_config.yml
+++ b/docs/_config.yml
@@ -6,5 +6,5 @@ markdown: kramdown
SPARK_VERSION: 0.8.0-SNAPSHOT
SPARK_VERSION_SHORT: 0.8.0
SCALA_VERSION: 2.9.3
-MESOS_VERSION: 0.9.0-incubating
+MESOS_VERSION: 0.13.0
SPARK_ISSUE_TRACKER_URL: https://spark-project.atlassian.net
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/c856860c/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 2da016a..4aed126 100644
--- a/pom.xml
+++ b/pom.xml
@@ -77,7 +77,7 @@
<java.version>1.5</java.version>
<scala.version>2.9.3</scala.version>
- <mesos.version>0.12.1</mesos.version>
+ <mesos.version>0.13.0</mesos.version>
<akka.version>2.0.5</akka.version>
<slf4j.version>1.7.2</slf4j.version>
<log4j.version>1.2.17</log4j.version>
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/c856860c/project/SparkBuild.scala
----------------------------------------------------------------------
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 77e211c..ed76717 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -204,7 +204,7 @@ object SparkBuild extends Build {
"it.unimi.dsi" % "fastutil" % "6.4.4",
"colt" % "colt" % "1.2.0",
"net.liftweb" % "lift-json_2.9.2" % "2.5",
- "org.apache.mesos" % "mesos" % "0.12.1",
+ "org.apache.mesos" % "mesos" % "0.13.0",
"io.netty" % "netty-all" % "4.0.0.Beta2",
"org.apache.derby" % "derby" % "10.4.2.0" % "test",
"org.apache.hadoop" % "hadoop-client" % hadoopVersion excludeAll(excludeJackson, excludeNetty, excludeAsm),
[48/50] git commit: Fix issue with spark_ec2 seeing empty security
groups
Posted by rx...@apache.org.
Fix issue with spark_ec2 seeing empty security groups
Under unknown, but occasional, circumstances, reservation.groups is empty
despite reservation.instances each having groups. This means that the
spark_ec2 get_existing_clusters() method would fail to find any instances.
To fix it, we simply use the instances' groups as the source of truth.
Note that this is actually just a revival of PR #827, now that the issue
has been reproduced.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/f589ce77
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/f589ce77
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/f589ce77
Branch: refs/heads/master
Commit: f589ce771a53dcbfda5f62fe0ac77164688ecc76
Parents: 2aff798
Author: Aaron Davidson <aa...@databricks.com>
Authored: Thu Sep 19 14:09:26 2013 -0700
Committer: Aaron Davidson <aa...@databricks.com>
Committed: Thu Sep 19 14:09:26 2013 -0700
----------------------------------------------------------------------
ec2/spark_ec2.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/f589ce77/ec2/spark_ec2.py
----------------------------------------------------------------------
diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py
index 419d0fe..6b7d202 100755
--- a/ec2/spark_ec2.py
+++ b/ec2/spark_ec2.py
@@ -364,12 +364,12 @@ def get_existing_cluster(conn, opts, cluster_name, die_on_error=True):
slave_nodes = []
for res in reservations:
active = [i for i in res.instances if is_active(i)]
- if len(active) > 0:
- group_names = [g.name for g in res.groups]
+ for inst in active:
+ group_names = [g.name for g in inst.groups]
if group_names == [cluster_name + "-master"]:
- master_nodes += res.instances
+ master_nodes.append(inst)
elif group_names == [cluster_name + "-slaves"]:
- slave_nodes += res.instances
+ slave_nodes.append(inst)
if any((master_nodes, slave_nodes)):
print ("Found %d master(s), %d slaves" %
(len(master_nodes), len(slave_nodes)))
[31/50] git commit: Change port from 3030 to 4040
Posted by rx...@apache.org.
Change port from 3030 to 4040
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/bddf1356
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/bddf1356
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/bddf1356
Branch: refs/heads/master
Commit: bddf1356708f42352934313c82f48dbce0056a68
Parents: 2425eb8
Author: Patrick Wendell <pw...@gmail.com>
Authored: Tue Sep 10 23:12:27 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Wed Sep 11 10:01:38 2013 -0700
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/ui/SparkUI.scala | 2 +-
core/src/test/scala/org/apache/spark/ui/UISuite.scala | 2 +-
docs/cluster-overview.md | 4 ++--
docs/configuration.md | 2 +-
docs/hardware-provisioning.md | 4 ++--
docs/monitoring.md | 6 +++---
ec2/spark_ec2.py | 3 +--
7 files changed, 11 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/bddf1356/core/src/main/scala/org/apache/spark/ui/SparkUI.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/ui/SparkUI.scala b/core/src/main/scala/org/apache/spark/ui/SparkUI.scala
index 48eb096..f1d86c0 100644
--- a/core/src/main/scala/org/apache/spark/ui/SparkUI.scala
+++ b/core/src/main/scala/org/apache/spark/ui/SparkUI.scala
@@ -84,6 +84,6 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
}
private[spark] object SparkUI {
- val DEFAULT_PORT = "3030"
+ val DEFAULT_PORT = "4040"
val STATIC_RESOURCE_DIR = "org/apache/spark/ui/static"
}
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/bddf1356/core/src/test/scala/org/apache/spark/ui/UISuite.scala
----------------------------------------------------------------------
diff --git a/core/src/test/scala/org/apache/spark/ui/UISuite.scala b/core/src/test/scala/org/apache/spark/ui/UISuite.scala
index 3321fb5..07c9f23 100644
--- a/core/src/test/scala/org/apache/spark/ui/UISuite.scala
+++ b/core/src/test/scala/org/apache/spark/ui/UISuite.scala
@@ -24,7 +24,7 @@ import org.eclipse.jetty.server.Server
class UISuite extends FunSuite {
test("jetty port increases under contention") {
- val startPort = 3030
+ val startPort = 4040
val server = new Server(startPort)
server.start()
val (jettyServer1, boundPort1) = JettyUtils.startJettyServer("localhost", startPort, Seq())
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/bddf1356/docs/cluster-overview.md
----------------------------------------------------------------------
diff --git a/docs/cluster-overview.md b/docs/cluster-overview.md
index 7025c23..f679cad 100644
--- a/docs/cluster-overview.md
+++ b/docs/cluster-overview.md
@@ -59,8 +59,8 @@ and `addFile`.
# Monitoring
-Each driver program has a web UI, typically on port 3030, that displays information about running
-tasks, executors, and storage usage. Simply go to `http://<driver-node>:3030` in a web browser to
+Each driver program has a web UI, typically on port 4040, that displays information about running
+tasks, executors, and storage usage. Simply go to `http://<driver-node>:4040` in a web browser to
access this UI. The [monitoring guide](monitoring.html) also describes other monitoring options.
# Job Scheduling
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/bddf1356/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index d4f8553..7940d41 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -111,7 +111,7 @@ Apart from these, the following properties are also available, and may be useful
</tr>
<tr>
<td>spark.ui.port</td>
- <td>3030</td>
+ <td>4040</td>
<td>
Port for your application's dashboard, which shows memory and workload data
</td>
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/bddf1356/docs/hardware-provisioning.md
----------------------------------------------------------------------
diff --git a/docs/hardware-provisioning.md b/docs/hardware-provisioning.md
index e5f054c..7902205 100644
--- a/docs/hardware-provisioning.md
+++ b/docs/hardware-provisioning.md
@@ -43,7 +43,7 @@ rest for the operating system and buffer cache.
How much memory you will need will depend on your application. To determine how much your
application uses for a certain dataset size, load part of your dataset in a Spark RDD and use the
-Storage tab of Spark's monitoring UI (`http://<driver-node>:3030`) to see its size in memory.
+Storage tab of Spark's monitoring UI (`http://<driver-node>:4040`) to see its size in memory.
Note that memory usage is greatly affected by storage level and serialization format -- see
the [tuning guide](tuning.html) for tips on how to reduce it.
@@ -59,7 +59,7 @@ In our experience, when the data is in memory, a lot of Spark applications are n
Using a **10 Gigabit** or higher network is the best way to make these applications faster.
This is especially true for "distributed reduce" applications such as group-bys, reduce-bys, and
SQL joins. In any given application, you can see how much data Spark shuffles across the network
-from the application's monitoring UI (`http://<driver-node>:3030`).
+from the application's monitoring UI (`http://<driver-node>:4040`).
# CPU Cores
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/bddf1356/docs/monitoring.md
----------------------------------------------------------------------
diff --git a/docs/monitoring.md b/docs/monitoring.md
index 0e3606f..5f456b9 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -7,7 +7,7 @@ There are several ways to monitor Spark applications.
# Web Interfaces
-Every SparkContext launches a web UI, by default on port 3030, that
+Every SparkContext launches a web UI, by default on port 4040, that
displays useful information about the application. This includes:
* A list of scheduler stages and tasks
@@ -15,9 +15,9 @@ displays useful information about the application. This includes:
* Information about the running executors
* Environmental information.
-You can access this interface by simply opening `http://<driver-node>:3030` in a web browser.
+You can access this interface by simply opening `http://<driver-node>:4040` in a web browser.
If multiple SparkContexts are running on the same host, they will bind to succesive ports
-beginning with 3030 (3031, 3032, etc).
+beginning with 4040 (4041, 4042, etc).
Spark's Standlone Mode cluster manager also has its own
[web UI](spark-standalone.html#monitoring-and-logging).
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/bddf1356/ec2/spark_ec2.py
----------------------------------------------------------------------
diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py
index 932e70d..419d0fe 100755
--- a/ec2/spark_ec2.py
+++ b/ec2/spark_ec2.py
@@ -215,11 +215,10 @@ def launch_cluster(conn, opts, cluster_name):
master_group.authorize(src_group=slave_group)
master_group.authorize('tcp', 22, 22, '0.0.0.0/0')
master_group.authorize('tcp', 8080, 8081, '0.0.0.0/0')
- master_group.authorize('tcp', 33000, 33000, '0.0.0.0/0')
master_group.authorize('tcp', 50030, 50030, '0.0.0.0/0')
master_group.authorize('tcp', 50070, 50070, '0.0.0.0/0')
master_group.authorize('tcp', 60070, 60070, '0.0.0.0/0')
- master_group.authorize('tcp', 3030, 3035, '0.0.0.0/0')
+ master_group.authorize('tcp', 4040, 4045, '0.0.0.0/0')
if opts.ganglia:
master_group.authorize('tcp', 5080, 5080, '0.0.0.0/0')
if slave_group.rules == []: # Group was just now created
[36/50] git commit: More updates to Spark on Mesos documentation.
Posted by rx...@apache.org.
More updates to Spark on Mesos documentation.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/8e2602dd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/8e2602dd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/8e2602dd
Branch: refs/heads/master
Commit: 8e2602dd7033deded36d225250f30d980bfa6ecd
Parents: a0f0c1b
Author: Benjamin Hindman <be...@gmail.com>
Authored: Wed Sep 11 16:08:54 2013 -0700
Committer: Benjamin Hindman <be...@gmail.com>
Committed: Wed Sep 11 16:08:54 2013 -0700
----------------------------------------------------------------------
docs/running-on-mesos.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/8e2602dd/docs/running-on-mesos.md
----------------------------------------------------------------------
diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md
index 443350c..322ff58 100644
--- a/docs/running-on-mesos.md
+++ b/docs/running-on-mesos.md
@@ -10,12 +10,12 @@ Spark can run on clusters managed by [Apache Mesos](http://mesos.apache.org/). F
3. Create a Spark "distribution" using `make-distribution.sh`.
4. Rename the `dist` directory created from `make-distribution.sh` to `spark-{{site.SPARK_VERSION}}`.
5. Create a `tar` archive: `tar czf spark-{{site.SPARK_VERSION}}.tar.gz spark-{{site.SPARK_VERSION}}`
-6. Upload this archive to your HDFS or another place accessible from Mesos via `http://`, e.g., [Amazon Simple Storage Service](http://aws.amazon.com/s3): `hadoop fs -put spark-{{site.SPARK_VERSION}}.tar.gz /path/to/spark-{{site.SPARK_VERSION}}.tar.gz`
+6. Upload this archive to HDFS or another place accessible from Mesos via `http://`, e.g., [Amazon Simple Storage Service](http://aws.amazon.com/s3): `hadoop fs -put spark-{{site.SPARK_VERSION}}.tar.gz /path/to/spark-{{site.SPARK_VERSION}}.tar.gz`
7. Create a file called `spark-env.sh` in Spark's `conf` directory, by copying `conf/spark-env.sh.template`, and add the following lines to it:
* `export MESOS_NATIVE_LIBRARY=<path to libmesos.so>`. This path is usually `<prefix>/lib/libmesos.so` (where the prefix is `/usr/local` by default, see above). Also, on Mac OS X, the library is called `libmesos.dylib` instead of `libmesos.so`.
* `export SPARK_EXECUTOR_URI=<path to spark-{{site.SPARK_VERSION}}.tar.gz uploaded above>`.
* `export MASTER=mesos://HOST:PORT` where HOST:PORT is the host and port (default: 5050) of your Mesos master (or `zk://...` if using Mesos with ZooKeeper).
-8. To run a Spark application against the cluster, when you create your `SparkContext`, pass the string `mesos://HOST:PORT` as the first parameter. In addition, you'll need to set the `spark.executor.uri` property. For example
+8. To run a Spark application against the cluster, when you create your `SparkContext`, pass the string `mesos://HOST:PORT` as the first parameter. In addition, you'll need to set the `spark.executor.uri` property. For example:
{% highlight scala %}
System.setProperty("spark.executor.uri", "<path to spark-{{site.SPARK_VERSION}}.tar.gz uploaded above>")
[09/50] git commit: Reword 'evenly distributed' to 'distributed with
a hash partitioner.
Posted by rx...@apache.org.
Reword 'evenly distributed' to 'distributed with a hash partitioner.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/6471bfec
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/6471bfec
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/6471bfec
Branch: refs/heads/master
Commit: 6471bfec736b033a94a96575255f2d4453082709
Parents: df5fd35
Author: Stephen Haberman <st...@exigencecorp.com>
Authored: Mon Sep 9 11:44:15 2013 -0500
Committer: Stephen Haberman <st...@exigencecorp.com>
Committed: Mon Sep 9 11:44:15 2013 -0500
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/rdd/RDD.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/6471bfec/core/src/main/scala/org/apache/spark/rdd/RDD.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/rdd/RDD.scala b/core/src/main/scala/org/apache/spark/rdd/RDD.scala
index 41a90f1..1082cba 100644
--- a/core/src/main/scala/org/apache/spark/rdd/RDD.scala
+++ b/core/src/main/scala/org/apache/spark/rdd/RDD.scala
@@ -282,8 +282,8 @@ abstract class RDD[T: ClassManifest](
* Note: With shuffle = true, you can actually coalesce to a larger number
* of partitions. This is useful if you have a small number of partitions,
* say 100, potentially with a few partitions being abnormally large. Calling
- * coalecse(1000, shuffle = true) will result in 1000 partitions with the
- * data evenly distributed into each partition.
+ * coalesce(1000, shuffle = true) will result in 1000 partitions with the
+ * data distributed using a hash partitioner.
*/
def coalesce(numPartitions: Int, shuffle: Boolean = false): RDD[T] = {
if (shuffle) {
[10/50] git commit: Use a set since shuffle could change order.
Posted by rx...@apache.org.
Use a set since shuffle could change order.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/59003d38
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/59003d38
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/59003d38
Branch: refs/heads/master
Commit: 59003d387d399d6280a42b4c79eef18cfdbfe72b
Parents: 6471bfe
Author: Stephen Haberman <st...@exigencecorp.com>
Authored: Mon Sep 9 11:45:03 2013 -0500
Committer: Stephen Haberman <st...@exigencecorp.com>
Committed: Mon Sep 9 11:45:03 2013 -0500
----------------------------------------------------------------------
core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/59003d38/core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
----------------------------------------------------------------------
diff --git a/core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala b/core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
index 6096149..c1df5e1 100644
--- a/core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
+++ b/core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
@@ -179,7 +179,7 @@ class RDDSuite extends FunSuite with SharedSparkContext {
// when shuffling, we can increase the number of partitions
val coalesced6 = data.coalesce(20, shuffle = true)
assert(coalesced6.partitions.size === 20)
- assert(coalesced6.collect().toList === (1 to 10).toList)
+ assert(coalesced6.collect().toSet === (1 to 10).toSet)
}
test("coalesced RDDs with locality") {
[46/50] git commit: Fix typo in Maven build docs
Posted by rx...@apache.org.
Fix typo in Maven build docs
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/ac0dd993
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/ac0dd993
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/ac0dd993
Branch: refs/heads/master
Commit: ac0dd99394faf9e0389d5a1340b729c02a55c363
Parents: dbd2c4f
Author: Jey Kottalam <je...@cs.berkeley.edu>
Authored: Sun Sep 15 13:29:22 2013 -0700
Committer: Jey Kottalam <je...@cs.berkeley.edu>
Committed: Sun Sep 15 13:29:22 2013 -0700
----------------------------------------------------------------------
docs/building-with-maven.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/ac0dd993/docs/building-with-maven.md
----------------------------------------------------------------------
diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md
index 3065427..19c01e1 100644
--- a/docs/building-with-maven.md
+++ b/docs/building-with-maven.md
@@ -40,10 +40,10 @@ For Apache Hadoop versions 1.x, Cloudera CDH MRv1, and other Hadoop versions wit
For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you should enable the "hadoop2-yarn" profile and set the "yarn.version" property:
# Apache Hadoop 2.0.5-alpha
- $ mvn -Phadoop2-yarn -Dhadoop.version=2.0.5-alpha -Pyarn.version=2.0.5-alpha -DskipTests clean package
+ $ mvn -Phadoop2-yarn -Dhadoop.version=2.0.5-alpha -Dyarn.version=2.0.5-alpha -DskipTests clean package
# Cloudera CDH 4.2.0 with MapReduce v2
- $ mvn -Phadoop2-yarn -Dhadoop.version=2.0.0-cdh4.2.0 -Pyarn.version=2.0.0-chd4.2.0 -DskipTests clean package
+ $ mvn -Phadoop2-yarn -Dhadoop.version=2.0.0-cdh4.2.0 -Dyarn.version=2.0.0-chd4.2.0 -DskipTests clean package
## Spark Tests in Maven ##
[04/50] git commit: Small tweaks to MLlib docs
Posted by rx...@apache.org.
Small tweaks to MLlib docs
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/7a5c4b64
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/7a5c4b64
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/7a5c4b64
Branch: refs/heads/master
Commit: 7a5c4b647bdd12d2a6d6285d0349680f1a848b01
Parents: 7d3204b
Author: Matei Zaharia <ma...@eecs.berkeley.edu>
Authored: Sun Sep 8 21:47:24 2013 -0700
Committer: Matei Zaharia <ma...@eecs.berkeley.edu>
Committed: Sun Sep 8 21:47:24 2013 -0700
----------------------------------------------------------------------
docs/mllib-guide.md | 18 ++++++++----------
.../apache/spark/mllib/classification/SVM.scala | 13 +++++++------
.../spark/mllib/regression/LinearRegression.scala | 4 ++--
3 files changed, 17 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/7a5c4b64/docs/mllib-guide.md
----------------------------------------------------------------------
diff --git a/docs/mllib-guide.md b/docs/mllib-guide.md
index 1a62999..1583ce4 100644
--- a/docs/mllib-guide.md
+++ b/docs/mllib-guide.md
@@ -4,7 +4,7 @@ title: Machine Learning Library (MLlib)
---
MLlib is a Spark implementation of some common machine learning (ML)
-functionality, as well associated unit tests and data generators. MLlib
+functionality, as well associated tests and data generators. MLlib
currently supports four common types of machine learning problem settings,
namely, binary classification, regression, clustering and collaborative
filtering, as well as an underlying gradient descent optimization primitive.
@@ -44,22 +44,20 @@ import org.apache.spark.mllib.regression.LabeledPoint
// Load and parse the data file
val data = sc.textFile("mllib/data/sample_svm_data.txt")
-val parsedData = data.map(line => {
+val parsedData = data.map { line =>
val parts = line.split(' ')
LabeledPoint(parts(0).toDouble, parts.tail.map(x => x.toDouble).toArray)
-})
+}
// Run training algorithm
val numIterations = 20
-val model = SVMWithSGD.train(
- parsedData,
- numIterations)
+val model = SVMWithSGD.train(parsedData, numIterations)
// Evaluate model on training examples and compute training error
-val labelAndPreds = parsedData.map(r => {
- val prediction = model.predict(r.features)
- (r.label, prediction)
-})
+val labelAndPreds = parsedData.map { point =>
+ val prediction = model.predict(point.features)
+ (point.label, prediction)
+}
val trainErr = labelAndPreds.filter(r => r._1 != r._2).count.toDouble / parsedData.count
println("trainError = " + trainErr)
{% endhighlight %}
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/7a5c4b64/mllib/src/main/scala/org/apache/spark/mllib/classification/SVM.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/classification/SVM.scala b/mllib/src/main/scala/org/apache/spark/mllib/classification/SVM.scala
index 3511e24..3b8f855 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/classification/SVM.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/classification/SVM.scala
@@ -29,7 +29,7 @@ import org.apache.spark.mllib.util.DataValidators
import org.jblas.DoubleMatrix
/**
- * Model built using SVM.
+ * Model for Support Vector Machines (SVMs).
*
* @param weights Weights computed for every feature.
* @param intercept Intercept computed for this model.
@@ -48,8 +48,8 @@ class SVMModel(
}
/**
- * Train an SVM using Stochastic Gradient Descent.
- * NOTE: Labels used in SVM should be {0, 1}
+ * Train a Support Vector Machine (SVM) using Stochastic Gradient Descent.
+ * NOTE: Labels used in SVM should be {0, 1}.
*/
class SVMWithSGD private (
var stepSize: Double,
@@ -79,7 +79,7 @@ class SVMWithSGD private (
}
/**
- * Top-level methods for calling SVM. NOTE: Labels used in SVM should be {0, 1}
+ * Top-level methods for calling SVM. NOTE: Labels used in SVM should be {0, 1}.
*/
object SVMWithSGD {
@@ -88,14 +88,15 @@ object SVMWithSGD {
* of iterations of gradient descent using the specified step size. Each iteration uses
* `miniBatchFraction` fraction of the data to calculate the gradient. The weights used in
* gradient descent are initialized using the initial weights provided.
- * NOTE: Labels used in SVM should be {0, 1}
+ *
+ * NOTE: Labels used in SVM should be {0, 1}.
*
* @param input RDD of (label, array of features) pairs.
* @param numIterations Number of iterations of gradient descent to run.
* @param stepSize Step size to be used for each iteration of gradient descent.
* @param regParam Regularization parameter.
* @param miniBatchFraction Fraction of data to be used per iteration.
- * @param initialWeights Initial set of weights to be used. Array should be equal in size to
+ * @param initialWeights Initial set of weights to be used. Array should be equal in size to
* the number of features in the data.
*/
def train(
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/7a5c4b64/mllib/src/main/scala/org/apache/spark/mllib/regression/LinearRegression.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/LinearRegression.scala b/mllib/src/main/scala/org/apache/spark/mllib/regression/LinearRegression.scala
index ae95ea2..597d55e 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/regression/LinearRegression.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/regression/LinearRegression.scala
@@ -43,7 +43,7 @@ class LinearRegressionModel(
}
/**
- * Train a regression model with no regularization using Stochastic Gradient Descent.
+ * Train a linear regression model with no regularization using Stochastic Gradient Descent.
*/
class LinearRegressionWithSGD private (
var stepSize: Double,
@@ -83,7 +83,7 @@ object LinearRegressionWithSGD {
* @param numIterations Number of iterations of gradient descent to run.
* @param stepSize Step size to be used for each iteration of gradient descent.
* @param miniBatchFraction Fraction of data to be used per iteration.
- * @param initialWeights Initial set of weights to be used. Array should be equal in size to
+ * @param initialWeights Initial set of weights to be used. Array should be equal in size to
* the number of features in the data.
*/
def train(
[41/50] git commit: Merge pull request #929 from pwendell/master
Posted by rx...@apache.org.
Merge pull request #929 from pwendell/master
Use different Hadoop version for YARN artifacts.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/c4c1db2d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/c4c1db2d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/c4c1db2d
Branch: refs/heads/master
Commit: c4c1db2dd5b2ec0a8182369ecdb0e14f4e199822
Parents: 45ec1cc e9eba8c
Author: Patrick Wendell <pw...@gmail.com>
Authored: Fri Sep 13 19:52:12 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Fri Sep 13 19:52:12 2013 -0700
----------------------------------------------------------------------
pom.xml | 11 ++++++-----
yarn/pom.xml | 5 +++++
2 files changed, 11 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
[50/50] git commit: Merge branch 'master' of github.com:mesos/spark
Posted by rx...@apache.org.
Merge branch 'master' of github.com:mesos/spark
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/119de802
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/119de802
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/119de802
Branch: refs/heads/master
Commit: 119de80294bd0cb82855bd1982c5371b661b6fd5
Parents: a106ed8 cd7222c
Author: Reynold Xin <re...@gmail.com>
Authored: Fri Sep 20 15:03:55 2013 -0700
Committer: Reynold Xin <re...@gmail.com>
Committed: Fri Sep 20 15:03:55 2013 -0700
----------------------------------------------------------------------
assembly/README | 11 +-
assembly/pom.xml | 6 +
conf/fairscheduler.xml.template | 18 +-
conf/metrics.properties.template | 50 ++-
core/pom.xml | 9 +
.../scala/org/apache/spark/CacheManager.scala | 8 +-
.../scala/org/apache/spark/SparkContext.scala | 25 +-
.../main/scala/org/apache/spark/SparkEnv.scala | 2 +-
.../scala/org/apache/spark/TaskContext.scala | 1 +
.../org/apache/spark/deploy/DeployMessage.scala | 4 +
.../org/apache/spark/deploy/JsonProtocol.scala | 4 +-
.../apache/spark/deploy/LocalSparkCluster.scala | 2 +-
.../org/apache/spark/deploy/master/Master.scala | 21 +-
.../org/apache/spark/executor/Executor.scala | 2 +-
.../apache/spark/executor/ExecutorSource.scala | 5 +-
.../apache/spark/metrics/MetricsConfig.scala | 7 +-
.../apache/spark/metrics/sink/GangliaSink.scala | 82 +++++
.../spark/metrics/sink/MetricsServlet.scala | 11 +-
.../spark/network/netty/ShuffleSender.scala | 3 +-
.../main/scala/org/apache/spark/rdd/RDD.scala | 17 +
.../apache/spark/scheduler/DAGScheduler.scala | 12 +-
.../spark/scheduler/DAGSchedulerSource.scala | 6 +-
.../org/apache/spark/scheduler/ResultTask.scala | 6 +-
.../apache/spark/scheduler/ShuffleMapTask.scala | 2 +-
.../scheduler/cluster/ClusterScheduler.scala | 2 +-
.../cluster/ClusterTaskSetManager.scala | 4 +-
.../scheduler/cluster/SchedulableBuilder.scala | 6 +-
.../cluster/SparkDeploySchedulerBackend.scala | 12 +-
.../spark/scheduler/local/LocalScheduler.scala | 2 +-
.../spark/storage/BlockManagerSource.scala | 6 +-
.../org/apache/spark/storage/DiskStore.scala | 2 +-
.../org/apache/spark/storage/StorageLevel.scala | 6 +-
.../scala/org/apache/spark/ui/SparkUI.scala | 7 +-
.../scala/org/apache/spark/ui/UIUtils.scala | 39 ++-
.../apache/spark/ui/UIWorkloadGenerator.scala | 4 +-
.../spark/ui/jobs/JobProgressListener.scala | 2 +-
.../org/apache/spark/ui/jobs/PoolTable.scala | 3 +-
.../org/apache/spark/ui/jobs/StageTable.scala | 7 +-
.../org/apache/spark/ui/storage/IndexPage.scala | 2 +-
.../scala/org/apache/spark/util/Utils.scala | 26 +-
.../org/apache/spark/CacheManagerSuite.scala | 91 ++++++
.../scala/org/apache/spark/JavaAPISuite.java | 2 +-
.../apache/spark/deploy/JsonProtocolSuite.scala | 92 ++++++
.../spark/metrics/MetricsConfigSuite.scala | 19 +-
.../scala/org/apache/spark/rdd/RDDSuite.scala | 14 +-
.../cluster/ClusterSchedulerSuite.scala | 8 +-
.../scheduler/local/LocalSchedulerSuite.scala | 8 +-
.../scala/org/apache/spark/ui/UISuite.scala | 2 +-
docs/_config.yml | 2 +-
docs/_layouts/global.html | 9 +-
docs/building-with-maven.md | 6 +-
docs/cluster-overview.md | 117 +++++++
docs/configuration.md | 34 +-
docs/contributing-to-spark.md | 24 +-
docs/ec2-scripts.md | 8 +-
docs/hadoop-third-party-distributions.md | 118 +++++++
docs/hardware-provisioning.md | 4 +-
docs/img/cluster-overview.png | Bin 0 -> 28011 bytes
docs/img/cluster-overview.pptx | Bin 0 -> 51771 bytes
docs/img/spark-logo-hd.png | Bin 0 -> 13512 bytes
docs/index.md | 26 +-
docs/job-scheduling.md | 168 ++++++++++
docs/mllib-guide.md | 192 ++++++++++-
docs/monitoring.md | 70 ++++
docs/python-programming-guide.md | 14 +-
docs/quick-start.md | 70 ++--
docs/running-on-mesos.md | 48 ++-
docs/running-on-yarn.md | 35 +-
docs/scala-programming-guide.md | 4 +-
docs/spark-standalone.md | 46 ++-
.../root/spark-ec2/ec2-variables.sh | 17 +
ec2/spark_ec2.py | 11 +-
examples/pom.xml | 14 -
make-distribution.sh | 2 +-
mllib/data/sample_svm_data.txt | 322 +++++++++++++++++++
.../apache/spark/mllib/classification/SVM.scala | 13 +-
.../mllib/regression/LinearRegression.scala | 4 +-
pom.xml | 236 +++++++-------
project/SparkBuild.scala | 17 +-
project/project/SparkPluginBuild.scala | 19 +-
python/pyspark/__init__.py | 5 +-
python/pyspark/context.py | 11 +
python/pyspark/rdd.py | 19 ++
python/pyspark/shell.py | 1 +
python/pyspark/storagelevel.py | 43 +++
repl-bin/pom.xml | 10 -
repl/pom.xml | 12 -
run-example | 8 +-
yarn/pom.xml | 66 ++--
.../spark/deploy/yarn/ApplicationMaster.scala | 78 +++--
.../yarn/ApplicationMasterArguments.scala | 2 +-
.../org/apache/spark/deploy/yarn/Client.scala | 5 +-
.../spark/deploy/yarn/ClientArguments.scala | 2 +-
.../spark/deploy/yarn/WorkerRunnable.scala | 3 +-
.../deploy/yarn/YarnAllocationHandler.scala | 13 +-
.../cluster/YarnClusterScheduler.scala | 3 +
96 files changed, 2068 insertions(+), 543 deletions(-)
----------------------------------------------------------------------
[32/50] git commit: Merge pull request #922 from pwendell/port-change
Posted by rx...@apache.org.
Merge pull request #922 from pwendell/port-change
Change default port number from 3030 to 4030.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/b9128d34
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/b9128d34
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/b9128d34
Branch: refs/heads/master
Commit: b9128d34bfcd44692bcae5771a17800fed0d4bf4
Parents: e07eef8 bddf135
Author: Patrick Wendell <pw...@gmail.com>
Authored: Wed Sep 11 10:03:06 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Wed Sep 11 10:03:06 2013 -0700
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/ui/SparkUI.scala | 2 +-
core/src/test/scala/org/apache/spark/ui/UISuite.scala | 2 +-
docs/cluster-overview.md | 4 ++--
docs/configuration.md | 2 +-
docs/hardware-provisioning.md | 4 ++--
docs/monitoring.md | 6 +++---
ec2/spark_ec2.py | 3 +--
7 files changed, 11 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
[06/50] git commit: Merge pull request #909 from mateiz/exec-id-fix
Posted by rx...@apache.org.
Merge pull request #909 from mateiz/exec-id-fix
Fix an instance where full standalone mode executor IDs were passed to
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/e9d4f44a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/e9d4f44a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/e9d4f44a
Branch: refs/heads/master
Commit: e9d4f44a7adfaf55d0fd312b81350638310c341d
Parents: 2447b1c f9b7f58
Author: Reynold Xin <re...@gmail.com>
Authored: Sun Sep 8 23:36:48 2013 -0700
Committer: Reynold Xin <re...@gmail.com>
Committed: Sun Sep 8 23:36:48 2013 -0700
----------------------------------------------------------------------
.../spark/scheduler/cluster/ClusterTaskSetManager.scala | 4 ++--
.../scheduler/cluster/SparkDeploySchedulerBackend.scala | 10 +++++-----
2 files changed, 7 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
[07/50] git commit: Merge pull request #890 from mridulm/master
Posted by rx...@apache.org.
Merge pull request #890 from mridulm/master
Fix hash bug
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/bf984e27
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/bf984e27
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/bf984e27
Branch: refs/heads/master
Commit: bf984e27457fb64baa1c1b32bbc2d7dd645c98c4
Parents: e9d4f44 1e2474b
Author: Matei Zaharia <ma...@gmail.com>
Authored: Sun Sep 8 23:50:24 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Sun Sep 8 23:50:24 2013 -0700
----------------------------------------------------------------------
.../apache/spark/network/netty/ShuffleSender.scala | 3 ++-
.../scala/org/apache/spark/storage/DiskStore.scala | 2 +-
core/src/main/scala/org/apache/spark/util/Utils.scala | 14 ++++++++++++++
3 files changed, 17 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
[12/50] git commit: Add metrics-ganglia to core pom file
Posted by rx...@apache.org.
Add metrics-ganglia to core pom file
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/2186d932
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/2186d932
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/2186d932
Branch: refs/heads/master
Commit: 2186d93285f34308410fdf2129d2268b98494421
Parents: 0456384
Author: Y.CORP.YAHOO.COM\tgraves <tg...@thatenemy-lm.champ.corp.yahoo.com>
Authored: Mon Sep 9 12:37:33 2013 -0500
Committer: Y.CORP.YAHOO.COM\tgraves <tg...@thatenemy-lm.champ.corp.yahoo.com>
Committed: Mon Sep 9 12:37:33 2013 -0500
----------------------------------------------------------------------
core/pom.xml | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/2186d932/core/pom.xml
----------------------------------------------------------------------
diff --git a/core/pom.xml b/core/pom.xml
index 5738b74..b5c5194 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -147,6 +147,10 @@
<artifactId>metrics-json</artifactId>
</dependency>
<dependency>
+ <groupId>com.codahale.metrics</groupId>
+ <artifactId>metrics-ganglia</artifactId>
+ </dependency>
+ <dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<scope>test</scope>
[11/50] git commit: Merge pull request #911 from pwendell/ganglia-sink
Posted by rx...@apache.org.
Merge pull request #911 from pwendell/ganglia-sink
Adding Manen dependency for Ganglia
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/04563849
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/04563849
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/04563849
Branch: refs/heads/master
Commit: 0456384939f8312024456347953ba748053da2f9
Parents: bf984e2 528fdba
Author: Matei Zaharia <ma...@gmail.com>
Authored: Mon Sep 9 09:57:54 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Mon Sep 9 09:57:54 2013 -0700
----------------------------------------------------------------------
pom.xml | 5 +++++
1 file changed, 5 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/04563849/pom.xml
----------------------------------------------------------------------
[21/50] git commit: Merge pull request #917 from pwendell/master
Posted by rx...@apache.org.
Merge pull request #917 from pwendell/master
Document libgfortran dependency for MLBase
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/8c14f4b7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/8c14f4b7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/8c14f4b7
Branch: refs/heads/master
Commit: 8c14f4b72269093a62510dbb2f1e954c103ffcd4
Parents: c81377b cefee1e
Author: shivaram <sh...@gmail.com>
Authored: Mon Sep 9 22:07:58 2013 -0700
Committer: shivaram <sh...@gmail.com>
Committed: Mon Sep 9 22:07:58 2013 -0700
----------------------------------------------------------------------
docs/mllib-guide.md | 7 +++++++
1 file changed, 7 insertions(+)
----------------------------------------------------------------------
[34/50] git commit: Changed localProperties to use ThreadLocal (not
DynamicVariable).
Posted by rx...@apache.org.
Changed localProperties to use ThreadLocal (not DynamicVariable).
The fact that DynamicVariable uses an InheritableThreadLocal
can cause problems where the properties end up being shared
across threads in certain circumstances.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/93c42532
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/93c42532
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/93c42532
Branch: refs/heads/master
Commit: 93c425327560e386aa23b42387cea24eae228e79
Parents: 91a59e6
Author: Kay Ousterhout <ka...@gmail.com>
Authored: Wed Sep 11 12:12:20 2013 -0700
Committer: Kay Ousterhout <ka...@gmail.com>
Committed: Wed Sep 11 13:01:39 2013 -0700
----------------------------------------------------------------------
.../scala/org/apache/spark/SparkContext.scala | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/93c42532/core/src/main/scala/org/apache/spark/SparkContext.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 29407bc..72540c7 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -27,7 +27,6 @@ import scala.collection.generic.Growable
import scala.collection.JavaConversions._
import scala.collection.mutable.ArrayBuffer
import scala.collection.mutable.HashMap
-import scala.util.DynamicVariable
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
@@ -257,20 +256,20 @@ class SparkContext(
private[spark] var checkpointDir: Option[String] = None
// Thread Local variable that can be used by users to pass information down the stack
- private val localProperties = new DynamicVariable[Properties](null)
+ private val localProperties = new ThreadLocal[Properties]
def initLocalProperties() {
- localProperties.value = new Properties()
+ localProperties.set(new Properties())
}
def setLocalProperty(key: String, value: String) {
- if (localProperties.value == null) {
- localProperties.value = new Properties()
+ if (localProperties.get() == null) {
+ localProperties.set(new Properties())
}
if (value == null) {
- localProperties.value.remove(key)
+ localProperties.get.remove(key)
} else {
- localProperties.value.setProperty(key, value)
+ localProperties.get.setProperty(key, value)
}
}
@@ -724,7 +723,7 @@ class SparkContext(
logInfo("Starting job: " + callSite)
val start = System.nanoTime
val result = dagScheduler.runJob(rdd, func, partitions, callSite, allowLocal, resultHandler,
- localProperties.value)
+ localProperties.get)
logInfo("Job finished: " + callSite + ", took " + (System.nanoTime - start) / 1e9 + " s")
rdd.doCheckpoint()
result
@@ -807,7 +806,8 @@ class SparkContext(
val callSite = Utils.formatSparkCallSite
logInfo("Starting job: " + callSite)
val start = System.nanoTime
- val result = dagScheduler.runApproximateJob(rdd, func, evaluator, callSite, timeout, localProperties.value)
+ val result = dagScheduler.runApproximateJob(rdd, func, evaluator, callSite, timeout,
+ localProperties.get)
logInfo("Job finished: " + callSite + ", took " + (System.nanoTime - start) / 1e9 + " s")
result
}
[08/50] git commit: Adding Manen dependency
Posted by rx...@apache.org.
Adding Manen dependency
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/528fdbae
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/528fdbae
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/528fdbae
Branch: refs/heads/master
Commit: 528fdbae97e58797f62d97f27527092c0a5fed42
Parents: b4e382c
Author: Patrick Wendell <pw...@gmail.com>
Authored: Mon Sep 9 09:32:18 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Mon Sep 9 09:32:18 2013 -0700
----------------------------------------------------------------------
pom.xml | 5 +++++
1 file changed, 5 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/528fdbae/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index c561b09..c747822 100644
--- a/pom.xml
+++ b/pom.xml
@@ -287,6 +287,11 @@
<version>3.0.0</version>
</dependency>
<dependency>
+ <groupId>com.codahale.metrics</groupId>
+ <artifactId>metrics-ganglia</artifactId>
+ <version>3.0.0</version>
+ </dependency>
+ <dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-compiler</artifactId>
<version>${scala.version}</version>
[14/50] git commit: Print out more friendly error if listFiles() fails
Posted by rx...@apache.org.
Print out more friendly error if listFiles() fails
listFiles() could return null if the I/O fails, and this currently results in an ugly NPE which is hard to diagnose.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/27726079
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/27726079
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/27726079
Branch: refs/heads/master
Commit: 27726079e4e6931c071de77e91f991cb1b249d02
Parents: 1e15feb
Author: Evan Chan <ev...@ooyala.com>
Authored: Mon Sep 9 12:58:12 2013 -0700
Committer: Evan Chan <ev...@ooyala.com>
Committed: Mon Sep 9 12:58:12 2013 -0700
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/util/Utils.scala | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/27726079/core/src/main/scala/org/apache/spark/util/Utils.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala
index 468800b..a95eae9 100644
--- a/core/src/main/scala/org/apache/spark/util/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/util/Utils.scala
@@ -457,12 +457,18 @@ private[spark] object Utils extends Logging {
def newDaemonFixedThreadPool(nThreads: Int): ThreadPoolExecutor =
Executors.newFixedThreadPool(nThreads, daemonThreadFactory).asInstanceOf[ThreadPoolExecutor]
+ private def listFilesSafely(file: File): Seq[File] = {
+ val files = file.listFiles()
+ if (files == null) throw new IOException("Failed to list files for dir: " + file)
+ files
+ }
+
/**
* Delete a file or directory and its contents recursively.
*/
def deleteRecursively(file: File) {
if (file.isDirectory) {
- for (child <- file.listFiles()) {
+ for (child <- listFilesSafely(file)) {
deleteRecursively(child)
}
}
[25/50] git commit: fix run-example script
Posted by rx...@apache.org.
fix run-example script
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/56b94078
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/56b94078
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/56b94078
Branch: refs/heads/master
Commit: 56b94078481786a0b0bbcbe3971ee4ef0326d694
Parents: 2425eb8
Author: Haoyuan Li <ha...@cs.berkeley.edu>
Authored: Tue Sep 10 23:03:09 2013 -0700
Committer: Haoyuan Li <ha...@cs.berkeley.edu>
Committed: Tue Sep 10 23:03:09 2013 -0700
----------------------------------------------------------------------
run-example | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/56b94078/run-example
----------------------------------------------------------------------
diff --git a/run-example b/run-example
index 24d83ba..85557d7 100755
--- a/run-example
+++ b/run-example
@@ -39,14 +39,14 @@ fi
# to avoid the -sources and -doc packages that are built by publish-local.
EXAMPLES_DIR="$FWDIR"/examples
SPARK_EXAMPLES_JAR=""
-if [ -e "$EXAMPLES_DIR"/target/scala-$SCALA_VERSION/*assembly*[0-9T].jar ]; then
+if [ -e "$EXAMPLES_DIR"/target/scala-$SCALA_VERSION/*assembly*[0-9T]*.jar ]; then
# Use the JAR from the SBT build
- export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR"/target/scala-$SCALA_VERSION/*assembly*[0-9T].jar`
+ export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR"/target/scala-$SCALA_VERSION/*assembly*[0-9T]*.jar`
fi
-if [ -e "$EXAMPLES_DIR"/target/spark-examples*[0-9T].jar ]; then
+if [ -e "$EXAMPLES_DIR"/target/spark-examples*[0-9T]*.jar ]; then
# Use the JAR from the Maven build
# TODO: this also needs to become an assembly!
- export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR"/target/spark-examples*[0-9T].jar`
+ export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR"/target/spark-examples*[0-9T]*.jar`
fi
if [[ -z $SPARK_EXAMPLES_JAR ]]; then
echo "Failed to find Spark examples assembly in $FWDIR/examples/target" >&2
[35/50] git commit: Updated Spark on Mesos documentation.
Posted by rx...@apache.org.
Updated Spark on Mesos documentation.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/a0f0c1be
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/a0f0c1be
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/a0f0c1be
Branch: refs/heads/master
Commit: a0f0c1bed23d800c56e0b1637ef267ef94eb6103
Parents: 91a59e6
Author: Benjamin Hindman <be...@gmail.com>
Authored: Wed Sep 11 16:05:25 2013 -0700
Committer: Benjamin Hindman <be...@gmail.com>
Committed: Wed Sep 11 16:05:25 2013 -0700
----------------------------------------------------------------------
docs/running-on-mesos.md | 33 ++++++++++++++++-----------------
1 file changed, 16 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/a0f0c1be/docs/running-on-mesos.md
----------------------------------------------------------------------
diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md
index eee7a45..443350c 100644
--- a/docs/running-on-mesos.md
+++ b/docs/running-on-mesos.md
@@ -3,24 +3,23 @@ layout: global
title: Running Spark on Mesos
---
-Spark can run on private clusters managed by the [Apache Mesos](http://incubator.apache.org/mesos/) resource manager. Follow the steps below to install Mesos and Spark:
-
-1. Download and build Spark using the instructions [here](index.html).
-2. Download Mesos {{site.MESOS_VERSION}} from a [mirror](http://www.apache.org/dyn/closer.cgi/incubator/mesos/mesos-{{site.MESOS_VERSION}}/).
-3. Configure Mesos using the `configure` script, passing the location of your `JAVA_HOME` using `--with-java-home`. Mesos comes with "template" configure scripts for different platforms, such as `configure.macosx`, that you can run. See the README file in Mesos for other options. **Note:** If you want to run Mesos without installing it into the default paths on your system (e.g. if you don't have administrative privileges to install it), you should also pass the `--prefix` option to `configure` to tell it where to install. For example, pass `--prefix=/home/user/mesos`. By default the prefix is `/usr/local`.
-4. Build Mesos using `make`, and then install it using `make install`.
-5. Create a file called `spark-env.sh` in Spark's `conf` directory, by copying `conf/spark-env.sh.template`, and add the following lines it:
- * `export MESOS_NATIVE_LIBRARY=<path to libmesos.so>`. This path is usually `<prefix>/lib/libmesos.so` (where the prefix is `/usr/local` by default). Also, on Mac OS X, the library is called `libmesos.dylib` instead of `.so`.
-6. Copy Spark and Mesos to the _same_ paths on all the nodes in the cluster (or, for Mesos, `make install` on every node).
-7. Configure Mesos for deployment:
- * On your master node, edit `<prefix>/var/mesos/deploy/masters` to list your master and `<prefix>/var/mesos/deploy/slaves` to list the slaves, where `<prefix>` is the prefix where you installed Mesos (`/usr/local` by default).
- * On all nodes, edit `<prefix>/var/mesos/conf/mesos.conf` and add the line `master=HOST:5050`, where HOST is your master node.
- * Run `<prefix>/sbin/mesos-start-cluster.sh` on your master to start Mesos. If all goes well, you should see Mesos's web UI on port 8080 of the master machine.
- * See Mesos's README file for more information on deploying it.
-8. To run a Spark application against the cluster, when you create your `SparkContext`, pass the string `mesos://HOST:5050` as the first parameter, where `HOST` is the machine running your Mesos master. In addition, pass the location of Spark on your nodes as the third parameter, and a list of JAR files containing your JAR's code as the fourth (these will automatically get copied to the workers). For example:
+Spark can run on clusters managed by [Apache Mesos](http://mesos.apache.org/). Follow the steps below to install Mesos and Spark:
+
+1. Download and build Spark using the instructions [here](index.html). **Note:** Don't forget to consider what version of HDFS you might want to use!
+2. Download, build, install, and start Mesos {{site.MESOS_VERSION}} on your cluster. You can download the Mesos distribution from a [mirror](http://www.apache.org/dyn/closer.cgi/mesos/{{site.MESOS_VERSION}}/). See the Mesos [Getting Started](http://mesos.apache.org/gettingstarted) page for more information. **Note:** If you want to run Mesos without installing it into the default paths on your system (e.g., if you don't have administrative privileges to install it), you should also pass the `--prefix` option to `configure` to tell it where to install. For example, pass `--prefix=/home/user/mesos`. By default the prefix is `/usr/local`.
+3. Create a Spark "distribution" using `make-distribution.sh`.
+4. Rename the `dist` directory created from `make-distribution.sh` to `spark-{{site.SPARK_VERSION}}`.
+5. Create a `tar` archive: `tar czf spark-{{site.SPARK_VERSION}}.tar.gz spark-{{site.SPARK_VERSION}}`
+6. Upload this archive to your HDFS or another place accessible from Mesos via `http://`, e.g., [Amazon Simple Storage Service](http://aws.amazon.com/s3): `hadoop fs -put spark-{{site.SPARK_VERSION}}.tar.gz /path/to/spark-{{site.SPARK_VERSION}}.tar.gz`
+7. Create a file called `spark-env.sh` in Spark's `conf` directory, by copying `conf/spark-env.sh.template`, and add the following lines to it:
+ * `export MESOS_NATIVE_LIBRARY=<path to libmesos.so>`. This path is usually `<prefix>/lib/libmesos.so` (where the prefix is `/usr/local` by default, see above). Also, on Mac OS X, the library is called `libmesos.dylib` instead of `libmesos.so`.
+ * `export SPARK_EXECUTOR_URI=<path to spark-{{site.SPARK_VERSION}}.tar.gz uploaded above>`.
+ * `export MASTER=mesos://HOST:PORT` where HOST:PORT is the host and port (default: 5050) of your Mesos master (or `zk://...` if using Mesos with ZooKeeper).
+8. To run a Spark application against the cluster, when you create your `SparkContext`, pass the string `mesos://HOST:PORT` as the first parameter. In addition, you'll need to set the `spark.executor.uri` property. For example
{% highlight scala %}
-new SparkContext("mesos://HOST:5050", "My App Name", "/home/user/spark", List("my-app.jar"))
+System.setProperty("spark.executor.uri", "<path to spark-{{site.SPARK_VERSION}}.tar.gz uploaded above>")
+val sc = new SparkContext("mesos://HOST:5050", "App Name", ...)
{% endhighlight %}
If you want to run Spark on Amazon EC2, you can use the Spark [EC2 launch scripts](ec2-scripts.html), which provide an easy way to launch a cluster with Mesos, Spark, and HDFS pre-configured. This will get you a cluster in about five minutes without any configuration on your part.
@@ -52,6 +51,6 @@ Again, this must be done *before* initializing a SparkContext.
You can run Spark and Mesos alongside your existing Hadoop cluster by just launching them as a separate service on the machines. To access Hadoop data from Spark, just use a hdfs:// URL (typically `hdfs://<namenode>:9000/path`, but you can find the right URL on your Hadoop Namenode's web UI).
-In addition, it is possible to also run Hadoop MapReduce on Mesos, to get better resource isolation and sharing between the two. In this case, Mesos will act as a unified scheduler that assigns cores to either Hadoop or Spark, as opposed to having them share resources via the Linux scheduler on each node. Please refer to the Mesos wiki page on [Running Hadoop on Mesos](https://github.com/mesos/mesos/wiki/Running-Hadoop-on-Mesos).
+In addition, it is possible to also run Hadoop MapReduce on Mesos, to get better resource isolation and sharing between the two. In this case, Mesos will act as a unified scheduler that assigns cores to either Hadoop or Spark, as opposed to having them share resources via the Linux scheduler on each node. Please refer to [Hadoop on Mesos](https://github.com/mesos/hadoop).
In either case, HDFS runs separately from Hadoop MapReduce, without going through Mesos.
[33/50] git commit: Merge pull request #919 from mateiz/jets3t
Posted by rx...@apache.org.
Merge pull request #919 from mateiz/jets3t
Add explicit jets3t dependency, which is excluded in hadoop-client
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/91a59e6b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/91a59e6b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/91a59e6b
Branch: refs/heads/master
Commit: 91a59e6b104523112d52d05a90a8fe6bc988161a
Parents: b9128d3 f117dc6
Author: Patrick Wendell <pw...@gmail.com>
Authored: Wed Sep 11 10:21:48 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Wed Sep 11 10:21:48 2013 -0700
----------------------------------------------------------------------
core/pom.xml | 5 +++++
pom.xml | 5 +++++
project/SparkBuild.scala | 1 +
3 files changed, 11 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/91a59e6b/project/SparkBuild.scala
----------------------------------------------------------------------
[29/50] git commit: SPARK-894 - Not all WebUI fields delivered VIA
JSON
Posted by rx...@apache.org.
SPARK-894 - Not all WebUI fields delivered VIA JSON
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/5dd875c5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/5dd875c5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/5dd875c5
Branch: refs/heads/master
Commit: 5dd875c5b569554262741f012e2e3857401c1677
Parents: 8c14f4b
Author: David McCauley <da...@wandisco.com>
Authored: Wed Sep 11 10:46:37 2013 +0100
Committer: David McCauley <da...@wandisco.com>
Committed: Wed Sep 11 10:46:37 2013 +0100
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/5dd875c5/core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala b/core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala
index a6be8ef..87a7034 100644
--- a/core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala
@@ -44,7 +44,9 @@ private[spark] object JsonProtocol {
("cores" -> obj.desc.maxCores) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerSlave) ~
- ("submitdate" -> obj.submitDate.toString)
+ ("submitdate" -> obj.submitDate.toString) ~
+ ("state" -> obj.state.toString) ~
+ ("duration" -> obj.duration)
}
def writeApplicationDescription(obj: ApplicationDescription) = {
[26/50] git commit: Merge pull request #921 from pwendell/master
Posted by rx...@apache.org.
Merge pull request #921 from pwendell/master
Fix HDFS access bug with assembly build.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/d40f1403
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/d40f1403
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/d40f1403
Branch: refs/heads/master
Commit: d40f1403f38a406676f6999c7eb889f68c9dfe1d
Parents: 2425eb8 0c1985b
Author: Patrick Wendell <pw...@gmail.com>
Authored: Tue Sep 10 23:05:29 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Tue Sep 10 23:05:29 2013 -0700
----------------------------------------------------------------------
assembly/pom.xml | 6 ++++++
project/SparkBuild.scala | 1 +
2 files changed, 7 insertions(+)
----------------------------------------------------------------------
[42/50] git commit: Explain yarn.version in Maven build docs
Posted by rx...@apache.org.
Explain yarn.version in Maven build docs
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/362ea0c0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/362ea0c0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/362ea0c0
Branch: refs/heads/master
Commit: 362ea0c0517ca0caaf5d3c12f8187198be59ca37
Parents: e9eba8c
Author: Patrick Wendell <pw...@gmail.com>
Authored: Sun Sep 15 12:36:58 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Sun Sep 15 12:40:49 2013 -0700
----------------------------------------------------------------------
docs/building-with-maven.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/362ea0c0/docs/building-with-maven.md
----------------------------------------------------------------------
diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md
index 7ecb601..3065427 100644
--- a/docs/building-with-maven.md
+++ b/docs/building-with-maven.md
@@ -37,13 +37,13 @@ For Apache Hadoop versions 1.x, Cloudera CDH MRv1, and other Hadoop versions wit
# Cloudera CDH 4.2.0 with MapReduce v1
$ mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package
-For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you should also enable the "hadoop2-yarn" profile:
+For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you should enable the "hadoop2-yarn" profile and set the "yarn.version" property:
# Apache Hadoop 2.0.5-alpha
- $ mvn -Phadoop2-yarn -Dhadoop.version=2.0.5-alpha -DskipTests clean package
+ $ mvn -Phadoop2-yarn -Dhadoop.version=2.0.5-alpha -Pyarn.version=2.0.5-alpha -DskipTests clean package
# Cloudera CDH 4.2.0 with MapReduce v2
- $ mvn -Phadoop2-yarn -Dhadoop.version=2.0.0-cdh4.2.0 -DskipTests clean package
+ $ mvn -Phadoop2-yarn -Dhadoop.version=2.0.0-cdh4.2.0 -Pyarn.version=2.0.0-chd4.2.0 -DskipTests clean package
## Spark Tests in Maven ##
[13/50] git commit: Merge pull request #912 from tgravescs/ganglia-pom
Posted by rx...@apache.org.
Merge pull request #912 from tgravescs/ganglia-pom
Add metrics-ganglia to core pom file
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/084fc369
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/084fc369
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/084fc369
Branch: refs/heads/master
Commit: 084fc369614766ce9ca3285c784afceddd111d25
Parents: 0456384 2186d93
Author: Matei Zaharia <ma...@gmail.com>
Authored: Mon Sep 9 12:01:35 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Mon Sep 9 12:01:35 2013 -0700
----------------------------------------------------------------------
core/pom.xml | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
[27/50] git commit: better expression
Posted by rx...@apache.org.
better expression
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/45d964b6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/45d964b6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/45d964b6
Branch: refs/heads/master
Commit: 45d964b60666a794ce8d6cc4e8451a3447327499
Parents: 56b9407
Author: Haoyuan Li <ha...@cs.berkeley.edu>
Authored: Tue Sep 10 23:18:22 2013 -0700
Committer: Haoyuan Li <ha...@cs.berkeley.edu>
Committed: Tue Sep 10 23:18:22 2013 -0700
----------------------------------------------------------------------
run-example | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/45d964b6/run-example
----------------------------------------------------------------------
diff --git a/run-example b/run-example
index 85557d7..08ec717 100755
--- a/run-example
+++ b/run-example
@@ -39,14 +39,14 @@ fi
# to avoid the -sources and -doc packages that are built by publish-local.
EXAMPLES_DIR="$FWDIR"/examples
SPARK_EXAMPLES_JAR=""
-if [ -e "$EXAMPLES_DIR"/target/scala-$SCALA_VERSION/*assembly*[0-9T]*.jar ]; then
+if [ -e "$EXAMPLES_DIR"/target/scala-$SCALA_VERSION/*assembly*[0-9Tg].jar ]; then
# Use the JAR from the SBT build
- export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR"/target/scala-$SCALA_VERSION/*assembly*[0-9T]*.jar`
+ export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR"/target/scala-$SCALA_VERSION/*assembly*[0-9Tg].jar`
fi
-if [ -e "$EXAMPLES_DIR"/target/spark-examples*[0-9T]*.jar ]; then
+if [ -e "$EXAMPLES_DIR"/target/spark-examples*[0-9Tg].jar ]; then
# Use the JAR from the Maven build
# TODO: this also needs to become an assembly!
- export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR"/target/spark-examples*[0-9T]*.jar`
+ export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR"/target/spark-examples*[0-9Tg].jar`
fi
if [[ -z $SPARK_EXAMPLES_JAR ]]; then
echo "Failed to find Spark examples assembly in $FWDIR/examples/target" >&2
[19/50] git commit: Merge pull request #915 from ooyala/master
Posted by rx...@apache.org.
Merge pull request #915 from ooyala/master
Get rid of / improve ugly NPE when Utils.deleteRecursively() fails
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/c81377b9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/c81377b9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/c81377b9
Branch: refs/heads/master
Commit: c81377b9eda431f9298045d613d0e46301b08c63
Parents: 61d2a01 fdb8b0e
Author: Matei Zaharia <ma...@gmail.com>
Authored: Mon Sep 9 20:16:19 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Mon Sep 9 20:16:19 2013 -0700
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/util/Utils.scala | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/c81377b9/core/src/main/scala/org/apache/spark/util/Utils.scala
----------------------------------------------------------------------
[17/50] git commit: Style fix: put body of if within curly braces
Posted by rx...@apache.org.
Style fix: put body of if within curly braces
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/fdb8b0ee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/fdb8b0ee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/fdb8b0ee
Branch: refs/heads/master
Commit: fdb8b0eec3e423e601de8c79658c4644717f4e05
Parents: 2772607
Author: Evan Chan <ev...@ooyala.com>
Authored: Mon Sep 9 14:29:32 2013 -0700
Committer: Evan Chan <ev...@ooyala.com>
Committed: Mon Sep 9 14:29:32 2013 -0700
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/util/Utils.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/fdb8b0ee/core/src/main/scala/org/apache/spark/util/Utils.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala
index a95eae9..a89e245 100644
--- a/core/src/main/scala/org/apache/spark/util/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/util/Utils.scala
@@ -459,7 +459,9 @@ private[spark] object Utils extends Logging {
private def listFilesSafely(file: File): Seq[File] = {
val files = file.listFiles()
- if (files == null) throw new IOException("Failed to list files for dir: " + file)
+ if (files == null) {
+ throw new IOException("Failed to list files for dir: " + file)
+ }
files
}
[44/50] git commit: Merge pull request #931 from pwendell/yarn-docs
Posted by rx...@apache.org.
Merge pull request #931 from pwendell/yarn-docs
Explain yarn.version in Maven build docs
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/9fb0b9d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/9fb0b9d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/9fb0b9d7
Branch: refs/heads/master
Commit: 9fb0b9d77fd383ad2cdd600edcc77c7b43c5e1c2
Parents: c4c1db2 362ea0c
Author: Patrick Wendell <pw...@gmail.com>
Authored: Sun Sep 15 13:02:53 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Sun Sep 15 13:02:53 2013 -0700
----------------------------------------------------------------------
docs/building-with-maven.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
[40/50] git commit: Use different Hadoop version for YARN artifacts.
Posted by rx...@apache.org.
Use different Hadoop version for YARN artifacts.
This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/e9eba8c3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/e9eba8c3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/e9eba8c3
Branch: refs/heads/master
Commit: e9eba8c3ce320364f20dbbb19e42df07f936214f
Parents: 45ec1cc
Author: Patrick Wendell <pw...@gmail.com>
Authored: Fri Sep 13 14:06:45 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Fri Sep 13 15:34:57 2013 -0700
----------------------------------------------------------------------
pom.xml | 11 ++++++-----
yarn/pom.xml | 5 +++++
2 files changed, 11 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/e9eba8c3/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 7bacbb1..2da016a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -82,6 +82,7 @@
<slf4j.version>1.7.2</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<hadoop.version>1.0.4</hadoop.version>
+ <yarn.version>0.23.7</yarn.version>
<hbase.version>0.94.6</hbase.version>
<PermGen>64m</PermGen>
@@ -382,7 +383,7 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-api</artifactId>
- <version>${hadoop.version}</version>
+ <version>${yarn.version}</version>
<exclusions>
<exclusion>
<groupId>asm</groupId>
@@ -413,7 +414,7 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-common</artifactId>
- <version>${hadoop.version}</version>
+ <version>${yarn.version}</version>
<exclusions>
<exclusion>
<groupId>asm</groupId>
@@ -444,7 +445,7 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
- <version>${hadoop.version}</version>
+ <version>${yarn.version}</version>
<exclusions>
<exclusion>
<groupId>asm</groupId>
@@ -700,8 +701,8 @@
<properties>
<hadoop.major.version>2</hadoop.major.version>
<!-- 0.23.* is same as 2.0.* - except hardened to run production jobs -->
- <!-- <hadoop.version>0.23.7</hadoop.version> -->
- <hadoop.version>2.0.5-alpha</hadoop.version>
+ <hadoop.version>0.23.7</hadoop.version>
+ <!--<hadoop.version>2.0.5-alpha</hadoop.version> -->
</properties>
<modules>
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/e9eba8c3/yarn/pom.xml
----------------------------------------------------------------------
diff --git a/yarn/pom.xml b/yarn/pom.xml
index 27b2002..21b650d 100644
--- a/yarn/pom.xml
+++ b/yarn/pom.xml
@@ -49,6 +49,11 @@
<artifactId>hadoop-yarn-client</artifactId>
</dependency>
<dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>${yarn.version}</version>
+ </dependency>
+ <dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
[45/50] git commit: Merge pull request #932 from
pwendell/mesos-version
Posted by rx...@apache.org.
Merge pull request #932 from pwendell/mesos-version
Bumping Mesos version to 0.13.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/dbd2c4fd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/dbd2c4fd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/dbd2c4fd
Branch: refs/heads/master
Commit: dbd2c4fd948e09a1a270b576ad745d1065006acc
Parents: 9fb0b9d c856860
Author: Patrick Wendell <pw...@gmail.com>
Authored: Sun Sep 15 13:20:41 2013 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Sun Sep 15 13:20:41 2013 -0700
----------------------------------------------------------------------
docs/_config.yml | 2 +-
pom.xml | 2 +-
project/SparkBuild.scala | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
[18/50] git commit: Merge pull request #916 from mateiz/mkdist-fix
Posted by rx...@apache.org.
Merge pull request #916 from mateiz/mkdist-fix
Fix copy issue in https://github.com/mesos/spark/pull/899
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/61d2a010
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/61d2a010
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/61d2a010
Branch: refs/heads/master
Commit: 61d2a010e1b2d25616c378185606e6ba7c2b916b
Parents: a85758c f5a8afa
Author: Matei Zaharia <ma...@gmail.com>
Authored: Mon Sep 9 18:21:01 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Mon Sep 9 18:21:01 2013 -0700
----------------------------------------------------------------------
make-distribution.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
[03/50] git commit: Merge pull request #905 from mateiz/docs2
Posted by rx...@apache.org.
Merge pull request #905 from mateiz/docs2
Job scheduling and cluster mode docs
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/7d3204b0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/7d3204b0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/7d3204b0
Branch: refs/heads/master
Commit: 7d3204b0568e802ba15eb482c1af4c37ebf07022
Parents: f1f8371 b458854
Author: Matei Zaharia <ma...@gmail.com>
Authored: Sun Sep 8 21:39:12 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Sun Sep 8 21:39:12 2013 -0700
----------------------------------------------------------------------
conf/fairscheduler.xml.template | 18 +-
.../scala/org/apache/spark/SparkContext.scala | 5 +-
.../scheduler/cluster/ClusterScheduler.scala | 2 +-
.../scheduler/cluster/SchedulableBuilder.scala | 6 +-
.../spark/scheduler/local/LocalScheduler.scala | 2 +-
.../apache/spark/ui/UIWorkloadGenerator.scala | 4 +-
.../spark/ui/jobs/JobProgressListener.scala | 2 +-
.../cluster/ClusterSchedulerSuite.scala | 8 +-
.../scheduler/local/LocalSchedulerSuite.scala | 8 +-
docs/_layouts/global.html | 7 +-
docs/cluster-overview.md | 117 +++++++++++++
docs/configuration.md | 32 ++--
docs/contributing-to-spark.md | 24 +--
docs/ec2-scripts.md | 8 +-
docs/hadoop-third-party-distributions.md | 62 +++++--
docs/img/cluster-overview.png | Bin 0 -> 28011 bytes
docs/img/cluster-overview.pptx | Bin 0 -> 51771 bytes
docs/img/spark-logo-hd.png | Bin 0 -> 13512 bytes
docs/index.md | 31 ++--
docs/job-scheduling.md | 168 +++++++++++++++++++
docs/monitoring.md | 30 +++-
docs/python-programming-guide.md | 12 +-
docs/quick-start.md | 70 ++++----
docs/running-on-mesos.md | 19 +--
docs/scala-programming-guide.md | 4 +-
docs/spark-standalone.md | 26 +--
26 files changed, 499 insertions(+), 166 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/7d3204b0/core/src/main/scala/org/apache/spark/SparkContext.scala
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/7d3204b0/docs/monitoring.md
----------------------------------------------------------------------
[28/50] git commit: Merge pull request #923 from haoyuan/master
Posted by rx...@apache.org.
Merge pull request #923 from haoyuan/master
fix run-example script
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/8432f27f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/8432f27f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/8432f27f
Branch: refs/heads/master
Commit: 8432f27fbed968995a2aa223907e9f16d99e536f
Parents: d40f140 45d964b
Author: Matei Zaharia <ma...@gmail.com>
Authored: Tue Sep 10 23:19:53 2013 -0700
Committer: Matei Zaharia <ma...@gmail.com>
Committed: Tue Sep 10 23:19:53 2013 -0700
----------------------------------------------------------------------
run-example | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------