You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/01/23 00:45:12 UTC
[1/3] git commit: Depend on Commons Math explicitly instead of
accidentally getting it from Hadoop (which stops working in 2.2.x) and also
use the newer commons-math3
Updated Branches:
refs/heads/master a1238bb5f -> 3184facdc
Depend on Commons Math explicitly instead of accidentally getting it from Hadoop (which stops working in 2.2.x) and also use the newer commons-math3
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/fd0c5b8c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/fd0c5b8c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/fd0c5b8c
Branch: refs/heads/master
Commit: fd0c5b8c577204f98831ece860e0b3e153f90d21
Parents: 749f842
Author: Sean Owen <so...@cloudera.com>
Authored: Wed Jan 22 22:25:49 2014 +0000
Committer: Sean Owen <so...@cloudera.com>
Committed: Wed Jan 22 22:25:49 2014 +0000
----------------------------------------------------------------------
graphx/pom.xml | 5 +++++
.../main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala | 2 +-
2 files changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/fd0c5b8c/graphx/pom.xml
----------------------------------------------------------------------
diff --git a/graphx/pom.xml b/graphx/pom.xml
index 4eca474..d97dbb8 100644
--- a/graphx/pom.xml
+++ b/graphx/pom.xml
@@ -38,6 +38,11 @@
<version>${project.version}</version>
</dependency>
<dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-math3</artifactId>
+ <version>3.2</version>
+ </dependency>
+ <dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
</dependency>
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/fd0c5b8c/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala
----------------------------------------------------------------------
diff --git a/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala b/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala
index c327ce7..79280f8 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala
@@ -18,7 +18,7 @@
package org.apache.spark.graphx.lib
import scala.util.Random
-import org.apache.commons.math.linear._
+import org.apache.commons.math3.linear._
import org.apache.spark.rdd._
import org.apache.spark.graphx._
[2/3] git commit: Also add graphx commons-math3 dependeny in sbt build
Posted by pw...@apache.org.
Also add graphx commons-math3 dependeny in sbt build
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/4476398f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/4476398f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/4476398f
Branch: refs/heads/master
Commit: 4476398f7dae1ea87d2b99cb60306cb3c221992f
Parents: fd0c5b8
Author: Sean Owen <so...@cloudera.com>
Authored: Wed Jan 22 22:40:41 2014 +0000
Committer: Sean Owen <so...@cloudera.com>
Committed: Wed Jan 22 22:40:41 2014 +0000
----------------------------------------------------------------------
project/SparkBuild.scala | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/4476398f/project/SparkBuild.scala
----------------------------------------------------------------------
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 151b1e7..76e3973 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -316,7 +316,10 @@ object SparkBuild extends Build {
) ++ assemblySettings ++ extraAssemblySettings
def graphxSettings = sharedSettings ++ Seq(
- name := "spark-graphx"
+ name := "spark-graphx",
+ libraryDependencies ++= Seq(
+ "org.apache.commons" % "commons-math3" % "3.2"
+ )
)
def bagelSettings = sharedSettings ++ Seq(
[3/3] git commit: Merge pull request #495 from
srowen/GraphXCommonsMathDependency
Posted by pw...@apache.org.
Merge pull request #495 from srowen/GraphXCommonsMathDependency
Fix graphx Commons Math dependency
`graphx` depends on Commons Math (2.x) in `SVDPlusPlus.scala`. However the module doesn't declare this dependency. It happens to work because it is included by Hadoop artifacts. But, I can tell you this isn't true as of a month or so ago. Building versus recent Hadoop would fail. (That's how we noticed.)
The simple fix is to declare the dependency, as it should be. But it's also worth noting that `commons-math` is the old-ish 2.x line, while `commons-math3` is where newer 3.x releases are. Drop-in replacement, but different artifact and package name. Changing this only usage to `commons-math3` works, tests pass, and isn't surprising that it does, so is probably also worth changing. (A comment in some test code also references `commons-math3`, FWIW.)
It does raise another question though: `mllib` looks like it uses the `jblas` `DoubleMatrix` for general purpose vector/matrix stuff. Should `graphx` really use Commons Math for this? Beyond the tiny scope here but worth asking.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/3184facd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/3184facd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/3184facd
Branch: refs/heads/master
Commit: 3184facdc5b1e9ded89133f9b1e4985c9ac78c55
Parents: a1238bb 4476398
Author: Patrick Wendell <pw...@gmail.com>
Authored: Wed Jan 22 15:45:04 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Wed Jan 22 15:45:04 2014 -0800
----------------------------------------------------------------------
graphx/pom.xml | 5 +++++
.../main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala | 2 +-
project/SparkBuild.scala | 5 ++++-
3 files changed, 10 insertions(+), 2 deletions(-)
----------------------------------------------------------------------