You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/01/23 00:45:22 UTC
git commit: Merge pull request #495 from
srowen/GraphXCommonsMathDependency
Updated Branches:
refs/heads/branch-0.9 dc5857a36 -> 828f7b46e
Merge pull request #495 from srowen/GraphXCommonsMathDependency
Fix graphx Commons Math dependency
`graphx` depends on Commons Math (2.x) in `SVDPlusPlus.scala`. However the module doesn't declare this dependency. It happens to work because it is included by Hadoop artifacts. But, I can tell you this isn't true as of a month or so ago. Building versus recent Hadoop would fail. (That's how we noticed.)
The simple fix is to declare the dependency, as it should be. But it's also worth noting that `commons-math` is the old-ish 2.x line, while `commons-math3` is where newer 3.x releases are. Drop-in replacement, but different artifact and package name. Changing this only usage to `commons-math3` works, tests pass, and isn't surprising that it does, so is probably also worth changing. (A comment in some test code also references `commons-math3`, FWIW.)
It does raise another question though: `mllib` looks like it uses the `jblas` `DoubleMatrix` for general purpose vector/matrix stuff. Should `graphx` really use Commons Math for this? Beyond the tiny scope here but worth asking.
(cherry picked from commit 3184facdc5b1e9ded89133f9b1e4985c9ac78c55)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/828f7b46
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/828f7b46
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/828f7b46
Branch: refs/heads/branch-0.9
Commit: 828f7b46ea7c6f7cf7f72bc03c61eeb9b929ac07
Parents: dc5857a
Author: Patrick Wendell <pw...@gmail.com>
Authored: Wed Jan 22 15:45:04 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Wed Jan 22 15:45:18 2014 -0800
----------------------------------------------------------------------
graphx/pom.xml | 5 +++++
.../main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala | 2 +-
project/SparkBuild.scala | 5 ++++-
3 files changed, 10 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/828f7b46/graphx/pom.xml
----------------------------------------------------------------------
diff --git a/graphx/pom.xml b/graphx/pom.xml
index 67a8e01..9073cf7 100644
--- a/graphx/pom.xml
+++ b/graphx/pom.xml
@@ -38,6 +38,11 @@
<version>${project.version}</version>
</dependency>
<dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-math3</artifactId>
+ <version>3.2</version>
+ </dependency>
+ <dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
</dependency>
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/828f7b46/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala
----------------------------------------------------------------------
diff --git a/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala b/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala
index c327ce7..79280f8 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala
@@ -18,7 +18,7 @@
package org.apache.spark.graphx.lib
import scala.util.Random
-import org.apache.commons.math.linear._
+import org.apache.commons.math3.linear._
import org.apache.spark.rdd._
import org.apache.spark.graphx._
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/828f7b46/project/SparkBuild.scala
----------------------------------------------------------------------
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 075e912..b891ffa 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -316,7 +316,10 @@ object SparkBuild extends Build {
) ++ assemblySettings ++ extraAssemblySettings
def graphxSettings = sharedSettings ++ Seq(
- name := "spark-graphx"
+ name := "spark-graphx",
+ libraryDependencies ++= Seq(
+ "org.apache.commons" % "commons-math3" % "3.2"
+ )
)
def bagelSettings = sharedSettings ++ Seq(