You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by miccagiann <gi...@git.apache.org> on 2014/07/06 17:24:48 UTC
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
GitHub user miccagiann opened a pull request:
https://github.com/apache/spark/pull/1311
Java examples included in 'mllib-linear-methods.md' file.
Standalone application examples are added to 'mllib-linear-methods.md' file written in Java.
This commit is related to the issue [Add full Java Examples in MLlib docs](https://issues.apache.org/jira/browse/SPARK-1945).
Also I changed the name of the sigmoid function from 'logit' to 'f'. This is because the logit function
is the inverse of sigmoid.
Thanks,
Michael
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/miccagiann/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1311.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1311
----
commit 15f0eb418e1027853a3706e3f44fd268ac37a3e8
Author: Michael Giannakopoulos <mi...@gmail.com>
Date: 2014-07-06T15:11:13Z
Java examples included in 'mllib-linear-methods.md' file.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14696323
--- Diff: docs/mllib-collaborative-filtering.md ---
@@ -99,7 +99,83 @@ val model = ALS.trainImplicit(ratings, rank, numIterations, alpha)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.recommendation.ALS;
+import org.apache.spark.mllib.recommendation.Rating;
+import org.apache.spark.mllib.recommendation.MatrixFactorizationModel;
+
+import scala.Tuple2;
+import scala.Product;
+
+public class CollaborativeFiltering {
+ public static void main( String[] args ) {
+ SparkConf conf = new SparkConf().setAppName("Collaborative Filtering Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse the data
+ String path = "{SPARK_HOME}/mllib/data/als/test.data";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<Rating> ratings = data.map(
+ new Function<String, Rating>() {
+ public Rating call( String s ) {
+ String[] sarray = s.split(",");
+ return new Rating(Integer.parseInt(sarray[0]), Integer.parseInt(sarray[1]), Double.parseDouble(sarray[2]));
--- End diff --
The line width should be at most 100.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14863037
--- Diff: docs/mllib-optimization.md ---
@@ -263,7 +267,110 @@ println("Loss of each step in training process")
loss.foreach(println)
println("Area under ROC = " + auROC)
{% endhighlight %}
-
+</div>
+
+<div data-lang="java" markdown="1">
+{% highlight java %}
+import java.util.Random;
+import java.util.Arrays;
+
+import scala.Product2;
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.optimization.*;
+import org.apache.spark.mllib.classification.LogisticRegressionModel;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+public class LBFGSExample {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("L-BFGS Example");
+ SparkContext sc = new SparkContext(conf);
+ String path = "{SPARK_HOME}/mllib/data/sample_libsvm_data.txt";
+ JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
+ int numFeatures = data.take(1).get(0).features().size();
+
+ // Split initial RDD into two... [60% training data, 40% testing data].
+ JavaRDD<LabeledPoint> trainingInit = data.filter(
+ new Function<LabeledPoint, Boolean>() {
+ public final Random random = new Random(11L);
+ public Boolean call(LabeledPoint p) {
+ if (random.nextDouble() <= 0.6)
+ return true;
+ else
+ return false;
+ }
+ }
+ );
+ JavaRDD<LabeledPoint> test = data.subtract(trainingInit);
+
+ // Append 1 into the training data as intercept.
+ JavaRDD<Tuple2<Object, Vector>> training = data.map(
+ new Function<LabeledPoint, Tuple2<Object, Vector>>() {
+ public Tuple2<Object, Vector> call(LabeledPoint p) {
+ return new Tuple2<Object, Vector>(p.label(), MLUtils.appendBias(p.features()));
+ }
+ });
+ training.cache();
+
+ // Run training algorithm to build the model.
+ int numCorrections = 10;
+ double convergenceTol = 1e-4;
+ int maxNumIterations = 20;
+ double regParam = 0.1;
+ Vector initialWeightsWithIntercept = Vectors.dense(new double[numFeatures + 1]);
+
+ Tuple2<Vector, double[]> result = LBFGS.runLBFGS(
+ JavaRDD.toRDD(training),
+ new LogisticGradient(),
+ new SquaredL2Updater(),
+ numCorrections,
+ convergenceTol,
+ maxNumIterations,
+ regParam,
+ initialWeightsWithIntercept);
+ Vector weightsWithIntercept = (Vector) result.productElement(0);
--- End diff --
use `._1()` (and `._2()` below)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862918
--- Diff: docs/mllib-dimensionality-reduction.md ---
@@ -91,4 +138,51 @@ val pc: Matrix = mat.computePrincipalComponents(10) // Principal components are
val projected: RowMatrix = mat.multiply(pc)
{% endhighlight %}
</div>
+
+<div data-lang="java" markdown="1">
+
+The following code demonstrates how to compute principal components on a tall-and-skinny `RowMatrix`
+and use them to project the vectors into a low-dimensional space.
+The number of columns should be small, e.g, less than 1000.
+
+{% highlight java %}
+import java.util.LinkedList;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.rdd.RDD;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.distributed.RowMatrix;
+
+public class PCA {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("PCA Example");
+ SparkContext sc = new SparkContext(conf);
+
+ double[][] array = ...
+ LinkedList<Vector> rowsList = new LinkedList<Vector>();
+ for (int i = 0; i < array.length; i++) {
+ Vector currentRow = Vectors.dense(array[i]);
+ rowsList.add(currentRow);
+ }
+ JavaRDD<Vector> rows = JavaSparkContext.fromSparkContext(sc).parallelize(rowsList);
+
+ // Create a RowMatrix from JavaRDD<Vector>.
+ RowMatrix mat = new RowMatrix(JavaRDD.toRDD(rows));
--- End diff --
ditto.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14863059
--- Diff: docs/mllib-linear-methods.md ---
@@ -242,7 +242,96 @@ Similarly, you can use replace `SVMWithSGD` by
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import java.util.Random;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.SparkContext;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.classification.*;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+public class SVMClassifier {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("SVM Classifier Example");
+ SparkContext sc = new SparkContext(conf);
+ String path = "{SPARK_HOME}/mllib/data/sample_libsvm_data.txt";
+ JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
+
+ // Split initial RDD into two... [60% training data, 40% testing data].
+ JavaRDD<LabeledPoint> training = data.filter(
+ new Function<LabeledPoint, Boolean>() {
+ public final Random random = new Random(11L);
+ public Boolean call(LabeledPoint p) {
+ if (random.nextDouble() <= 0.6)
+ return true;
+ else
+ return false;
+ }
+ }
+ );
+ training.cache();
+ JavaRDD<LabeledPoint> test = data.subtract(training);
+
+ // Run training algorithm to build the model.
+ int numIterations = 100;
+ final SVMModel model = SVMWithSGD.train(JavaRDD.toRDD(training), numIterations);
+
+ // Clear the default threshold.
+ model.clearThreshold();
+
+ // Compute raw scores on the test set.
+ JavaRDD<Tuple2<Object, Object>> scoreAndLabels = test.map(
+ new Function<LabeledPoint, Tuple2<Object, Object>>() {
+ public final SVMModel m = model;
+ public Tuple2<Object, Object> call(LabeledPoint p) {
+ Double score = m.predict(p.features());
+ return new Tuple2<Object, Object>(score, p.label());
+ }
+ }
+ );
+
+ // Get evaluation metrics.
+ BinaryClassificationMetrics metrics =
+ new BinaryClassificationMetrics(JavaRDD.toRDD(scoreAndLabels));
+ double auROC = metrics.areaUnderROC();
+
+ System.out.println("Area under ROC = " + auROC);
+ }
+}
+{% endhighlight %}
+
+The `SVMWithSGD.train()` method by default performs L2 regularization with the
+regularization parameter set to 1.0. If we want to configure this algorithm, we
+can customize `SVMWithSGD` further by creating a new object directly and
+calling setter methods. All other MLlib algorithms support customization in
+this way as well. For example, the following code produces an L1 regularized
+variant of SVMs with regularization parameter set to 0.1, and runs the training
+algorithm for 200 iterations.
+
+{% highlight java %}
+import org.apache.spark.mllib.optimization.L1Updater;
+
+SVMWithSGD svmAlg = new SVMWithSGD();
+svmAlg.optimizer().setNumIterations(200);
+svmAlg.optimizer().setRegParam(0.1);
--- End diff --
use the builder pattern:
~~~
svmAlg.optimier()
.setNumIterations(200)
.setRegParam(0.1)
.setUpdater(new L1Updater());
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862920
--- Diff: docs/mllib-linear-methods.md ---
@@ -242,7 +242,96 @@ Similarly, you can use replace `SVMWithSGD` by
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import java.util.Random;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.SparkContext;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.classification.*;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+public class SVMClassifier {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("SVM Classifier Example");
+ SparkContext sc = new SparkContext(conf);
+ String path = "{SPARK_HOME}/mllib/data/sample_libsvm_data.txt";
+ JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
+
+ // Split initial RDD into two... [60% training data, 40% testing data].
+ JavaRDD<LabeledPoint> training = data.filter(
--- End diff --
Shall we use `RDD.randomSplit` here? The code below is not correct because you use the same random seed for all partitions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by miccagiann <gi...@git.apache.org>.
Github user miccagiann commented on the pull request:
https://github.com/apache/spark/pull/1311#issuecomment-48416509
Hi guys,
I have make a new commit which includes java examples in files "docs/mllib-clustering.md" and "docs/mllib-collaborative-filtering.md". These examples are equivalent to the scala examples provided in the documentation, with the only difference being that should be run by users as standalone applications.
Thanks,
Michael
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862906
--- Diff: docs/mllib-clustering.md ---
@@ -69,7 +69,54 @@ println("Within Set Sum of Squared Errors = " + WSSSE)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.clustering.KMeans;
+import org.apache.spark.mllib.clustering.KMeansModel;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Vector;
+
+public class KMeansExample {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("K-means Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse data
+ String path = "{SPARK_HOME}/data/kmeans_data.txt";
--- End diff --
We moved all data files to `data/mllib` in #1394 . Could you merge the current master and update the path here and in other places? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14696416
--- Diff: docs/mllib-linear-methods.md ---
@@ -242,7 +242,98 @@ Similarly, you can use replace `SVMWithSGD` by
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.SparkContext;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.classification.*;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+import java.util.Random;
--- End diff --
For imports we usually have four groups:
1. java
2. scala
3. 3rd party
4. spark
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862910
--- Diff: docs/mllib-collaborative-filtering.md ---
@@ -99,7 +99,88 @@ val model = ALS.trainImplicit(ratings, rank, numIterations, alpha)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import scala.Product;
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
--- End diff --
ditto: order imports alphabetically
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by miccagiann <gi...@git.apache.org>.
Github user miccagiann commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14926802
--- Diff: docs/mllib-linear-methods.md ---
@@ -242,7 +242,96 @@ Similarly, you can use replace `SVMWithSGD` by
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import java.util.Random;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.SparkContext;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.classification.*;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+public class SVMClassifier {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("SVM Classifier Example");
+ SparkContext sc = new SparkContext(conf);
+ String path = "{SPARK_HOME}/mllib/data/sample_libsvm_data.txt";
+ JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
+
+ // Split initial RDD into two... [60% training data, 40% testing data].
+ JavaRDD<LabeledPoint> training = data.filter(
+ new Function<LabeledPoint, Boolean>() {
+ public final Random random = new Random(11L);
+ public Boolean call(LabeledPoint p) {
+ if (random.nextDouble() <= 0.6)
+ return true;
+ else
+ return false;
+ }
+ }
+ );
+ training.cache();
+ JavaRDD<LabeledPoint> test = data.subtract(training);
+
+ // Run training algorithm to build the model.
+ int numIterations = 100;
+ final SVMModel model = SVMWithSGD.train(JavaRDD.toRDD(training), numIterations);
+
+ // Clear the default threshold.
+ model.clearThreshold();
+
+ // Compute raw scores on the test set.
+ JavaRDD<Tuple2<Object, Object>> scoreAndLabels = test.map(
+ new Function<LabeledPoint, Tuple2<Object, Object>>() {
+ public final SVMModel m = model;
+ public Tuple2<Object, Object> call(LabeledPoint p) {
+ Double score = m.predict(p.features());
--- End diff --
The only requirement for an inner class to call a variable declared in the outer class is by declaring the variable as final. I have fixed it! Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14696245
--- Diff: docs/mllib-clustering.md ---
@@ -69,7 +69,55 @@ println("Within Set Sum of Squared Errors = " + WSSSE)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.clustering.KMeans;
+import org.apache.spark.mllib.clustering.KMeansModel;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Vector;
+
+public class Classifier {
--- End diff --
Is `KMeansExample` better?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862915
--- Diff: docs/mllib-collaborative-filtering.md ---
@@ -99,7 +99,88 @@ val model = ALS.trainImplicit(ratings, rank, numIterations, alpha)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import scala.Product;
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.recommendation.ALS;
+import org.apache.spark.mllib.recommendation.Rating;
+import org.apache.spark.mllib.recommendation.MatrixFactorizationModel;
+
+public class CollaborativeFiltering {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("Collaborative Filtering Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse the data
+ String path = "/home/michael/workspace/spark/mllib/data/als/test.data";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<Rating> ratings = data.map(
+ new Function<String, Rating>() {
+ public Rating call(String s) {
+ String[] sarray = s.split(",");
+ return new Rating(Integer.parseInt(sarray[0]), Integer.parseInt(sarray[1]),
+ Double.parseDouble(sarray[2]));
+ }
+ }
+ );
+
+ // Build the recommendation model using ALS
+ int rank = 10;
+ int numIterations = 20;
+ MatrixFactorizationModel model = ALS.train(JavaRDD.toRDD(ratings), rank, numIterations, 0.01);
+
+ // Evaluate the model on rating data
+ JavaRDD<Tuple2<Object, Object>> userProducts = ratings.map(
+ new Function<Rating, Tuple2<Object, Object>>() {
+ public Tuple2<Object, Object> call(Rating r) {
+ return new Tuple2<Object, Object>(r.user(), r.product());
+ }
+ }
+ );
+ JavaPairRDD<Tuple2<Integer, Integer>, Double> predictions = JavaPairRDD.fromJavaRDD(
+ model.predict(JavaRDD.toRDD(userProducts)).toJavaRDD().map(
+ new Function<Rating, Tuple2<Tuple2<Integer, Integer>, Double>>() {
+ public Tuple2<Tuple2<Integer, Integer>, Double> call(Rating r){
+ return new Tuple2<Tuple2<Integer, Integer>, Double>(
+ new Tuple2<Integer, Integer>(r.user(), r.product()), r.rating());
+ }
+ }
+ ));
+ JavaPairRDD<Tuple2<Integer, Integer>, Tuple2<Double, Double>> ratesAndPreds =
+ JavaPairRDD.fromJavaRDD(ratings.map(
+ new Function<Rating, Tuple2<Tuple2<Integer, Integer>, Double>>() {
+ public Tuple2<Tuple2<Integer, Integer>, Double> call(Rating r){
+ return new Tuple2<Tuple2<Integer, Integer>, Double>(
+ new Tuple2<Integer, Integer>(r.user(), r.product()), r.rating());
+ }
+ }
+ )).join(predictions);
+ double MSE = JavaDoubleRDD.fromRDD(JavaRDD.toRDD(ratesAndPreds.map(
+ new Function<Tuple2<Tuple2<Integer, Integer>, Tuple2<Double, Double>>, Object>() {
+ public Object call(Tuple2<Tuple2<Integer, Integer>, Tuple2<Double, Double>> key_value) {
--- End diff --
`key_value` -> `keyValue` or `pair`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862917
--- Diff: docs/mllib-dimensionality-reduction.md ---
@@ -57,10 +57,57 @@ val U: RowMatrix = svd.U // The U factor is a RowMatrix.
val s: Vector = svd.s // The singular values are stored in a local dense vector.
val V: Matrix = svd.V // The V factor is a local dense matrix.
{% endhighlight %}
+
+Same code applies to `IndexedRowMatrix`.
+The only difference that the `U` matrix becomes an `IndexedRowMatrix`.
</div>
+<div data-lang="java" markdown="1">
+In order to run the following standalone application using Spark framework make
+sure that you follow the instructions provided at section [Standalone
+Applications](quick-start.html) of the quick-start guide. What is more, you
+should include to your build file *spark-mllib* as a dependency.
+
+{% highlight java %}
+import java.util.LinkedList;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.rdd.RDD;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.distributed.RowMatrix;
+import org.apache.spark.mllib.linalg.SingularValueDecomposition;
+
+public class SVD {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("SVD Example");
+ SparkContext sc = new SparkContext(conf);
+
+ double[][] array = ...
+ LinkedList<Vector> rowsList = new LinkedList<Vector>();
+ for (int i = 0; i < array.length; i++) {
+ Vector currentRow = Vectors.dense(array[i]);
+ rowsList.add(currentRow);
+ }
+ JavaRDD<Vector> rows = JavaSparkContext.fromSparkContext(sc).parallelize(rowsList);
+
+ // Create a RowMatrix from JavaRDD<Vector>.
+ RowMatrix mat = new RowMatrix(JavaRDD.toRDD(rows));
--- End diff --
ditto: `.rdd()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by miccagiann <gi...@git.apache.org>.
Github user miccagiann commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14926282
--- Diff: docs/mllib-linear-methods.md ---
@@ -242,7 +242,96 @@ Similarly, you can use replace `SVMWithSGD` by
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import java.util.Random;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.SparkContext;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.classification.*;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+public class SVMClassifier {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("SVM Classifier Example");
+ SparkContext sc = new SparkContext(conf);
+ String path = "{SPARK_HOME}/mllib/data/sample_libsvm_data.txt";
+ JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
+
+ // Split initial RDD into two... [60% training data, 40% testing data].
+ JavaRDD<LabeledPoint> training = data.filter(
--- End diff --
I am going to use the `JavaRDD.sample` method without replacement. I tried to use `RDD.randomSplit` without success...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862939
--- Diff: docs/mllib-linear-methods.md ---
@@ -338,7 +427,74 @@ and [`LassoWithSGD`](api/scala/index.html#org.apache.spark.mllib.regression.Lass
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. The corresponding Java example to
+the Scala snippet provided, is presented bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.regression.LinearRegressionWithSGD;
+import org.apache.spark.mllib.regression.LinearRegressionModel;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Vector;
+
+public class LinearRegression {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("Linear Regression Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse the data
+ String path = "{SPARK_HOME}/mllib/data/ridge-data/lpsa.data";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<LabeledPoint> parsedData = data.map(
+ new Function<String, LabeledPoint>() {
+ public LabeledPoint call(String line) {
+ String[] parts = line.split(",");
+ String[] features = parts[1].split(" ");
+ double[] v = new double[features.length];
+ for (int i = 0; i < features.length - 1; i++)
+ v[i] = Double.parseDouble(features[i]);
+ return new LabeledPoint(Double.parseDouble(parts[0]), Vectors.dense(v));
+ }
+ }
+ );
+
+ // Building the model
+ int numIterations = 100;
+ final LinearRegressionModel model =
+ LinearRegressionWithSGD.train(JavaRDD.toRDD(parsedData), numIterations);
+
+ // Evaluate model on training examples and compute training error
+ JavaRDD<double[]> valuesAndPreds = parsedData.map(
--- End diff --
It is shorter but not intuitive to use `double[]` instead of `Tuple2<Double, Double>`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1311#issuecomment-48114638
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862911
--- Diff: docs/mllib-collaborative-filtering.md ---
@@ -99,7 +99,88 @@ val model = ALS.trainImplicit(ratings, rank, numIterations, alpha)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import scala.Product;
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.recommendation.ALS;
+import org.apache.spark.mllib.recommendation.Rating;
+import org.apache.spark.mllib.recommendation.MatrixFactorizationModel;
+
+public class CollaborativeFiltering {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("Collaborative Filtering Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse the data
+ String path = "/home/michael/workspace/spark/mllib/data/als/test.data";
--- End diff --
Please use a relative path.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862916
--- Diff: docs/mllib-collaborative-filtering.md ---
@@ -99,7 +99,88 @@ val model = ALS.trainImplicit(ratings, rank, numIterations, alpha)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import scala.Product;
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.recommendation.ALS;
+import org.apache.spark.mllib.recommendation.Rating;
+import org.apache.spark.mllib.recommendation.MatrixFactorizationModel;
+
+public class CollaborativeFiltering {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("Collaborative Filtering Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse the data
+ String path = "/home/michael/workspace/spark/mllib/data/als/test.data";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<Rating> ratings = data.map(
+ new Function<String, Rating>() {
+ public Rating call(String s) {
+ String[] sarray = s.split(",");
+ return new Rating(Integer.parseInt(sarray[0]), Integer.parseInt(sarray[1]),
+ Double.parseDouble(sarray[2]));
+ }
+ }
+ );
+
+ // Build the recommendation model using ALS
+ int rank = 10;
+ int numIterations = 20;
+ MatrixFactorizationModel model = ALS.train(JavaRDD.toRDD(ratings), rank, numIterations, 0.01);
+
+ // Evaluate the model on rating data
+ JavaRDD<Tuple2<Object, Object>> userProducts = ratings.map(
+ new Function<Rating, Tuple2<Object, Object>>() {
+ public Tuple2<Object, Object> call(Rating r) {
+ return new Tuple2<Object, Object>(r.user(), r.product());
+ }
+ }
+ );
+ JavaPairRDD<Tuple2<Integer, Integer>, Double> predictions = JavaPairRDD.fromJavaRDD(
+ model.predict(JavaRDD.toRDD(userProducts)).toJavaRDD().map(
+ new Function<Rating, Tuple2<Tuple2<Integer, Integer>, Double>>() {
+ public Tuple2<Tuple2<Integer, Integer>, Double> call(Rating r){
+ return new Tuple2<Tuple2<Integer, Integer>, Double>(
+ new Tuple2<Integer, Integer>(r.user(), r.product()), r.rating());
+ }
+ }
+ ));
+ JavaPairRDD<Tuple2<Integer, Integer>, Tuple2<Double, Double>> ratesAndPreds =
+ JavaPairRDD.fromJavaRDD(ratings.map(
+ new Function<Rating, Tuple2<Tuple2<Integer, Integer>, Double>>() {
+ public Tuple2<Tuple2<Integer, Integer>, Double> call(Rating r){
+ return new Tuple2<Tuple2<Integer, Integer>, Double>(
+ new Tuple2<Integer, Integer>(r.user(), r.product()), r.rating());
+ }
+ }
+ )).join(predictions);
+ double MSE = JavaDoubleRDD.fromRDD(JavaRDD.toRDD(ratesAndPreds.map(
+ new Function<Tuple2<Tuple2<Integer, Integer>, Tuple2<Double, Double>>, Object>() {
+ public Object call(Tuple2<Tuple2<Integer, Integer>, Tuple2<Double, Double>> key_value) {
+ Tuple2<Tuple2<Integer, Integer>, Tuple2<Double, Double>> val =
+ (Tuple2<Tuple2<Integer, Integer>, Tuple2<Double, Double>>) key_value.productElement(1);
+ Double err = ((Double) val.productElement(0)) - ((Double) val.productElement(1));
+ return err * err;
+ }
+ }
+ ))).mean();
--- End diff --
If we pick the values after join (line 163), this block could be simplified:
~~~
JavaRDD<Tuple2<Double, Double>> ratesAndPreds =
JavaPairRDD.fromJavaRDD(ratings.map(
new Function<Rating, Tuple2<Tuple2<Integer, Integer>, Double>>() {
public Tuple2<Tuple2<Integer, Integer>, Double> call(Rating r) {
return new Tuple2<Tuple2<Integer, Integer>, Double>(
new Tuple2<Integer, Integer>(r.user(), r.product()), r.rating());
}
}
)).join(predictions).values();
double MSE = JavaDoubleRDD.fromRDD(ratesAndPreds.map(
new Function<Tuple2<Double, Double>, Object>() {
public Object call(Tuple2<Double, Double> pair) {
double err = pair._1() - pair._2();
return err * err;
}
}
).rdd()).mean();
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by miccagiann <gi...@git.apache.org>.
Github user miccagiann commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14800873
--- Diff: docs/mllib-linear-methods.md ---
@@ -338,7 +429,76 @@ and [`LassoWithSGD`](api/scala/index.html#org.apache.spark.mllib.regression.Lass
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. The corresponding Java example to
+the Scala snippet provided, is presented bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.regression.LinearRegressionWithSGD;
+import org.apache.spark.mllib.regression.LinearRegressionModel;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Vector;
+
+import scala.Product2;
+
+public class LinearRegression {
+ public static void main( String[] args ) {
+ SparkConf conf = new SparkConf().setAppName("Linear Regression Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse the data
+ String path = "{SPARK_HOME}/mllib/data/ridge-data/lpsa.data";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<LabeledPoint> parsedData = data.map(
+ new Function<String, LabeledPoint>() {
+ public LabeledPoint call(String line) {
+ String[] parts = line.split(",");
+ String[] features = parts[1].split(" ");
+ double[] v = new double[features.length];
+ for (int i = 0; i < features.length - 1; i++)
+ v[i] = Double.parseDouble(features[i]);
+ return new LabeledPoint(Double.parseDouble(parts[0]), Vectors.dense(v));
+ }
+ }
+ );
+
+ // Building the model
+ int numIterations = 100;
+ final LinearRegressionModel model = LinearRegressionWithSGD.train(JavaRDD.toRDD(parsedData), numIterations);
+
+ // Evaluate model on training examples and compute training error
+ JavaRDD<double[]> valuesAndPreds = parsedData.map(
--- End diff --
From my understanding the LabeledPoint instances consist of two fields:
a) The first field is the label of the point.
b) The second field is a double[] array (Vector[Double]) that consists of all the attributes' values.
I assume that I should switch the values in 'valuesAndPreds' variable so as prediction to proceed the actual label of each datapoint included in ths JavaRDD<double[]>.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862952
--- Diff: docs/mllib-optimization.md ---
@@ -263,7 +267,110 @@ println("Loss of each step in training process")
loss.foreach(println)
println("Area under ROC = " + auROC)
{% endhighlight %}
-
+</div>
+
+<div data-lang="java" markdown="1">
+{% highlight java %}
+import java.util.Random;
+import java.util.Arrays;
+
+import scala.Product2;
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.optimization.*;
+import org.apache.spark.mllib.classification.LogisticRegressionModel;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+public class LBFGSExample {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("L-BFGS Example");
+ SparkContext sc = new SparkContext(conf);
+ String path = "{SPARK_HOME}/mllib/data/sample_libsvm_data.txt";
+ JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
+ int numFeatures = data.take(1).get(0).features().size();
+
+ // Split initial RDD into two... [60% training data, 40% testing data].
+ JavaRDD<LabeledPoint> trainingInit = data.filter(
--- End diff --
ditto: use `randomSplit`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862921
--- Diff: docs/mllib-linear-methods.md ---
@@ -242,7 +242,96 @@ Similarly, you can use replace `SVMWithSGD` by
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import java.util.Random;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.SparkContext;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.classification.*;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+public class SVMClassifier {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("SVM Classifier Example");
+ SparkContext sc = new SparkContext(conf);
+ String path = "{SPARK_HOME}/mllib/data/sample_libsvm_data.txt";
+ JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
+
+ // Split initial RDD into two... [60% training data, 40% testing data].
+ JavaRDD<LabeledPoint> training = data.filter(
+ new Function<LabeledPoint, Boolean>() {
+ public final Random random = new Random(11L);
+ public Boolean call(LabeledPoint p) {
+ if (random.nextDouble() <= 0.6)
+ return true;
+ else
+ return false;
+ }
+ }
+ );
+ training.cache();
+ JavaRDD<LabeledPoint> test = data.subtract(training);
+
+ // Run training algorithm to build the model.
+ int numIterations = 100;
+ final SVMModel model = SVMWithSGD.train(JavaRDD.toRDD(training), numIterations);
+
+ // Clear the default threshold.
+ model.clearThreshold();
+
+ // Compute raw scores on the test set.
+ JavaRDD<Tuple2<Object, Object>> scoreAndLabels = test.map(
+ new Function<LabeledPoint, Tuple2<Object, Object>>() {
+ public final SVMModel m = model;
+ public Tuple2<Object, Object> call(LabeledPoint p) {
+ Double score = m.predict(p.features());
--- End diff --
Does it work if we use "model" directly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1311#issuecomment-48433888
@miccagiann Thanks for adding Java examples! Please add `[SPARK-1945][MLLIB]` to the title of this PR so that it is easier to find.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862907
--- Diff: docs/mllib-clustering.md ---
@@ -69,7 +69,54 @@ println("Within Set Sum of Squared Errors = " + WSSSE)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.clustering.KMeans;
+import org.apache.spark.mllib.clustering.KMeansModel;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Vector;
+
+public class KMeansExample {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("K-means Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse data
+ String path = "{SPARK_HOME}/data/kmeans_data.txt";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<Vector> parsedData = data.map(
+ new Function<String, Vector>() {
+ public Vector call(String s) {
+ String[] sarray = s.split(" ");
+ double[] values = new double[sarray.length];
+ for (int i = 0; i < sarray.length; i++)
+ values[i] = Double.parseDouble(sarray[i]);
+ return Vectors.dense(values);
+ }
+ }
+ );
+
+ // Cluster the data into two classes using KMeans
+ int numClusters = 2;
+ int numIterations = 20;
+ KMeansModel clusters = KMeans.train(JavaRDD.toRDD(parsedData), numClusters, numIterations);
--- End diff --
`JavaRDD.toRDD(parsedData)` -> `parsedData.rdd()` (simpler)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862947
--- Diff: docs/mllib-linear-methods.md ---
@@ -338,7 +427,74 @@ and [`LassoWithSGD`](api/scala/index.html#org.apache.spark.mllib.regression.Lass
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. The corresponding Java example to
+the Scala snippet provided, is presented bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.regression.LinearRegressionWithSGD;
+import org.apache.spark.mllib.regression.LinearRegressionModel;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Vector;
+
+public class LinearRegression {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("Linear Regression Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse the data
+ String path = "{SPARK_HOME}/mllib/data/ridge-data/lpsa.data";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<LabeledPoint> parsedData = data.map(
+ new Function<String, LabeledPoint>() {
+ public LabeledPoint call(String line) {
+ String[] parts = line.split(",");
+ String[] features = parts[1].split(" ");
+ double[] v = new double[features.length];
+ for (int i = 0; i < features.length - 1; i++)
+ v[i] = Double.parseDouble(features[i]);
+ return new LabeledPoint(Double.parseDouble(parts[0]), Vectors.dense(v));
+ }
+ }
+ );
+
+ // Building the model
+ int numIterations = 100;
+ final LinearRegressionModel model =
+ LinearRegressionWithSGD.train(JavaRDD.toRDD(parsedData), numIterations);
+
+ // Evaluate model on training examples and compute training error
+ JavaRDD<double[]> valuesAndPreds = parsedData.map(
+ new Function<LabeledPoint, double[]>() {
+ public final LinearRegressionModel m = model;
+ public double[] call(LabeledPoint point) {
+ double prediction = m.predict(point.features());
+ return new double[] {prediction, point.label()};
+ }
+ }
+ );
+ JavaRDD<Object> MSERdd = valuesAndPreds.map(
+ new Function<double[], Object>() {
+ public Object call(double[] lp) {
+ return Math.pow(lp[0] - lp[1], 2.0);
+ }
+ }
+ );
+ JavaDoubleRDD MSEDoubleRdd = new JavaDoubleRDD(JavaRDD.toRDD(MSERdd));
+ double MSE = MSEDoubleRdd.mean();
--- End diff --
Could you chain the operations from line 480 and get `MSE` directly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14863023
--- Diff: docs/mllib-optimization.md ---
@@ -263,7 +267,110 @@ println("Loss of each step in training process")
loss.foreach(println)
println("Area under ROC = " + auROC)
{% endhighlight %}
-
+</div>
+
+<div data-lang="java" markdown="1">
+{% highlight java %}
+import java.util.Random;
+import java.util.Arrays;
+
+import scala.Product2;
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.optimization.*;
+import org.apache.spark.mllib.classification.LogisticRegressionModel;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+public class LBFGSExample {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("L-BFGS Example");
+ SparkContext sc = new SparkContext(conf);
+ String path = "{SPARK_HOME}/mllib/data/sample_libsvm_data.txt";
+ JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
+ int numFeatures = data.take(1).get(0).features().size();
+
+ // Split initial RDD into two... [60% training data, 40% testing data].
+ JavaRDD<LabeledPoint> trainingInit = data.filter(
+ new Function<LabeledPoint, Boolean>() {
+ public final Random random = new Random(11L);
+ public Boolean call(LabeledPoint p) {
+ if (random.nextDouble() <= 0.6)
+ return true;
+ else
+ return false;
+ }
+ }
+ );
+ JavaRDD<LabeledPoint> test = data.subtract(trainingInit);
+
+ // Append 1 into the training data as intercept.
+ JavaRDD<Tuple2<Object, Vector>> training = data.map(
+ new Function<LabeledPoint, Tuple2<Object, Vector>>() {
+ public Tuple2<Object, Vector> call(LabeledPoint p) {
+ return new Tuple2<Object, Vector>(p.label(), MLUtils.appendBias(p.features()));
+ }
+ });
+ training.cache();
+
+ // Run training algorithm to build the model.
+ int numCorrections = 10;
+ double convergenceTol = 1e-4;
+ int maxNumIterations = 20;
+ double regParam = 0.1;
+ Vector initialWeightsWithIntercept = Vectors.dense(new double[numFeatures + 1]);
+
+ Tuple2<Vector, double[]> result = LBFGS.runLBFGS(
+ JavaRDD.toRDD(training),
--- End diff --
use 2 spaces for continuous indentation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14696472
--- Diff: docs/mllib-linear-methods.md ---
@@ -338,7 +429,76 @@ and [`LassoWithSGD`](api/scala/index.html#org.apache.spark.mllib.regression.Lass
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. The corresponding Java example to
+the Scala snippet provided, is presented bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.regression.LinearRegressionWithSGD;
+import org.apache.spark.mllib.regression.LinearRegressionModel;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Vector;
+
+import scala.Product2;
+
+public class LinearRegression {
+ public static void main( String[] args ) {
+ SparkConf conf = new SparkConf().setAppName("Linear Regression Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse the data
+ String path = "{SPARK_HOME}/mllib/data/ridge-data/lpsa.data";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<LabeledPoint> parsedData = data.map(
+ new Function<String, LabeledPoint>() {
+ public LabeledPoint call(String line) {
+ String[] parts = line.split(",");
+ String[] features = parts[1].split(" ");
+ double[] v = new double[features.length];
+ for (int i = 0; i < features.length - 1; i++)
+ v[i] = Double.parseDouble(features[i]);
+ return new LabeledPoint(Double.parseDouble(parts[0]), Vectors.dense(v));
+ }
+ }
+ );
+
+ // Building the model
+ int numIterations = 100;
+ final LinearRegressionModel model = LinearRegressionWithSGD.train(JavaRDD.toRDD(parsedData), numIterations);
+
+ // Evaluate model on training examples and compute training error
+ JavaRDD<double[]> valuesAndPreds = parsedData.map(
--- End diff --
In MLlib examples, we put prediction as the first element and the label/response as the second element.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862914
--- Diff: docs/mllib-collaborative-filtering.md ---
@@ -99,7 +99,88 @@ val model = ALS.trainImplicit(ratings, rank, numIterations, alpha)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import scala.Product;
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.recommendation.ALS;
+import org.apache.spark.mllib.recommendation.Rating;
+import org.apache.spark.mllib.recommendation.MatrixFactorizationModel;
+
+public class CollaborativeFiltering {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("Collaborative Filtering Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse the data
+ String path = "/home/michael/workspace/spark/mllib/data/als/test.data";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<Rating> ratings = data.map(
+ new Function<String, Rating>() {
+ public Rating call(String s) {
+ String[] sarray = s.split(",");
+ return new Rating(Integer.parseInt(sarray[0]), Integer.parseInt(sarray[1]),
+ Double.parseDouble(sarray[2]));
+ }
+ }
+ );
+
+ // Build the recommendation model using ALS
+ int rank = 10;
+ int numIterations = 20;
+ MatrixFactorizationModel model = ALS.train(JavaRDD.toRDD(ratings), rank, numIterations, 0.01);
+
+ // Evaluate the model on rating data
+ JavaRDD<Tuple2<Object, Object>> userProducts = ratings.map(
+ new Function<Rating, Tuple2<Object, Object>>() {
+ public Tuple2<Object, Object> call(Rating r) {
+ return new Tuple2<Object, Object>(r.user(), r.product());
+ }
+ }
+ );
+ JavaPairRDD<Tuple2<Integer, Integer>, Double> predictions = JavaPairRDD.fromJavaRDD(
+ model.predict(JavaRDD.toRDD(userProducts)).toJavaRDD().map(
--- End diff --
`userProducts.rdd()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1311
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1311#issuecomment-49570135
LGTM. All changes are in documentation, so I skipped Jenkins and merged it into master. Thanks for your contribution!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862909
--- Diff: docs/mllib-clustering.md ---
@@ -69,7 +69,54 @@ println("Within Set Sum of Squared Errors = " + WSSSE)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.clustering.KMeans;
+import org.apache.spark.mllib.clustering.KMeansModel;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Vector;
+
+public class KMeansExample {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("K-means Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse data
+ String path = "{SPARK_HOME}/data/kmeans_data.txt";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<Vector> parsedData = data.map(
+ new Function<String, Vector>() {
+ public Vector call(String s) {
+ String[] sarray = s.split(" ");
+ double[] values = new double[sarray.length];
+ for (int i = 0; i < sarray.length; i++)
+ values[i] = Double.parseDouble(sarray[i]);
+ return Vectors.dense(values);
+ }
+ }
+ );
+
+ // Cluster the data into two classes using KMeans
+ int numClusters = 2;
+ int numIterations = 20;
+ KMeansModel clusters = KMeans.train(JavaRDD.toRDD(parsedData), numClusters, numIterations);
+
+ // Evaluate clustering by computing Within Set Sum of Squared Errors
+ double WSSSE = clusters.computeCost(JavaRDD.toRDD(parsedData));
--- End diff --
ditto: `JavaRDD.toRDD(parsedData)` -> `parsedData.rdd()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14696398
--- Diff: docs/mllib-linear-methods.md ---
@@ -151,10 +151,10 @@ L(\wv;\x,y) := \log(1+\exp( -y \wv^T \x)).
Logistic regression algorithm outputs a logistic regression model, which makes predictions by
applying the logistic function
`\[
-\mathrm{logit}(z) = \frac{1}{1 + e^{-z}}
+\mathrm{f}(z) = \frac{1}{1 + e^{-z}}
--- End diff --
Thanks for catching it!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14862912
--- Diff: docs/mllib-collaborative-filtering.md ---
@@ -99,7 +99,88 @@ val model = ALS.trainImplicit(ratings, rank, numIterations, alpha)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import scala.Product;
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.recommendation.ALS;
+import org.apache.spark.mllib.recommendation.Rating;
+import org.apache.spark.mllib.recommendation.MatrixFactorizationModel;
+
+public class CollaborativeFiltering {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("Collaborative Filtering Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse the data
+ String path = "/home/michael/workspace/spark/mllib/data/als/test.data";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<Rating> ratings = data.map(
+ new Function<String, Rating>() {
+ public Rating call(String s) {
+ String[] sarray = s.split(",");
+ return new Rating(Integer.parseInt(sarray[0]), Integer.parseInt(sarray[1]),
+ Double.parseDouble(sarray[2]));
+ }
+ }
+ );
+
+ // Build the recommendation model using ALS
+ int rank = 10;
+ int numIterations = 20;
+ MatrixFactorizationModel model = ALS.train(JavaRDD.toRDD(ratings), rank, numIterations, 0.01);
--- End diff --
`ratings.rdd()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1311#issuecomment-48434585
@miccagiann In Spark, we follow https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
for Scala code style, while Java code should be similar. I'm going to mark some style issues inline. Please update the code. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14696311
--- Diff: docs/mllib-collaborative-filtering.md ---
@@ -99,7 +99,83 @@ val model = ALS.trainImplicit(ratings, rank, numIterations, alpha)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.recommendation.ALS;
+import org.apache.spark.mllib.recommendation.Rating;
+import org.apache.spark.mllib.recommendation.MatrixFactorizationModel;
+
+import scala.Tuple2;
--- End diff --
put Scala imports in front of Spark imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by miccagiann <gi...@git.apache.org>.
Github user miccagiann commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14915960
--- Diff: docs/mllib-linear-methods.md ---
@@ -242,7 +242,96 @@ Similarly, you can use replace `SVMWithSGD` by
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import java.util.Random;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.SparkContext;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.classification.*;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+public class SVMClassifier {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("SVM Classifier Example");
+ SparkContext sc = new SparkContext(conf);
+ String path = "{SPARK_HOME}/mllib/data/sample_libsvm_data.txt";
+ JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
+
+ // Split initial RDD into two... [60% training data, 40% testing data].
+ JavaRDD<LabeledPoint> training = data.filter(
--- End diff --
Really good call!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by miccagiann <gi...@git.apache.org>.
Github user miccagiann commented on the pull request:
https://github.com/apache/spark/pull/1311#issuecomment-48855958
Hello guys,
I have provided Java examples for the following documentation files:
mllib-clustering.md
mllib-collaborative-filtering.md
mllib-dimensionality-reduction.md
mllib-linear-methods.md
mllib-optimization.md
Enjoy and do not hesitate to contact me for any remark/correction
Thanks,
Michael
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by miccagiann <gi...@git.apache.org>.
Github user miccagiann commented on the pull request:
https://github.com/apache/spark/pull/1311#issuecomment-48683036
I have modified my previous patches so as to comply with coding standards. Please notify me, if anything is not correct.
Thanks,
Michael
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14696246
--- Diff: docs/mllib-clustering.md ---
@@ -69,7 +69,55 @@ println("Within Set Sum of Squared Errors = " + WSSSE)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.clustering.KMeans;
+import org.apache.spark.mllib.clustering.KMeansModel;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Vector;
+
+public class Classifier {
+ public static void main( String[] args ) {
--- End diff --
use two spaces for indentation and remove extra spaces inside `main(...)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14863048
--- Diff: docs/mllib-optimization.md ---
@@ -263,7 +267,110 @@ println("Loss of each step in training process")
loss.foreach(println)
println("Area under ROC = " + auROC)
{% endhighlight %}
-
+</div>
+
+<div data-lang="java" markdown="1">
+{% highlight java %}
+import java.util.Random;
+import java.util.Arrays;
+
+import scala.Product2;
+import scala.Tuple2;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.util.MLUtils;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.optimization.*;
+import org.apache.spark.mllib.classification.LogisticRegressionModel;
+import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
+
+public class LBFGSExample {
+ public static void main(String[] args) {
+ SparkConf conf = new SparkConf().setAppName("L-BFGS Example");
+ SparkContext sc = new SparkContext(conf);
+ String path = "{SPARK_HOME}/mllib/data/sample_libsvm_data.txt";
+ JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
+ int numFeatures = data.take(1).get(0).features().size();
+
+ // Split initial RDD into two... [60% training data, 40% testing data].
+ JavaRDD<LabeledPoint> trainingInit = data.filter(
+ new Function<LabeledPoint, Boolean>() {
+ public final Random random = new Random(11L);
+ public Boolean call(LabeledPoint p) {
+ if (random.nextDouble() <= 0.6)
+ return true;
+ else
+ return false;
+ }
+ }
+ );
+ JavaRDD<LabeledPoint> test = data.subtract(trainingInit);
+
+ // Append 1 into the training data as intercept.
+ JavaRDD<Tuple2<Object, Vector>> training = data.map(
+ new Function<LabeledPoint, Tuple2<Object, Vector>>() {
+ public Tuple2<Object, Vector> call(LabeledPoint p) {
+ return new Tuple2<Object, Vector>(p.label(), MLUtils.appendBias(p.features()));
+ }
+ });
+ training.cache();
+
+ // Run training algorithm to build the model.
+ int numCorrections = 10;
+ double convergenceTol = 1e-4;
+ int maxNumIterations = 20;
+ double regParam = 0.1;
+ Vector initialWeightsWithIntercept = Vectors.dense(new double[numFeatures + 1]);
+
+ Tuple2<Vector, double[]> result = LBFGS.runLBFGS(
+ JavaRDD.toRDD(training),
+ new LogisticGradient(),
+ new SquaredL2Updater(),
+ numCorrections,
+ convergenceTol,
+ maxNumIterations,
+ regParam,
+ initialWeightsWithIntercept);
+ Vector weightsWithIntercept = (Vector) result.productElement(0);
+ double[] loss = (double[]) result.productElement(1);
+
+ final LogisticRegressionModel model = new LogisticRegressionModel(
+ Vectors.dense(Arrays.copyOf(weightsWithIntercept.toArray(), weightsWithIntercept.size() - 1)),
+ (weightsWithIntercept.toArray())[weightsWithIntercept.size() - 1]);
+
+ // Clear the default threshold.
+ model.clearThreshold();
+
+ // Compute raw scores on the test set.
+ JavaRDD<Tuple2<Object, Object>> scoreAndLabels = test.map(
+ new Function<LabeledPoint, Tuple2<Object, Object>>() {
+ public final LogisticRegressionModel m = model;
--- End diff --
fix indentation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1945][MLLIB] Documentation Improvements...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1311#issuecomment-48862778
@miccagiann I made one pass through the example code. Besides inline comments:
1. We moved mllib's data to `data/mllib` in #1394 . Could you please update the paths in this PR? Thanks!
2. Please use `.rdd()` instead of `JavaRDD.toRDD`.
3. Please use `RDD.randomSplit` to obtain `training` and `test`, which should work in Java.
The rest looks good to me. Thanks for adding Java examples!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Java examples included in 'mllib-linear-method...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1311#discussion_r14696372
--- Diff: docs/mllib-clustering.md ---
@@ -69,7 +69,55 @@ println("Within Set Sum of Squared Errors = " + WSSSE)
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
-calling `.rdd()` on your `JavaRDD` object.
+calling `.rdd()` on your `JavaRDD` object. A standalone application example
+that is equivalent to the provided example in Scala is given bellow:
+
+{% highlight java %}
+import org.apache.spark.api.java.*;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.mllib.clustering.KMeans;
+import org.apache.spark.mllib.clustering.KMeansModel;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.linalg.Vector;
+
+public class Classifier {
+ public static void main( String[] args ) {
+ SparkConf conf = new SparkConf().setAppName("K-means Example");
+ JavaSparkContext sc = new JavaSparkContext(conf);
+
+ // Load and parse data
+ String path = "{SPARK_HOME}/data/kmeans_data.txt";
+ JavaRDD<String> data = sc.textFile(path);
+ JavaRDD<Vector> parsedData = data.map(
+ new Function<String, Vector>() {
+ public Vector call( String s ) {
+ String[] sarray = s.split(" ");
+ double[] values = new double[sarray.length];
+ for (int i = 0; i < sarray.length; i++)
+ values[i] = Double.parseDouble(sarray[i]);
+ return Vectors.dense(values);
+ }
+ }
+ );
+
+ // Cluster the data into two classes using KMeans
+ int numClusters = 2;
+ int numIterations = 20;
+ KMeansModel clusters = KMeans.train(JavaRDD.toRDD(parsedData), numClusters, numIterations);
+
+ // Evaluate clustering by computing Within Set Sum of Squared Errors
+ double WSSSE = clusters.computeCost(JavaRDD.toRDD(parsedData));
+ System.out.println("Within Set Sum of Squared Errors = " + WSSSE);
+ }
+}
+{% endhighlight %}
+
+In order to run the following standalone application using Spark framework make
+sure that you follow the instructions provided at section [Standalone
+Applications](quick-start.html) of the quick-start guide. What is more, you
+should include to your **pom.xml** file both *spark-core_2.10* and
--- End diff --
pom is specific to maven. we also support sbt for build. It should be sufficient to say "include spark-mllib` as dependencies. spark-mllib depends on spark-core so it is not necessary to include spark-core explicitly. Also, better not mention the scala version since then we need to update it if we upgrade to scala 2.11.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---