You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yanboliang <gi...@git.apache.org> on 2016/05/27 08:10:59 UTC
[GitHub] spark pull request: [SPARK-15605] [ML] [Examples] Remove JavaDevel...
GitHub user yanboliang opened a pull request:
https://github.com/apache/spark/pull/13353
[SPARK-15605] [ML] [Examples] Remove JavaDeveloperApiExample
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yanboliang/spark spark-15605
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13353.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13353
----
commit 742cc12fd73e7fb7febef4eff98d803f442d19c2
Author: Yanbo Liang <yb...@gmail.com>
Date: 2016-05-27T08:10:11Z
Remove JavaDeveloperApiExample
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13353
LGTM @yanboliang do you want to try committing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13353
**[Test build #59736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59736/consoleFull)** for PR 13353 at commit [`92317a7`](https://github.com/apache/spark/commit/92317a7ceb041b8e6a7cdbb2dfe44bd65a6d4a27).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` // where index i corresponds to class i (i = 0, 1).`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13353
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59736/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the issue:
https://github.com/apache/spark/pull/13353
@jkbradley Yes, I have found another way to fix this issue. Please let me know whether this fix is ok to you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13353
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaD...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:
https://github.com/apache/spark/pull/13353#discussion_r65408086
--- Diff: examples/src/main/java/org/apache/spark/examples/ml/JavaDeveloperApiExample.java ---
@@ -88,153 +95,156 @@ public static void main(String[] args) throws Exception {
}
if (sumPredictions != 0.0) {
throw new Exception("MyJavaLogisticRegression predicted something other than 0," +
- " even though all coefficients are 0!");
+ " even though all coefficients are 0!");
}
spark.stop();
}
-}
-
-/**
- * Example of defining a type of {@link Classifier}.
- *
- * Note: Some IDEs (e.g., IntelliJ) will complain that this will not compile due to
- * {@link org.apache.spark.ml.param.Params#set} using incompatible return types.
- * However, this should still compile and run successfully.
- */
-class MyJavaLogisticRegression
- extends Classifier<Vector, MyJavaLogisticRegression, MyJavaLogisticRegressionModel> {
-
- MyJavaLogisticRegression() {
- init();
- }
-
- MyJavaLogisticRegression(String uid) {
- this.uid_ = uid;
- init();
- }
-
- private String uid_ = Identifiable$.MODULE$.randomUID("myJavaLogReg");
-
- @Override
- public String uid() {
- return uid_;
- }
/**
- * Param for max number of iterations
- * <p>
- * NOTE: The usual way to add a parameter to a model or algorithm is to include:
- * - val myParamName: ParamType
- * - def getMyParamName
- * - def setMyParamName
+ * Example of defining a type of {@link Classifier}.
+ *
+ * Note: Some IDEs (e.g., IntelliJ) will complain that this will not compile due to
+ * {@link org.apache.spark.ml.param.Params#set} using incompatible return types.
+ * However, this should still compile and run successfully.
*/
- IntParam maxIter = new IntParam(this, "maxIter", "max number of iterations");
-
- int getMaxIter() { return (Integer) getOrDefault(maxIter); }
-
- private void init() {
- setMaxIter(100);
- }
-
- // The parameter setter is in this class since it should return type MyJavaLogisticRegression.
- MyJavaLogisticRegression setMaxIter(int value) {
- return (MyJavaLogisticRegression) set(maxIter, value);
- }
+ public abstract static class MyJavaLogisticRegression
+ extends Classifier<Vector, MyJavaLogisticRegression, MyJavaLogisticRegressionModel> {
- // This method is used by fit().
- // In Java, we have to make it public since Java does not understand Scala's protected modifier.
- public MyJavaLogisticRegressionModel train(Dataset<?> dataset) {
- // Extract columns from data using helper method.
- JavaRDD<LabeledPoint> oldDataset = extractLabeledPoints(dataset).toJavaRDD();
-
- // Do learning to estimate the coefficients vector.
- int numFeatures = oldDataset.take(1).get(0).features().size();
- Vector coefficients = Vectors.zeros(numFeatures); // Learning would happen here.
-
- // Create a model, and return it.
- return new MyJavaLogisticRegressionModel(uid(), coefficients).setParent(this);
- }
-
- @Override
- public MyJavaLogisticRegression copy(ParamMap extra) {
- return defaultCopy(extra);
- }
-}
+ MyJavaLogisticRegression() {
+ init();
+ }
-/**
- * Example of defining a type of {@link ClassificationModel}.
- *
- * Note: Some IDEs (e.g., IntelliJ) will complain that this will not compile due to
- * {@link org.apache.spark.ml.param.Params#set} using incompatible return types.
- * However, this should still compile and run successfully.
- */
-class MyJavaLogisticRegressionModel
- extends ClassificationModel<Vector, MyJavaLogisticRegressionModel> {
+ MyJavaLogisticRegression(String uid) {
+ this.uid_ = uid;
+ init();
+ }
- private Vector coefficients_;
- public Vector coefficients() { return coefficients_; }
+ private String uid_;
+ public abstract String uid();
+
+ /**
+ * Param for max number of iterations
+ * <p>
+ * NOTE: The usual way to add a parameter to a model or algorithm is to include:
+ * - val myParamName: ParamType
--- End diff --
This is kinda confusing for a Java example having Scala code, no?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15605] [ML] [Examples] Remove JavaDeveloperApiExa...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/13353
@jkbradley Yes, we can fix it, but it may involves lots of changes which is not reasonable to be get in during QA. The main cause of this bug is that we only implement abstract class ```JavaParams``` who is the wrappers of Scala trait ```Params```, and we still need Java-friendly wrappers for all other traits who extends from Scala ```Params```. Then all inherited traits will be generated correctly in Java.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15605] [ML] [Examples] Remove JavaDevel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13353#issuecomment-222088440
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59478/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13353
Merged to master/2.0 but the JIRA is not closed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15605] [ML] [Examples] Remove JavaDevel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13353#issuecomment-222088437
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13353
**[Test build #59844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59844/consoleFull)** for PR 13353 at commit [`b6e5f28`](https://github.com/apache/spark/commit/b6e5f2868e3d13f27a1c21235a304f36b3a5d81c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15605] [ML] [Examples] Remove JavaDeveloperApiExa...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/13353
I don't think the answer is to remove it; the example used to work. We should identify which PR broke it and try to fix it. I'd guess it's some change to Params or another abstraction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15605] [ML] [Examples] Remove JavaDevel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13353#issuecomment-222087060
**[Test build #59478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59478/consoleFull)** for PR 13353 at commit [`742cc12`](https://github.com/apache/spark/commit/742cc12fd73e7fb7febef4eff98d803f442d19c2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13353
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15605] [ML] [Examples] Remove JavaDeveloperApiExa...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/13353
cc @mengxr @jkbradley
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaD...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13353
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15605] [ML] [Examples] Remove JavaDevel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13353#issuecomment-222088355
**[Test build #59478 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59478/consoleFull)** for PR 13353 at commit [`742cc12`](https://github.com/apache/spark/commit/742cc12fd73e7fb7febef4eff98d803f442d19c2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the issue:
https://github.com/apache/spark/pull/13353
@srowen Please help to merge it. I have not set up commit environment completely and I will try to get it work in a few days. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaD...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/13353#discussion_r65518060
--- Diff: examples/src/main/java/org/apache/spark/examples/ml/JavaDeveloperApiExample.java ---
@@ -88,153 +95,156 @@ public static void main(String[] args) throws Exception {
}
if (sumPredictions != 0.0) {
throw new Exception("MyJavaLogisticRegression predicted something other than 0," +
- " even though all coefficients are 0!");
+ " even though all coefficients are 0!");
}
spark.stop();
}
-}
-
-/**
- * Example of defining a type of {@link Classifier}.
- *
- * Note: Some IDEs (e.g., IntelliJ) will complain that this will not compile due to
- * {@link org.apache.spark.ml.param.Params#set} using incompatible return types.
- * However, this should still compile and run successfully.
- */
-class MyJavaLogisticRegression
- extends Classifier<Vector, MyJavaLogisticRegression, MyJavaLogisticRegressionModel> {
-
- MyJavaLogisticRegression() {
- init();
- }
-
- MyJavaLogisticRegression(String uid) {
- this.uid_ = uid;
- init();
- }
-
- private String uid_ = Identifiable$.MODULE$.randomUID("myJavaLogReg");
-
- @Override
- public String uid() {
- return uid_;
- }
/**
- * Param for max number of iterations
- * <p>
- * NOTE: The usual way to add a parameter to a model or algorithm is to include:
- * - val myParamName: ParamType
- * - def getMyParamName
- * - def setMyParamName
+ * Example of defining a type of {@link Classifier}.
+ *
+ * Note: Some IDEs (e.g., IntelliJ) will complain that this will not compile due to
+ * {@link org.apache.spark.ml.param.Params#set} using incompatible return types.
+ * However, this should still compile and run successfully.
*/
- IntParam maxIter = new IntParam(this, "maxIter", "max number of iterations");
-
- int getMaxIter() { return (Integer) getOrDefault(maxIter); }
-
- private void init() {
- setMaxIter(100);
- }
-
- // The parameter setter is in this class since it should return type MyJavaLogisticRegression.
- MyJavaLogisticRegression setMaxIter(int value) {
- return (MyJavaLogisticRegression) set(maxIter, value);
- }
+ public abstract static class MyJavaLogisticRegression
+ extends Classifier<Vector, MyJavaLogisticRegression, MyJavaLogisticRegressionModel> {
- // This method is used by fit().
- // In Java, we have to make it public since Java does not understand Scala's protected modifier.
- public MyJavaLogisticRegressionModel train(Dataset<?> dataset) {
- // Extract columns from data using helper method.
- JavaRDD<LabeledPoint> oldDataset = extractLabeledPoints(dataset).toJavaRDD();
-
- // Do learning to estimate the coefficients vector.
- int numFeatures = oldDataset.take(1).get(0).features().size();
- Vector coefficients = Vectors.zeros(numFeatures); // Learning would happen here.
-
- // Create a model, and return it.
- return new MyJavaLogisticRegressionModel(uid(), coefficients).setParent(this);
- }
-
- @Override
- public MyJavaLogisticRegression copy(ParamMap extra) {
- return defaultCopy(extra);
- }
-}
+ MyJavaLogisticRegression() {
+ init();
+ }
-/**
- * Example of defining a type of {@link ClassificationModel}.
- *
- * Note: Some IDEs (e.g., IntelliJ) will complain that this will not compile due to
- * {@link org.apache.spark.ml.param.Params#set} using incompatible return types.
- * However, this should still compile and run successfully.
- */
-class MyJavaLogisticRegressionModel
- extends ClassificationModel<Vector, MyJavaLogisticRegressionModel> {
+ MyJavaLogisticRegression(String uid) {
+ this.uid_ = uid;
+ init();
+ }
- private Vector coefficients_;
- public Vector coefficients() { return coefficients_; }
+ private String uid_;
+ public abstract String uid();
+
+ /**
+ * Param for max number of iterations
+ * <p>
+ * NOTE: The usual way to add a parameter to a model or algorithm is to include:
+ * - val myParamName: ParamType
--- End diff --
As discussed with @jkbradley , we will remove this example for 2.0. I will address this comments when we add this example back.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13353
**[Test build #59736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59736/consoleFull)** for PR 13353 at commit [`92317a7`](https://github.com/apache/spark/commit/92317a7ceb041b8e6a7cdbb2dfe44bd65a6d4a27).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13353
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59844/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the issue:
https://github.com/apache/spark/pull/13353
@jkbradley I found the main problem is that the ```uid``` defined in Java class will be initialized after Scala traits such as ```HasFeaturesCol```. So when ```HasFeaturesCol``` being constructed, they get ```uid``` with null, which will cause ```Param.shouldOwn``` check fail. I think moving the ```uid``` as constructor argument of ```Classifier``` will solve this problem. But it involves public API changes such as ```Classifier```, ```Regressor```, ```ProbabilisticClassifier```, etc and should be further defined, I agree to remove this for now.
I will track this issue and send my proposal of fix to JIRA for discussion. I think we can find an appropriate fix it at 2.1 and adding this example back.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13353
**[Test build #59844 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59844/consoleFull)** for PR 13353 at commit [`b6e5f28`](https://github.com/apache/spark/commit/b6e5f2868e3d13f27a1c21235a304f36b3a5d81c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15605] [ML] [Examples] Remove JavaDeveloperApiExa...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/13353
I checked and found that this example is broken since Spark 1.4.0. Because before Spark 1.4.0, we don't use ```Params.shouldOwn``` method to validate that the input param belongs to the instance. If we do this validation, it will also fail due to the Java compatibility issue mentioned above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the issue:
https://github.com/apache/spark/pull/13353
@jkbradley @MLnick I removed this example now and it's ready to be get in. I think we can left the JIRA open and feel free to assign it to me for further work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/13353
@yanboliang Thanks for tracking down the issue. With your suggested fix, we could leave the existing no-argument constructors for compatibility but deprecate them. Since few people have complained about this so far, I'd be OK with fixing it now or in 2.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13353: [SPARK-15605] [ML] [Examples] Fix broken ML JavaDevelope...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/13353
Thanks for exploring this! I had hoped we could figure out how to fix the abstractions. However, if it is not feasible in a short time, then maybe we should remove the example and add it back in later once we can fix the abstractions. I'll spend some more time trying to track down the change in the abstractions which broke things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org