You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yanboliang <gi...@git.apache.org> on 2016/05/12 10:08:41 UTC
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
GitHub user yanboliang opened a pull request:
https://github.com/apache/spark/pull/13076
[SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit for classification
## What changes were proposed in this pull request?
Audit Scala API for classification, almost all issues were related MultilayerPerceptronClassifier.
* Fix one wrong param getter/setter method: getOptimizer -> getSolver
* Add missing setter for "solver" and "stepSize".
* Make GD solver take effect.
* Update docs, annotations and fix other minor issues.
## How was this patch tested?
Existing unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yanboliang/spark spark-15292
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13076.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13076
----
commit e9099076ec2ccc79dc14e731c70cff6cc68fe4ed
Author: Yanbo Liang <yb...@gmail.com>
Date: 2016-05-12T10:05:45Z
ML 2.0 QA: Scala APIs audit for classification
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:
https://github.com/apache/spark/pull/13076#discussion_r63142140
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala ---
@@ -65,22 +63,23 @@ private[ml] trait MultilayerPerceptronParams extends PredictorParams
"it is adjusted to the size of this data. Recommended size is between 10 and 1000",
ParamValidators.gt(0))
- /** @group getParam */
+ /** @group expertGetParam */
final def getBlockSize: Int = $(blockSize)
/**
- * Allows setting the solver: minibatch gradient descent (gd) or l-bfgs.
- * l-bfgs is the default one.
+ * The solver algorithm for optimization.
+ * Supported options: "gd" (minibatch gradient descent) or "l-bfgs".
+ * Default: "l-bfgs"
*
* @group expertParam
*/
final val solver: Param[String] = new Param[String](this, "solver",
- " Allows setting the solver: minibatch gradient descent (gd) or l-bfgs. " +
- " l-bfgs is the default one.",
- ParamValidators.inArray[String](Array("gd", "l-bfgs")))
+ "The solver algorithm for optimization. Supported options: " +
+ s"${MultilayerPerceptronClassifier.supportedSolvers.mkString(", ")}. (Default l-bfgs)",
+ ParamValidators.inArray[String](MultilayerPerceptronClassifier.supportedSolvers))
- /** @group getParam */
- final def getOptimizer: String = $(solver)
+ /** @group expertGetParam */
+ final def getSolver: String = $(solver)
/**
* Model weights. Can be returned either after training or after explicit setting
--- End diff --
See my comment [here](https://github.com/apache/spark/pull/12943/files/7cc27e91a9cfd234268c0b3bb3e05c22dddb8ce0#r62917637). I think this doc can be improved to indicate this can set the weights for initializing training, or explicitly set (and override) the weights on a trained model.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-218723662
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219595234
Thanks for these fixes!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219613838
@yanboliang Changing `weights` to `initialWeights` makes a perfect sense, thank you. There were few requests for MLP to provide `rawPredictions`. Since you are working on the API, do you think we can add this too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/13076#discussion_r63452941
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala ---
@@ -204,16 +238,26 @@ class MultilayerPerceptronClassifier @Since("1.5.0") (
val labels = myLayers.last
val lpData = extractLabeledPoints(dataset)
val data = lpData.map(lp => LabelConverter.encodeLabeledPoint(lp, labels))
- val topology = FeedForwardTopology.multiLayerPerceptron(myLayers, true)
+ val topology = FeedForwardTopology.multiLayerPerceptron(myLayers, softmaxOnTop = true)
val trainer = new FeedForwardTrainer(topology, myLayers(0), myLayers.last)
- if (isDefined(weights)) {
- trainer.setWeights($(weights))
+ if (isDefined(initialWeights)) {
+ trainer.setWeights($(initialWeights))
} else {
trainer.setSeed($(seed))
}
- trainer.LBFGSOptimizer
- .setConvergenceTol($(tol))
- .setNumIterations($(maxIter))
+ if ($(solver) == MultilayerPerceptronClassifier.LBFGS) {
+ trainer.LBFGSOptimizer
+ .setConvergenceTol($(tol))
+ .setNumIterations($(maxIter))
+ } else if ($(solver) == MultilayerPerceptronClassifier.GD) {
+ trainer.SGDOptimizer
+ .setNumIterations($(maxIter))
+ .setConvergenceTol($(tol))
+ .setStepSize($(stepSize))
+ } else {
+ throw new UnsupportedOperationException(
--- End diff --
IllegalArgumentException
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:
https://github.com/apache/spark/pull/13076#discussion_r63153534
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala ---
@@ -65,22 +63,23 @@ private[ml] trait MultilayerPerceptronParams extends PredictorParams
"it is adjusted to the size of this data. Recommended size is between 10 and 1000",
ParamValidators.gt(0))
- /** @group getParam */
+ /** @group expertGetParam */
final def getBlockSize: Int = $(blockSize)
/**
- * Allows setting the solver: minibatch gradient descent (gd) or l-bfgs.
- * l-bfgs is the default one.
+ * The solver algorithm for optimization.
+ * Supported options: "gd" (minibatch gradient descent) or "l-bfgs".
+ * Default: "l-bfgs"
*
* @group expertParam
*/
final val solver: Param[String] = new Param[String](this, "solver",
- " Allows setting the solver: minibatch gradient descent (gd) or l-bfgs. " +
- " l-bfgs is the default one.",
- ParamValidators.inArray[String](Array("gd", "l-bfgs")))
+ "The solver algorithm for optimization. Supported options: " +
+ s"${MultilayerPerceptronClassifier.supportedSolvers.mkString(", ")}. (Default l-bfgs)",
+ ParamValidators.inArray[String](MultilayerPerceptronClassifier.supportedSolvers))
- /** @group getParam */
- final def getOptimizer: String = $(solver)
+ /** @group expertGetParam */
+ final def getSolver: String = $(solver)
--- End diff --
ah right - I did wonder why MiMa test didn't pick it up :) This looks fine, 👍 for adding @since annotation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219631618
**[Test build #58668 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58668/consoleFull)** for PR 13076 at commit [`13e7a5b`](https://github.com/apache/spark/commit/13e7a5be72b65a288280bdfc5faa02fc24807f1a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219346632
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58624/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/13076#discussion_r63152101
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala ---
@@ -65,22 +63,23 @@ private[ml] trait MultilayerPerceptronParams extends PredictorParams
"it is adjusted to the size of this data. Recommended size is between 10 and 1000",
ParamValidators.gt(0))
- /** @group getParam */
+ /** @group expertGetParam */
final def getBlockSize: Int = $(blockSize)
/**
- * Allows setting the solver: minibatch gradient descent (gd) or l-bfgs.
- * l-bfgs is the default one.
+ * The solver algorithm for optimization.
+ * Supported options: "gd" (minibatch gradient descent) or "l-bfgs".
+ * Default: "l-bfgs"
*
* @group expertParam
*/
final val solver: Param[String] = new Param[String](this, "solver",
- " Allows setting the solver: minibatch gradient descent (gd) or l-bfgs. " +
- " l-bfgs is the default one.",
- ParamValidators.inArray[String](Array("gd", "l-bfgs")))
+ "The solver algorithm for optimization. Supported options: " +
+ s"${MultilayerPerceptronClassifier.supportedSolvers.mkString(", ")}. (Default l-bfgs)",
+ ParamValidators.inArray[String](MultilayerPerceptronClassifier.supportedSolvers))
- /** @group getParam */
- final def getOptimizer: String = $(solver)
+ /** @group expertGetParam */
+ final def getSolver: String = $(solver)
--- End diff --
Never mind, this function and corresponding param is added after Spark 1.6, so it will not take incompatible issue. I think we should add ```@Since("2.0.0")``` annotation to them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219346630
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-218969694
cc @avulanov
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-218723566
**[Test build #58475 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58475/consoleFull)** for PR 13076 at commit [`e909907`](https://github.com/apache/spark/commit/e9099076ec2ccc79dc14e731c70cff6cc68fe4ed).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219745995
@yanboliang Sounds good! No one is working on that right now, it would be great if you could.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219346592
**[Test build #58624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58624/consoleFull)** for PR 13076 at commit [`0cdebb0`](https://github.com/apache/spark/commit/0cdebb0792e43dd83c7f57c9ca6d1e5fea3f5396).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-218723663
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58475/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219626509
**[Test build #58668 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58668/consoleFull)** for PR 13076 at commit [`13e7a5b`](https://github.com/apache/spark/commit/13e7a5be72b65a288280bdfc5faa02fc24807f1a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/13076#discussion_r63452938
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala ---
@@ -181,12 +206,21 @@ class MultilayerPerceptronClassifier @Since("1.5.0") (
def setSeed(value: Long): this.type = set(seed, value)
/**
- * Sets the model weights.
+ * Sets the value of param [[initialWeights]].
*
- * @group expertParam
+ * @group expertSetParam
+ */
+ @Since("2.0.0")
+ def setInitialWeights(value: Vector): this.type = set(initialWeights, value)
+
+ /**
+ * Sets the value of param [[stepSize]].
--- End diff --
State that this param only applies if solver = "gd"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/13076#discussion_r63308195
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala ---
@@ -65,22 +63,23 @@ private[ml] trait MultilayerPerceptronParams extends PredictorParams
"it is adjusted to the size of this data. Recommended size is between 10 and 1000",
ParamValidators.gt(0))
- /** @group getParam */
+ /** @group expertGetParam */
final def getBlockSize: Int = $(blockSize)
/**
- * Allows setting the solver: minibatch gradient descent (gd) or l-bfgs.
- * l-bfgs is the default one.
+ * The solver algorithm for optimization.
+ * Supported options: "gd" (minibatch gradient descent) or "l-bfgs".
+ * Default: "l-bfgs"
*
* @group expertParam
*/
final val solver: Param[String] = new Param[String](this, "solver",
- " Allows setting the solver: minibatch gradient descent (gd) or l-bfgs. " +
- " l-bfgs is the default one.",
- ParamValidators.inArray[String](Array("gd", "l-bfgs")))
+ "The solver algorithm for optimization. Supported options: " +
+ s"${MultilayerPerceptronClassifier.supportedSolvers.mkString(", ")}. (Default l-bfgs)",
+ ParamValidators.inArray[String](MultilayerPerceptronClassifier.supportedSolvers))
- /** @group getParam */
- final def getOptimizer: String = $(solver)
+ /** @group expertGetParam */
+ final def getSolver: String = $(solver)
/**
* Model weights. Can be returned either after training or after explicit setting
--- End diff --
I think this can only used to set initial weights for training. The original documents said we can also get the ```weights``` after training, I speculate that it means we can get the result ```weights``` from ```MultilayerPerceptronClassificationModel.weights``` which is different from this one. So I rename this param to ```initialWeights``` and can differentiate it with the member variable of ```MultilayerPerceptronClassificationModel```.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:
https://github.com/apache/spark/pull/13076#discussion_r63141737
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala ---
@@ -65,22 +63,23 @@ private[ml] trait MultilayerPerceptronParams extends PredictorParams
"it is adjusted to the size of this data. Recommended size is between 10 and 1000",
ParamValidators.gt(0))
- /** @group getParam */
+ /** @group expertGetParam */
final def getBlockSize: Int = $(blockSize)
/**
- * Allows setting the solver: minibatch gradient descent (gd) or l-bfgs.
- * l-bfgs is the default one.
+ * The solver algorithm for optimization.
+ * Supported options: "gd" (minibatch gradient descent) or "l-bfgs".
+ * Default: "l-bfgs"
*
* @group expertParam
*/
final val solver: Param[String] = new Param[String](this, "solver",
- " Allows setting the solver: minibatch gradient descent (gd) or l-bfgs. " +
- " l-bfgs is the default one.",
- ParamValidators.inArray[String](Array("gd", "l-bfgs")))
+ "The solver algorithm for optimization. Supported options: " +
+ s"${MultilayerPerceptronClassifier.supportedSolvers.mkString(", ")}. (Default l-bfgs)",
+ ParamValidators.inArray[String](MultilayerPerceptronClassifier.supportedSolvers))
- /** @group getParam */
- final def getOptimizer: String = $(solver)
+ /** @group expertGetParam */
+ final def getSolver: String = $(solver)
--- End diff --
This will be a binary incompatible change. It's @Experimental so I guess it is ok for Spark 2.0 (cc @jkbradley)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219631693
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219626330
Yeah I agree - let's add that for next release.
On Tue, 17 May 2016 at 07:37, Yanbo Liang <no...@github.com> wrote:
> @avulanov <https://github.com/avulanov> Thanks for your comments. To
> provide rawPredictions is make sense, but I think it's a new feature and
> we should do it in the next release cycle. I can work on it if no one has
> started. We have code freeze for 2.0 and only API audit, bug fix and
> documentation update should be merged.
>
> —
> You are receiving this because you commented.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/13076#issuecomment-219623827>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13076
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-218715755
**[Test build #58475 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58475/consoleFull)** for PR 13076 at commit [`e909907`](https://github.com/apache/spark/commit/e9099076ec2ccc79dc14e731c70cff6cc68fe4ed).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/13076#discussion_r63452937
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala ---
@@ -65,22 +63,23 @@ private[ml] trait MultilayerPerceptronParams extends PredictorParams
"it is adjusted to the size of this data. Recommended size is between 10 and 1000",
ParamValidators.gt(0))
- /** @group getParam */
+ /** @group expertGetParam */
final def getBlockSize: Int = $(blockSize)
/**
- * Allows setting the solver: minibatch gradient descent (gd) or l-bfgs.
- * l-bfgs is the default one.
+ * The solver algorithm for optimization.
+ * Supported options: "gd" (minibatch gradient descent) or "l-bfgs".
+ * Default: "l-bfgs"
*
* @group expertParam
*/
final val solver: Param[String] = new Param[String](this, "solver",
- " Allows setting the solver: minibatch gradient descent (gd) or l-bfgs. " +
- " l-bfgs is the default one.",
- ParamValidators.inArray[String](Array("gd", "l-bfgs")))
+ "The solver algorithm for optimization. Supported options: " +
+ s"${MultilayerPerceptronClassifier.supportedSolvers.mkString(", ")}. (Default l-bfgs)",
+ ParamValidators.inArray[String](MultilayerPerceptronClassifier.supportedSolvers))
- /** @group getParam */
- final def getOptimizer: String = $(solver)
+ /** @group expertGetParam */
+ final def getSolver: String = $(solver)
/**
* Model weights. Can be returned either after training or after explicit setting
--- End diff --
I like this approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219631695
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58668/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-220394624
LGTM
Merging with master
Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219343385
**[Test build #58624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58624/consoleFull)** for PR 13076 at commit [`0cdebb0`](https://github.com/apache/spark/commit/0cdebb0792e43dd83c7f57c9ca6d1e5fea3f5396).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15292] [ML] ML 2.0 QA: Scala APIs audit...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/13076#issuecomment-219623827
@avulanov Thanks for your comments. To provide ```rawPredictions``` is make sense, but I think it's a new feature and we should do it in the next release cycle. I can work on it if no one has started. We have code freeze for 2.0 and only API audit, bug fix and documentation update should be merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org