You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yanboliang <gi...@git.apache.org> on 2016/04/29 15:37:02 UTC
[GitHub] spark pull request: [SPARK-14314] [SPARK-14315] Model persistence ...
GitHub user yanboliang opened a pull request:
https://github.com/apache/spark/pull/12778
[SPARK-14314] [SPARK-14315] Model persistence in SparkR (glm & kmeans)
## What changes were proposed in this pull request?
SparkR ```glm``` and ```kmeans``` model persistence.
## How was this patch tested?
Unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yanboliang/spark spark-14311
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12778.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12778
----
commit d6c7b6c1079a683e940cfaf27067759a5ffcf233
Author: Yanbo Liang <yb...@gmail.com>
Date: 2016-04-29T13:33:36Z
Model persistence in SparkR (glm & kmeans)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14314] [SPARK-14315] [ML] [SparkR] Mode...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12778#issuecomment-215731109
**[Test build #57336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57336/consoleFull)** for PR 12778 at commit [`d6c7b6c`](https://github.com/apache/spark/commit/d6c7b6c1079a683e940cfaf27067759a5ffcf233).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` class GeneralizedLinearRegressionWrapperWriter(instance: GeneralizedLinearRegressionWrapper)`
* ` class KMeansWrapperWriter(instance: KMeansWrapper) extends MLWriter `
* ` class KMeansWrapperReader extends MLReader[KMeansWrapper] `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14314] [SPARK-14315] [ML] [SparkR] Mode...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12778#issuecomment-215731456
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14314] [SPARK-14315] [ML] [SparkR] Mode...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/12778#issuecomment-215759504
cc @mengxr
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14314] [SPARK-14315] [ML] [SparkR] Mode...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/12778#discussion_r61594078
--- Diff: mllib/src/main/scala/org/apache/spark/ml/r/GeneralizedLinearRegressionWrapper.scala ---
@@ -17,65 +17,34 @@
package org.apache.spark.ml.r
+import org.apache.hadoop.fs.Path
+import org.json4s._
+import org.json4s.JsonDSL._
+import org.json4s.jackson.JsonMethods._
+
import org.apache.spark.ml.{Pipeline, PipelineModel}
import org.apache.spark.ml.attribute.AttributeGroup
import org.apache.spark.ml.feature.RFormula
import org.apache.spark.ml.regression._
+import org.apache.spark.ml.util._
import org.apache.spark.sql._
private[r] class GeneralizedLinearRegressionWrapper private (
- pipeline: PipelineModel,
- val features: Array[String]) {
+ val pipeline: PipelineModel,
+ val rFeatures: Array[String],
+ val rCoefficients: Array[Double],
+ val rDispersion: Double,
+ val rNullDeviance: Double,
+ val rDeviance: Double,
+ val rResidualDegreeOfFreedomNull: Long,
+ val rResidualDegreeOfFreedom: Long,
+ val rAic: Double,
+ val rNumIterations: Int,
+ val isLoaded: Boolean = false) extends MLWritable {
private val glm: GeneralizedLinearRegressionModel =
pipeline.stages(1).asInstanceOf[GeneralizedLinearRegressionModel]
- lazy val rFeatures: Array[String] = if (glm.getFitIntercept) {
- Array("(Intercept)") ++ features
- } else {
- features
- }
-
- lazy val rCoefficients: Array[Double] = if (glm.getFitIntercept) {
- Array(glm.intercept) ++ glm.coefficients.toArray ++
- rCoefficientStandardErrors ++ rTValues ++ rPValues
- } else {
- glm.coefficients.toArray ++ rCoefficientStandardErrors ++ rTValues ++ rPValues
- }
-
- private lazy val rCoefficientStandardErrors = if (glm.getFitIntercept) {
- Array(glm.summary.coefficientStandardErrors.last) ++
- glm.summary.coefficientStandardErrors.dropRight(1)
- } else {
- glm.summary.coefficientStandardErrors
- }
-
- private lazy val rTValues = if (glm.getFitIntercept) {
- Array(glm.summary.tValues.last) ++ glm.summary.tValues.dropRight(1)
- } else {
- glm.summary.tValues
- }
-
- private lazy val rPValues = if (glm.getFitIntercept) {
- Array(glm.summary.pValues.last) ++ glm.summary.pValues.dropRight(1)
- } else {
- glm.summary.pValues
- }
-
- lazy val rDispersion: Double = glm.summary.dispersion
-
- lazy val rNullDeviance: Double = glm.summary.nullDeviance
-
- lazy val rDeviance: Double = glm.summary.deviance
-
- lazy val rResidualDegreeOfFreedomNull: Long = glm.summary.residualDegreeOfFreedomNull
-
- lazy val rResidualDegreeOfFreedom: Long = glm.summary.residualDegreeOfFreedom
-
- lazy val rAic: Double = glm.summary.aic
-
- lazy val rNumIterations: Int = glm.summary.numIterations
-
lazy val rDevianceResiduals: DataFrame = glm.summary.residuals()
--- End diff --
Because we did not store ```summary``` in ```GeneralizedLinearRegression.save()```, a saved-loaded model can get ```summary```. We add ```isLoaded``` to distinguish whether the model is save-loaded and corresponding R interface will not call this interface to escape from throwing exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14314] [SPARK-14315] [ML] [SparkR] Mode...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/12778#discussion_r61593663
--- Diff: R/pkg/R/mllib.R ---
@@ -110,15 +110,18 @@ setMethod("summary", signature(object = "GeneralizedLinearRegressionModel"),
aic <- callJMethod(jobj, "rAic")
iter <- callJMethod(jobj, "rNumIterations")
family <- callJMethod(jobj, "rFamily")
-
- deviance.resid <- dataFrame(deviance.resid)
+ deviance.resid <- if (is.loaded) {
+ NULL
+ } else {
+ dataFrame(callJMethod(jobj, "rDevianceResiduals"))
+ }
coefficients <- matrix(coefficients, ncol = 4)
colnames(coefficients) <- c("Estimate", "Std. Error", "t value", "Pr(>|t|)")
rownames(coefficients) <- unlist(features)
ans <- list(deviance.resid = deviance.resid, coefficients = coefficients,
dispersion = dispersion, null.deviance = null.deviance,
deviance = deviance, df.null = df.null, df.residual = df.residual,
- aic = aic, iter = iter, family = family)
+ aic = aic, iter = iter, family = family, is.loaded = is.loaded)
--- End diff --
Value of type DataFrame such as ```deviance.resid``` can not be get from ```saved-loaded``` model, so I add one parameter ```is.loaded``` to tell users whether the model is ```saved-loaded```.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14314] [SPARK-14315] [ML] [SparkR] Mode...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12778#issuecomment-215731463
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57336/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14314] [SPARK-14315] [ML] [SparkR] Mode...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12778
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14314] [SPARK-14315] [ML] [SparkR] Mode...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12778#issuecomment-215713746
**[Test build #57336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57336/consoleFull)** for PR 12778 at commit [`d6c7b6c`](https://github.com/apache/spark/commit/d6c7b6c1079a683e940cfaf27067759a5ffcf233).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14314] [SPARK-14315] [ML] [SparkR] Mode...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/12778#issuecomment-215802398
LGTM. Merged into master and closed #12680 and #12683. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org