You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by wangmiao1981 <gi...@git.apache.org> on 2016/10/05 23:41:42 UTC
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
GitHub user wangmiao1981 opened a pull request:
https://github.com/apache/spark/pull/15365
[SPARK-17157][SPARKR]: Add multiclass logistic regression SparkR Wrapper
## What changes were proposed in this pull request?
As we discussed in #14818, I added a separate R wrapper spark.logit for logistic regression.
This single interface supports both binary and multinomial logistic regression. It also has "predict" and "summary" for binary logistic regression.
## How was this patch tested?
New unit tests are added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangmiao1981/spark glm
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15365.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15365
----
commit 7f3d40d090604e1edeadfe7e747869290be61c2f
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-10-05T00:23:18Z
add spark.logit
commit 5bfd1320000791bed933b10dc3af8f0857d6be99
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-10-05T23:36:54Z
add unit tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66857/consoleFull)** for PR 15365 at commit [`08babe5`](https://github.com/apache/spark/commit/08babe50df42bc4247a65f24cc95241891b555a8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84988830
--- Diff: R/pkg/inst/tests/testthat/test_mllib.R ---
@@ -587,6 +587,59 @@ test_that("spark.isotonicRegression", {
unlink(modelPath)
})
+test_that("spark.logit", {
+ # test binary logistic regression
+ label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+ feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+ binary_data <- as.data.frame(cbind(label, feature))
+ binary_df <- createDataFrame(binary_data)
+
+ blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+ blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+ expect_equal(blr_predict$prediction, c(0, 0, 0, 0, 0))
+ blr_model1 <- spark.logit(binary_df, label ~ feature, threshold = 0.0)
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/15365
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84988302
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,173 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' \itemize{
+#' \item{"auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
--- End diff --
There is no `numClasses` parameter. The algorithm infers the number of classes. I changed to `number of classes` to avoid confusion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67293/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82529615
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
+#' value may be 0. The class with largest value p/t is predicted, where p is the original probability
+#' of that class and t is the class's threshold. Note: When thresholds is set, any user-set
+#' value for threshold will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is NULL.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
+#' blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+#'
+#' # summary of binary logistic regression
+#' blr_summary <- summary(blr_model)
+#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold, "threshold", "F-Measure"))
+#' # save fitted model to input path
+#' path <- "path/to/model"
+#' write.ml(blr_model, path)
+#'
+#' # can also read back the saved model and print
+#' savedModel <- read.ml(path)
+#' summary(savedModel)
+#'
+#' # multinomial logistic regression
+#'
+#' label <- c(0.0, 1.0, 2.0, 0.0, 0.0)
+#' feature1 <- c(4.845940, 5.64480, 7.430381, 6.464263, 5.555667)
+#' feature2 <- c(2.941319, 2.614812, 2.162451, 3.339474, 2.970987)
+#' feature3 <- c(1.322733, 1.348044, 3.861237, 9.686976, 3.447130)
+#' feature4 <- c(1.3246388, 0.5510444, 0.9225810, 1.2147881, 1.6020842)
+#' data <- as.data.frame(cbind(label, feature1, feature2, feature3, feature4))
+#' df <- suppressWarnings(createDataFrame(data))
+#'
+#' model <- spark.logit(df, label ~ ., family = "multinomial", thresholds=c(0, 1, 1))
+#' predict1 <- collect(select(predict(model, df), "prediction"))
+#' }
+#' @note spark.logit since 2.1.0
+setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula"),
+ function(data, formula, regParam = 0.0, elasticNetParam = 0.0, maxIter = 100,
+ tol = 1E-6, fitIntercept = TRUE, family = "auto", standardization = TRUE,
+ threshold = 0.5, thresholds = NULL, weightCol = NULL, aggregationDepth = 2) {
--- End diff --
should it support `probabilityCol`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r83121532
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
+#' value may be 0. The class with largest value p/t is predicted, where p is the original probability
+#' of that class and t is the class's threshold. Note: When thresholds is set, any user-set
+#' value for threshold will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is NULL.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
+#' blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+#'
+#' # summary of binary logistic regression
+#' blr_summary <- summary(blr_model)
+#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold, "threshold", "F-Measure"))
+#' # save fitted model to input path
+#' path <- "path/to/model"
+#' write.ml(blr_model, path)
+#'
+#' # can also read back the saved model and print
+#' savedModel <- read.ml(path)
+#' summary(savedModel)
+#'
+#' # multinomial logistic regression
+#'
+#' label <- c(0.0, 1.0, 2.0, 0.0, 0.0)
+#' feature1 <- c(4.845940, 5.64480, 7.430381, 6.464263, 5.555667)
+#' feature2 <- c(2.941319, 2.614812, 2.162451, 3.339474, 2.970987)
+#' feature3 <- c(1.322733, 1.348044, 3.861237, 9.686976, 3.447130)
+#' feature4 <- c(1.3246388, 0.5510444, 0.9225810, 1.2147881, 1.6020842)
+#' data <- as.data.frame(cbind(label, feature1, feature2, feature3, feature4))
+#' df <- suppressWarnings(createDataFrame(data))
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82493602
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
+#' value may be 0. The class with largest value p/t is predicted, where p is the original probability
+#' of that class and t is the class's threshold. Note: When thresholds is set, any user-set
+#' value for threshold will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is NULL.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
--- End diff --
I see. Thanks for your explanation!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66407/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66468/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/15365
merged to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84832641
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,173 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' \itemize{
+#' \item{"auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".}
+#' \item{"binomial": Binary logistic regression with pivoting.}
+#' \item{"multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".}
+#' }
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param thresholds in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds c(1-p, p). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. In multiclass (or binary) classification to adjust the probability of
+#' predicting each class. Array must have length equal to the number of classes, with values > 0,
+#' excepting that at most one value may be 0. The class with largest value p/t is predicted, where p
+#' is the original probability of that class and t is the class's threshold. Note: When thresholds
+#' is set, any user-set value for threshold will be cleared. If both threshold and thresholds are
+#' set, then they must be equivalent. Default is 0.5.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param probabilityCol column name for predicted class conditional probabilities. Default is "probability".
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
+#' blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+#'
+#' # summary of binary logistic regression
+#' blr_summary <- summary(blr_model)
+#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold, "threshold", "F-Measure"))
+#' # save fitted model to input path
+#' path <- "path/to/model"
+#' write.ml(blr_model, path)
+#'
+#' # can also read back the saved model and print
+#' savedModel <- read.ml(path)
+#' summary(savedModel)
+#'
+#' # multinomial logistic regression
+#'
+#' label <- c(0.0, 1.0, 2.0, 0.0, 0.0)
+#' feature1 <- c(4.845940, 5.64480, 7.430381, 6.464263, 5.555667)
+#' feature2 <- c(2.941319, 2.614812, 2.162451, 3.339474, 2.970987)
+#' feature3 <- c(1.322733, 1.348044, 3.861237, 9.686976, 3.447130)
+#' feature4 <- c(1.3246388, 0.5510444, 0.9225810, 1.2147881, 1.6020842)
+#' data <- as.data.frame(cbind(label, feature1, feature2, feature3, feature4))
+#' df <- createDataFrame(data)
+#'
+#' model <- spark.logit(df, label ~ ., family = "multinomial", thresholds=c(0, 1, 1))
+#' predict1 <- collect(select(predict(model, df), "prediction"))
+#' }
+#' @note spark.logit since 2.1.0
+setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula"),
+ function(data, formula, regParam = 0.0, elasticNetParam = 0.0, maxIter = 100,
+ tol = 1E-6, fitIntercept = TRUE, family = "auto", standardization = TRUE,
+ thresholds = 0.5, weightCol = NULL, aggregationDepth = 2,
+ probabilityCol = "probability") {
+ formula <- paste0(deparse(formula), collapse = "")
+
+ if (is.null(weightCol)) {
+ weightCol <- ""
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.LogisticRegressionWrapper", "fit",
+ data@sdf, formula, as.numeric(regParam),
+ as.numeric(elasticNetParam), as.integer(maxIter),
+ as.numeric(tol), as.logical(fitIntercept),
+ as.character(family), as.logical(standardization),
+ as.array(thresholds), as.character(weightCol),
+ as.integer(aggregationDepth), as.character(probabilityCol))
+ new("LogisticRegressionModel", jobj = jobj)
+ })
+
+# Predicted values based on an LogisticRegressionModel model
+
+#' @param newData a SparkDataFrame for testing.
+#' @return \code{predict} returns the predicted values based on an LogisticRegressionModel.
+#' @rdname spark.logit
+#' @aliases predict,LogisticRegressionModel,SparkDataFrame-method
+#' @export
+#' @note predict(LogisticRegressionModel) since 2.1.0
+setMethod("predict", signature(object = "LogisticRegressionModel"),
+ function(object, newData) {
+ predict_internal(object, newData)
+ })
+
+# Get the summary of an LogisticRegressionModel
+
+#' @param object an LogisticRegressionModel fitted by \code{spark.logit}
+#' @return \code{summary} returns the Binary Logistic regression results of a given model as lists. Note that
+#' Multinomial logistic regression summary is not available now.
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
--- End diff --
This is not the right alias..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
Sure. I will do it. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82528971
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
--- End diff --
Array is not often used in R, could we handle vector?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66563/consoleFull)** for PR 15365 at commit [`1921221`](https://github.com/apache/spark/commit/1921221964131b9cf8ba500b2dcf2f2219d3b20c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84832848
--- Diff: R/pkg/inst/tests/testthat/test_mllib.R ---
@@ -587,6 +587,59 @@ test_that("spark.isotonicRegression", {
unlink(modelPath)
})
+test_that("spark.logit", {
+ # test binary logistic regression
+ label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+ feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+ binary_data <- as.data.frame(cbind(label, feature))
+ binary_df <- createDataFrame(binary_data)
+
+ blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+ blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+ expect_equal(blr_predict$prediction, c(0, 0, 0, 0, 0))
+ blr_model1 <- spark.logit(binary_df, label ~ feature, threshold = 0.0)
--- End diff --
ditto thresholds
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66563 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66563/consoleFull)** for PR 15365 at commit [`1921221`](https://github.com/apache/spark/commit/1921221964131b9cf8ba500b2dcf2f2219d3b20c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82528616
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
--- End diff --
roxygen2 is going to trim all the whitespaces, you will need to add formatting with \item and so on
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67531/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84832712
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,173 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' \itemize{
+#' \item{"auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".}
+#' \item{"binomial": Binary logistic regression with pivoting.}
+#' \item{"multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".}
+#' }
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param thresholds in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds c(1-p, p). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. In multiclass (or binary) classification to adjust the probability of
+#' predicting each class. Array must have length equal to the number of classes, with values > 0,
+#' excepting that at most one value may be 0. The class with largest value p/t is predicted, where p
+#' is the original probability of that class and t is the class's threshold. Note: When thresholds
+#' is set, any user-set value for threshold will be cleared. If both threshold and thresholds are
+#' set, then they must be equivalent. Default is 0.5.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param probabilityCol column name for predicted class conditional probabilities. Default is "probability".
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
+#' blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+#'
+#' # summary of binary logistic regression
+#' blr_summary <- summary(blr_model)
+#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold, "threshold", "F-Measure"))
+#' # save fitted model to input path
+#' path <- "path/to/model"
+#' write.ml(blr_model, path)
+#'
+#' # can also read back the saved model and print
+#' savedModel <- read.ml(path)
+#' summary(savedModel)
+#'
+#' # multinomial logistic regression
+#'
+#' label <- c(0.0, 1.0, 2.0, 0.0, 0.0)
+#' feature1 <- c(4.845940, 5.64480, 7.430381, 6.464263, 5.555667)
+#' feature2 <- c(2.941319, 2.614812, 2.162451, 3.339474, 2.970987)
+#' feature3 <- c(1.322733, 1.348044, 3.861237, 9.686976, 3.447130)
+#' feature4 <- c(1.3246388, 0.5510444, 0.9225810, 1.2147881, 1.6020842)
+#' data <- as.data.frame(cbind(label, feature1, feature2, feature3, feature4))
+#' df <- createDataFrame(data)
+#'
+#' model <- spark.logit(df, label ~ ., family = "multinomial", thresholds=c(0, 1, 1))
+#' predict1 <- collect(select(predict(model, df), "prediction"))
+#' }
+#' @note spark.logit since 2.1.0
+setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula"),
+ function(data, formula, regParam = 0.0, elasticNetParam = 0.0, maxIter = 100,
+ tol = 1E-6, fitIntercept = TRUE, family = "auto", standardization = TRUE,
+ thresholds = 0.5, weightCol = NULL, aggregationDepth = 2,
+ probabilityCol = "probability") {
+ formula <- paste0(deparse(formula), collapse = "")
+
+ if (is.null(weightCol)) {
+ weightCol <- ""
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.LogisticRegressionWrapper", "fit",
+ data@sdf, formula, as.numeric(regParam),
+ as.numeric(elasticNetParam), as.integer(maxIter),
+ as.numeric(tol), as.logical(fitIntercept),
+ as.character(family), as.logical(standardization),
+ as.array(thresholds), as.character(weightCol),
+ as.integer(aggregationDepth), as.character(probabilityCol))
+ new("LogisticRegressionModel", jobj = jobj)
+ })
+
+# Predicted values based on an LogisticRegressionModel model
+
+#' @param newData a SparkDataFrame for testing.
+#' @return \code{predict} returns the predicted values based on an LogisticRegressionModel.
+#' @rdname spark.logit
+#' @aliases predict,LogisticRegressionModel,SparkDataFrame-method
+#' @export
+#' @note predict(LogisticRegressionModel) since 2.1.0
+setMethod("predict", signature(object = "LogisticRegressionModel"),
+ function(object, newData) {
+ predict_internal(object, newData)
+ })
+
+# Get the summary of an LogisticRegressionModel
+
+#' @param object an LogisticRegressionModel fitted by \code{spark.logit}
+#' @return \code{summary} returns the Binary Logistic regression results of a given model as lists. Note that
+#' Multinomial logistic regression summary is not available now.
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @export
+#' @note summary(LogisticRegressionModel) since 2.1.0
+setMethod("summary", signature(object = "LogisticRegressionModel"),
+ function(object) {
+ jobj <- object@jobj
+ is.loaded <- callJMethod(jobj, "isLoaded")
+
+ if (is.loaded) {
+ stop("Loaded model doesn't have training summary.")
+ }
+
+ roc <- dataFrame(callJMethod(jobj, "roc"))
+
+ areaUnderROC <- callJMethod(jobj, "areaUnderROC")
+
+ pr <- dataFrame(callJMethod(jobj, "pr"))
+
+ fMeasureByThreshold <-
+ dataFrame(callJMethod(jobj, "fMeasureByThreshold"))
--- End diff --
are there reasons these are all in 2 lines?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/15365
LGTM.
Let's see if anyone has any other comments.
Could you open a JIRA on Vector/SparseVector/DenseVector?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84999720
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,173 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' \itemize{
+#' \item{"auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".}
+#' \item{"binomial": Binary logistic regression with pivoting.}
+#' \item{"multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".}
+#' }
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param thresholds in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds c(1-p, p). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. In multiclass (or binary) classification to adjust the probability of
+#' predicting each class. Array must have length equal to the number of classes, with values > 0,
+#' excepting that at most one value may be 0. The class with largest value p/t is predicted, where p
+#' is the original probability of that class and t is the class's threshold. Note: When thresholds
+#' is set, any user-set value for threshold will be cleared. If both threshold and thresholds are
+#' set, then they must be equivalent. Default is 0.5.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param probabilityCol column name for predicted class conditional probabilities. Default is "probability".
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
+#' blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+#'
+#' # summary of binary logistic regression
+#' blr_summary <- summary(blr_model)
+#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold, "threshold", "F-Measure"))
+#' # save fitted model to input path
+#' path <- "path/to/model"
+#' write.ml(blr_model, path)
+#'
+#' # can also read back the saved model and print
+#' savedModel <- read.ml(path)
+#' summary(savedModel)
+#'
+#' # multinomial logistic regression
+#'
+#' label <- c(0.0, 1.0, 2.0, 0.0, 0.0)
+#' feature1 <- c(4.845940, 5.64480, 7.430381, 6.464263, 5.555667)
+#' feature2 <- c(2.941319, 2.614812, 2.162451, 3.339474, 2.970987)
+#' feature3 <- c(1.322733, 1.348044, 3.861237, 9.686976, 3.447130)
+#' feature4 <- c(1.3246388, 0.5510444, 0.9225810, 1.2147881, 1.6020842)
+#' data <- as.data.frame(cbind(label, feature1, feature2, feature3, feature4))
+#' df <- createDataFrame(data)
+#'
+#' model <- spark.logit(df, label ~ ., family = "multinomial", thresholds=c(0, 1, 1))
+#' predict1 <- collect(select(predict(model, df), "prediction"))
+#' }
+#' @note spark.logit since 2.1.0
+setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula"),
+ function(data, formula, regParam = 0.0, elasticNetParam = 0.0, maxIter = 100,
+ tol = 1E-6, fitIntercept = TRUE, family = "auto", standardization = TRUE,
+ thresholds = 0.5, weightCol = NULL, aggregationDepth = 2,
+ probabilityCol = "probability") {
+ formula <- paste0(deparse(formula), collapse = "")
+
+ if (is.null(weightCol)) {
+ weightCol <- ""
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.LogisticRegressionWrapper", "fit",
+ data@sdf, formula, as.numeric(regParam),
+ as.numeric(elasticNetParam), as.integer(maxIter),
+ as.numeric(tol), as.logical(fitIntercept),
+ as.character(family), as.logical(standardization),
+ as.array(thresholds), as.character(weightCol),
+ as.integer(aggregationDepth), as.character(probabilityCol))
+ new("LogisticRegressionModel", jobj = jobj)
+ })
+
+# Predicted values based on an LogisticRegressionModel model
+
+#' @param newData a SparkDataFrame for testing.
+#' @return \code{predict} returns the predicted values based on an LogisticRegressionModel.
+#' @rdname spark.logit
+#' @aliases predict,LogisticRegressionModel,SparkDataFrame-method
+#' @export
+#' @note predict(LogisticRegressionModel) since 2.1.0
+setMethod("predict", signature(object = "LogisticRegressionModel"),
+ function(object, newData) {
+ predict_internal(object, newData)
+ })
+
+# Get the summary of an LogisticRegressionModel
+
+#' @param object an LogisticRegressionModel fitted by \code{spark.logit}
+#' @return \code{summary} returns the Binary Logistic regression results of a given model as lists. Note that
+#' Multinomial logistic regression summary is not available now.
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
--- End diff --
this is the `summary` method, not `spark.logit` method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
cc @sethah @yanboliang
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66405/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #67526 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67526/consoleFull)** for PR 15365 at commit [`d0452ae`](https://github.com/apache/spark/commit/d0452ae1f92b9d72a620942be269f965c5810392).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by vectorijk <gi...@git.apache.org>.
Github user vectorijk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82497112
--- Diff: R/pkg/R/mllib.R ---
@@ -117,7 +124,7 @@ NULL
#' @export
#' @seealso \link{spark.glm}, \link{glm},
#' @seealso \link{spark.als}, \link{spark.gaussianMixture}, \link{spark.isoreg}, \link{spark.kmeans},
-#' @seealso \link{spark.mlp}, \link{spark.naiveBayes}, \link{spark.survreg}
+#' @seealso \link{spark.mlp}, \link{spark.naiveBayes}, \link{spark.survreg}, \link{spark.logit}
--- End diff --
same here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82448237
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
+#' value may be 0. The class with largest value p/t is predicted, where p is the original probability
+#' of that class and t is the class's threshold. Note: When thresholds is set, any user-set
+#' value for threshold will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is NULL.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
--- End diff --
why is `suppressWarnings` needed here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r83122799
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
+#' value may be 0. The class with largest value p/t is predicted, where p is the original probability
+#' of that class and t is the class's threshold. Note: When thresholds is set, any user-set
+#' value for threshold will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is NULL.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
+#' blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+#'
+#' # summary of binary logistic regression
+#' blr_summary <- summary(blr_model)
+#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold, "threshold", "F-Measure"))
+#' # save fitted model to input path
+#' path <- "path/to/model"
+#' write.ml(blr_model, path)
+#'
+#' # can also read back the saved model and print
+#' savedModel <- read.ml(path)
+#' summary(savedModel)
+#'
+#' # multinomial logistic regression
+#'
+#' label <- c(0.0, 1.0, 2.0, 0.0, 0.0)
+#' feature1 <- c(4.845940, 5.64480, 7.430381, 6.464263, 5.555667)
+#' feature2 <- c(2.941319, 2.614812, 2.162451, 3.339474, 2.970987)
+#' feature3 <- c(1.322733, 1.348044, 3.861237, 9.686976, 3.447130)
+#' feature4 <- c(1.3246388, 0.5510444, 0.9225810, 1.2147881, 1.6020842)
+#' data <- as.data.frame(cbind(label, feature1, feature2, feature3, feature4))
+#' df <- suppressWarnings(createDataFrame(data))
+#'
+#' model <- spark.logit(df, label ~ ., family = "multinomial", thresholds=c(0, 1, 1))
+#' predict1 <- collect(select(predict(model, df), "prediction"))
+#' }
+#' @note spark.logit since 2.1.0
+setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula"),
+ function(data, formula, regParam = 0.0, elasticNetParam = 0.0, maxIter = 100,
+ tol = 1E-6, fitIntercept = TRUE, family = "auto", standardization = TRUE,
+ threshold = 0.5, thresholds = NULL, weightCol = NULL, aggregationDepth = 2) {
--- End diff --
perhaps turning that into a list/array on the Scala side so it becomes list on the R side?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84832837
--- Diff: R/pkg/inst/tests/testthat/test_mllib.R ---
@@ -587,6 +587,59 @@ test_that("spark.isotonicRegression", {
unlink(modelPath)
})
+test_that("spark.logit", {
+ # test binary logistic regression
+ label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+ feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+ binary_data <- as.data.frame(cbind(label, feature))
+ binary_df <- createDataFrame(binary_data)
+
+ blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
--- End diff --
I would update the example to use `thresholds` instead
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66468 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66468/consoleFull)** for PR 15365 at commit [`0811fc3`](https://github.com/apache/spark/commit/0811fc3c71b665b99aec1e794d3782c98563e84c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r83143464
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
+#' value may be 0. The class with largest value p/t is predicted, where p is the original probability
+#' of that class and t is the class's threshold. Note: When thresholds is set, any user-set
+#' value for threshold will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is NULL.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
+#' blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+#'
+#' # summary of binary logistic regression
+#' blr_summary <- summary(blr_model)
+#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold, "threshold", "F-Measure"))
+#' # save fitted model to input path
+#' path <- "path/to/model"
+#' write.ml(blr_model, path)
+#'
+#' # can also read back the saved model and print
+#' savedModel <- read.ml(path)
+#' summary(savedModel)
+#'
+#' # multinomial logistic regression
+#'
+#' label <- c(0.0, 1.0, 2.0, 0.0, 0.0)
+#' feature1 <- c(4.845940, 5.64480, 7.430381, 6.464263, 5.555667)
+#' feature2 <- c(2.941319, 2.614812, 2.162451, 3.339474, 2.970987)
+#' feature3 <- c(1.322733, 1.348044, 3.861237, 9.686976, 3.447130)
+#' feature4 <- c(1.3246388, 0.5510444, 0.9225810, 1.2147881, 1.6020842)
+#' data <- as.data.frame(cbind(label, feature1, feature2, feature3, feature4))
+#' df <- suppressWarnings(createDataFrame(data))
+#'
+#' model <- spark.logit(df, label ~ ., family = "multinomial", thresholds=c(0, 1, 1))
+#' predict1 <- collect(select(predict(model, df), "prediction"))
+#' }
+#' @note spark.logit since 2.1.0
+setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula"),
+ function(data, formula, regParam = 0.0, elasticNetParam = 0.0, maxIter = 100,
+ tol = 1E-6, fitIntercept = TRUE, family = "auto", standardization = TRUE,
+ threshold = 0.5, thresholds = NULL, weightCol = NULL, aggregationDepth = 2) {
--- End diff --
Let me check whether it is feasible to do that, because the trait is shared by other algorithms.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
I just pick the name for simplicity. Hope to receive feedback from the community and I can make changes accordingly. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
@vectorijk Thanks for your information! I installed e1071 and installed tex package. I just want to find what causes the error.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84833023
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,173 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' \itemize{
+#' \item{"auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
--- End diff --
is there supposed to be a `numClasses` parameter?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/15365
re: https://github.com/apache/spark/pull/15365#discussion_r84387432
This is definitely an issue as Vector/SparseVector/DenseVector is mapped to environment in R, and within the DataFrame, environment is hard to operate with.
It might likely be how serde/type mapping is handling this but from my digging so far I haven't pinpointed where and how we fix this. It could be separated into another JIRA though.
I'll look at the rest of this PR today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #67293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67293/consoleFull)** for PR 15365 at commit [`a222de7`](https://github.com/apache/spark/commit/a222de7efe93aea51f9de755f4fd29eb4ff8554a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66405/consoleFull)** for PR 15365 at commit [`5bfd132`](https://github.com/apache/spark/commit/5bfd1320000791bed933b10dc3af8f0857d6be99).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
re-test please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67526/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/15365
Sure- I recall that discussion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66857/consoleFull)** for PR 15365 at commit [`08babe5`](https://github.com/apache/spark/commit/08babe50df42bc4247a65f24cc95241891b555a8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r83121852
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
--- End diff --
I don't quite understand this comment. There is a JIRA SPARK-11543 discussing the expected relationship, but it is not implemented yet. So, we keep both threshold and thresholds on Scala side.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
I see the error message on local test:
LaTeX errors when creating PDF version.
This typically indicates Rd problems.
* checking PDF version of manual without hyperrefs or index ... ERROR
Re-running with no redirection of stdout/stderr.
Hmm ... looks like a package
Error in texi2dvi(file = file, pdf = TRUE, clean = clean, quiet = quiet, :
pdflatex is not available
Error in texi2dvi(file = file, pdf = TRUE, clean = clean, quiet = quiet, :
pdflatex is not available
Error in running tools::texi2pdf()
You may want to clean up by 'rm -rf /var/folders/s_/83b0sgvj2kl2kwq4stvft_pm0000gn/T//RtmpXHJrOk/Rd2pdfac8961d3ab54'
* DONE
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #67293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67293/consoleFull)** for PR 15365 at commit [`a222de7`](https://github.com/apache/spark/commit/a222de7efe93aea51f9de755f4fd29eb4ff8554a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by vectorijk <gi...@git.apache.org>.
Github user vectorijk commented on the issue:
https://github.com/apache/spark/pull/15365
@wangmiao1981 I saw the similar error on Jekin. Same with question with you.
Regarding to `e1071`, I think we only need to install that package locally.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
@felixcheung I fixed the cran errors. It is ready to review now. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #67526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67526/consoleFull)** for PR 15365 at commit [`d0452ae`](https://github.com/apache/spark/commit/d0452ae1f92b9d72a620942be269f965c5810392).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
Thanks for your response! I will do some research on how to bring Vectors back at spare time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
ping @felixcheung
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66407 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66407/consoleFull)** for PR 15365 at commit [`0f4f551`](https://github.com/apache/spark/commit/0f4f5510e595459b7eb399f5d32d8a9a9304ba84).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82529069
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
+#' value may be 0. The class with largest value p/t is predicted, where p is the original probability
+#' of that class and t is the class's threshold. Note: When thresholds is set, any user-set
+#' value for threshold will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is NULL.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
+#' blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+#'
+#' # summary of binary logistic regression
+#' blr_summary <- summary(blr_model)
+#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold, "threshold", "F-Measure"))
+#' # save fitted model to input path
+#' path <- "path/to/model"
+#' write.ml(blr_model, path)
+#'
+#' # can also read back the saved model and print
+#' savedModel <- read.ml(path)
+#' summary(savedModel)
+#'
+#' # multinomial logistic regression
+#'
+#' label <- c(0.0, 1.0, 2.0, 0.0, 0.0)
+#' feature1 <- c(4.845940, 5.64480, 7.430381, 6.464263, 5.555667)
+#' feature2 <- c(2.941319, 2.614812, 2.162451, 3.339474, 2.970987)
+#' feature3 <- c(1.322733, 1.348044, 3.861237, 9.686976, 3.447130)
+#' feature4 <- c(1.3246388, 0.5510444, 0.9225810, 1.2147881, 1.6020842)
+#' data <- as.data.frame(cbind(label, feature1, feature2, feature3, feature4))
+#' df <- suppressWarnings(createDataFrame(data))
+#'
+#' model <- spark.logit(df, label ~ ., family = "multinomial", thresholds=c(0, 1, 1))
+#' predict1 <- collect(select(predict(model, df), "prediction"))
+#' }
+#' @note spark.logit since 2.1.0
+setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula"),
+ function(data, formula, regParam = 0.0, elasticNetParam = 0.0, maxIter = 100,
+ tol = 1E-6, fitIntercept = TRUE, family = "auto", standardization = TRUE,
+ threshold = 0.5, thresholds = NULL, weightCol = NULL, aggregationDepth = 2) {
+ formula <- paste0(deparse(formula), collapse = "")
+
+ if (is.null(weightCol)) {
+ weightCol <- ""
+ }
+
+ if (!is.null(thresholds)) {
+ thresholds <- as.array(thresholds)
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.LogisticRegressionWrapper", "fit",
+ data@sdf, formula, as.numeric(regParam),
+ as.numeric(elasticNetParam), as.integer(maxIter),
+ as.numeric(tol), as.logical(fitIntercept),
+ as.character(family), as.logical(standardization),
+ as.numeric(threshold), thresholds,
+ as.character(weightCol), as.integer(aggregationDepth))
+ new("LogisticRegressionModel", jobj = jobj)
+ })
+
+# Predicted values based on an LogisticRegressionModel model
+
+#' @param newData a SparkDataFrame for testing.
+#' @return \code{predict} returns the predicted values based on an LogisticRegressionModel.
+#' @rdname spark.logit
--- End diff --
add `@aliases`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #67531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67531/consoleFull)** for PR 15365 at commit [`031cf9b`](https://github.com/apache/spark/commit/031cf9b8efa104de0bc288b5833ffc03b2fe7097).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r85001451
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,173 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' \itemize{
+#' \item{"auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".}
+#' \item{"binomial": Binary logistic regression with pivoting.}
+#' \item{"multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".}
+#' }
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param thresholds in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds c(1-p, p). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. In multiclass (or binary) classification to adjust the probability of
+#' predicting each class. Array must have length equal to the number of classes, with values > 0,
+#' excepting that at most one value may be 0. The class with largest value p/t is predicted, where p
+#' is the original probability of that class and t is the class's threshold. Note: When thresholds
+#' is set, any user-set value for threshold will be cleared. If both threshold and thresholds are
+#' set, then they must be equivalent. Default is 0.5.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param probabilityCol column name for predicted class conditional probabilities. Default is "probability".
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
+#' blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+#'
+#' # summary of binary logistic regression
+#' blr_summary <- summary(blr_model)
+#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold, "threshold", "F-Measure"))
+#' # save fitted model to input path
+#' path <- "path/to/model"
+#' write.ml(blr_model, path)
+#'
+#' # can also read back the saved model and print
+#' savedModel <- read.ml(path)
+#' summary(savedModel)
+#'
+#' # multinomial logistic regression
+#'
+#' label <- c(0.0, 1.0, 2.0, 0.0, 0.0)
+#' feature1 <- c(4.845940, 5.64480, 7.430381, 6.464263, 5.555667)
+#' feature2 <- c(2.941319, 2.614812, 2.162451, 3.339474, 2.970987)
+#' feature3 <- c(1.322733, 1.348044, 3.861237, 9.686976, 3.447130)
+#' feature4 <- c(1.3246388, 0.5510444, 0.9225810, 1.2147881, 1.6020842)
+#' data <- as.data.frame(cbind(label, feature1, feature2, feature3, feature4))
+#' df <- createDataFrame(data)
+#'
+#' model <- spark.logit(df, label ~ ., family = "multinomial", thresholds=c(0, 1, 1))
+#' predict1 <- collect(select(predict(model, df), "prediction"))
+#' }
+#' @note spark.logit since 2.1.0
+setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula"),
+ function(data, formula, regParam = 0.0, elasticNetParam = 0.0, maxIter = 100,
+ tol = 1E-6, fitIntercept = TRUE, family = "auto", standardization = TRUE,
+ thresholds = 0.5, weightCol = NULL, aggregationDepth = 2,
+ probabilityCol = "probability") {
+ formula <- paste0(deparse(formula), collapse = "")
+
+ if (is.null(weightCol)) {
+ weightCol <- ""
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.LogisticRegressionWrapper", "fit",
+ data@sdf, formula, as.numeric(regParam),
+ as.numeric(elasticNetParam), as.integer(maxIter),
+ as.numeric(tol), as.logical(fitIntercept),
+ as.character(family), as.logical(standardization),
+ as.array(thresholds), as.character(weightCol),
+ as.integer(aggregationDepth), as.character(probabilityCol))
+ new("LogisticRegressionModel", jobj = jobj)
+ })
+
+# Predicted values based on an LogisticRegressionModel model
+
+#' @param newData a SparkDataFrame for testing.
+#' @return \code{predict} returns the predicted values based on an LogisticRegressionModel.
+#' @rdname spark.logit
+#' @aliases predict,LogisticRegressionModel,SparkDataFrame-method
+#' @export
+#' @note predict(LogisticRegressionModel) since 2.1.0
+setMethod("predict", signature(object = "LogisticRegressionModel"),
+ function(object, newData) {
+ predict_internal(object, newData)
+ })
+
+# Get the summary of an LogisticRegressionModel
+
+#' @param object an LogisticRegressionModel fitted by \code{spark.logit}
+#' @return \code{summary} returns the Binary Logistic regression results of a given model as lists. Note that
+#' Multinomial logistic regression summary is not available now.
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
--- End diff --
Oh. I see. I looked the wrong line number.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84832132
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,173 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
--- End diff --
if this is supposed to be numeric, could you change it to 0.0 or 1.0 consistently throughout this comment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84833158
--- Diff: mllib/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala ---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.r
+
+import org.apache.hadoop.fs.Path
+import org.json4s._
+import org.json4s.JsonDSL._
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.ml.{Pipeline, PipelineModel}
+import org.apache.spark.ml.attribute.AttributeGroup
+import org.apache.spark.ml.classification.{BinaryLogisticRegressionSummary, LogisticRegression, LogisticRegressionModel}
+import org.apache.spark.ml.feature.RFormula
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.{DataFrame, Dataset}
+
+private[r] class LogisticRegressionWrapper private (
+ val pipeline: PipelineModel,
+ val features: Array[String],
+ val isLoaded: Boolean = false) extends MLWritable {
+
+ private val logisticRegressionModel: LogisticRegressionModel =
+ pipeline.stages(1).asInstanceOf[LogisticRegressionModel]
+
+ lazy val totalIterations: Int = logisticRegressionModel.summary.totalIterations
+
+ lazy val objectiveHistory: Array[Double] = logisticRegressionModel.summary.objectiveHistory
+
+ lazy val roc: DataFrame =
+ logisticRegressionModel.summary.asInstanceOf[BinaryLogisticRegressionSummary].roc
--- End diff --
consider another `val` for `logisticRegressionModel.summary.asInstanceOf[BinaryLogisticRegressionSummary]` to avoid duplication
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
My local tests passed.
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82481714
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
+#' value may be 0. The class with largest value p/t is predicted, where p is the original probability
+#' of that class and t is the class's threshold. Note: When thresholds is set, any user-set
+#' value for threshold will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is NULL.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
--- End diff --
I followed other test examples. If it is not necessary, I can remove all of them as a followup JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
For the name, as we previously discussed, we can't use glm as interface changes and glm only support binominal logistic regression. We don't use glmnet because current spark.logit only provides logistic regressions which are subset of glmnet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84384123
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82528944
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
--- End diff --
do you have a more R-like representation for `(Array(1-p, p))`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66405/consoleFull)** for PR 15365 at commit [`5bfd132`](https://github.com/apache/spark/commit/5bfd1320000791bed933b10dc3af8f0857d6be99).
* This patch **fails R style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
From Jekins, I saw the error:
WARNING: There was 1 warning.
NOTE: There were 3 notes.
See
'/home/jenkins/workspace/SparkPullRequestBuilder/R/SparkR.Rcheck/00check.log'
for details.
Had CRAN check errors; see logs.
How can I access the above file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66428/consoleFull)** for PR 15365 at commit [`a172bb4`](https://github.com/apache/spark/commit/a172bb4a5ff1ce0f6c5d1bd79c74dc8abc7a14a8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82529145
--- Diff: mllib/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala ---
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.r
+
+import org.apache.hadoop.fs.Path
+import org.json4s._
+import org.json4s.JsonDSL._
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.ml.{Pipeline, PipelineModel}
+import org.apache.spark.ml.attribute.AttributeGroup
+import org.apache.spark.ml.classification.{BinaryLogisticRegressionSummary, LogisticRegression, LogisticRegressionModel}
+import org.apache.spark.ml.feature.RFormula
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.{DataFrame, Dataset}
+
+private[r] class LogisticRegressionWrapper private (
+ val pipeline: PipelineModel,
+ val features: Array[String],
+ val isLoaded: Boolean = false) extends MLWritable {
+
+ private val logisticRegressionModel: LogisticRegressionModel =
+ pipeline.stages(1).asInstanceOf[LogisticRegressionModel]
+
+ lazy val totalIterations: Int = logisticRegressionModel.summary.totalIterations
+
+ lazy val objectiveHistory: Array[Double] = logisticRegressionModel.summary.objectiveHistory
+
+ lazy val roc: DataFrame =
+ logisticRegressionModel.summary.asInstanceOf[BinaryLogisticRegressionSummary].roc
+
+ lazy val areaUnderROC: Double =
+ logisticRegressionModel.summary.asInstanceOf[BinaryLogisticRegressionSummary].areaUnderROC
+
+ lazy val pr: DataFrame =
+ logisticRegressionModel.summary.asInstanceOf[BinaryLogisticRegressionSummary].pr
+
+ lazy val fMeasureByThreshold: DataFrame =
+ logisticRegressionModel.summary.asInstanceOf[BinaryLogisticRegressionSummary]
+ .fMeasureByThreshold
+
+ lazy val precisionByThreshold: DataFrame =
+ logisticRegressionModel.summary.asInstanceOf[BinaryLogisticRegressionSummary]
+ .precisionByThreshold
+
+ lazy val recallByThreshold: DataFrame =
+ logisticRegressionModel.summary.asInstanceOf[BinaryLogisticRegressionSummary]
+ .recallByThreshold
+
+ def transform(dataset: Dataset[_]): DataFrame = {
+ pipeline.transform(dataset).drop(logisticRegressionModel.getFeaturesCol)
+ }
+
+ override def write: MLWriter = new LogisticRegressionWrapper.LogisticRegressionWrapperWriter(this)
+}
+
+private[r] object LogisticRegressionWrapper
+ extends MLReadable[LogisticRegressionWrapper] {
+
+ def fit( // scalastyle:ignore
+ data: DataFrame,
+ formula: String,
+ regParam: Double,
+ elasticNetParam: Double,
+ maxIter: Int,
+ tol: Double,
+ fitIntercept: Boolean,
+ family: String,
+ standardization: Boolean,
+ threshold: Double,
+ thresholds: Array[Double],
+ weightCol: String,
+ aggregationDepth: Int
+ ): LogisticRegressionWrapper = {
+
+ val rFormula = new RFormula()
+ .setFormula(formula)
+ .setFeaturesCol("features")
--- End diff --
shouldn't set feature col - see checkDataColumns
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r83121575
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
--- End diff --
Modified to `c(p, 1-p)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by vectorijk <gi...@git.apache.org>.
Github user vectorijk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82497107
--- Diff: R/pkg/R/mllib.R ---
@@ -105,7 +112,7 @@ setClass("KSTest", representation(jobj = "jobj"))
#' @seealso \link{spark.glm}, \link{glm},
#' @seealso \link{spark.als}, \link{spark.gaussianMixture}, \link{spark.isoreg}, \link{spark.kmeans},
#' @seealso \link{spark.lda}, \link{spark.mlp}, \link{spark.naiveBayes}, \link{spark.survreg}
-#' @seealso \link{read.ml}
+#' @seealso \link{spark.logit}, \link{read.ml}
--- End diff --
this group of links could be sorted
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66428/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66563/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/15365
It would be great to get some feedback on the name `spark.logit`
What do folks think about it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #67531 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67531/consoleFull)** for PR 15365 at commit [`031cf9b`](https://github.com/apache/spark/commit/031cf9b8efa104de0bc288b5833ffc03b2fe7097).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66468 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66468/consoleFull)** for PR 15365 at commit [`0811fc3`](https://github.com/apache/spark/commit/0811fc3c71b665b99aec1e794d3782c98563e84c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84988891
--- Diff: mllib/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala ---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.r
+
+import org.apache.hadoop.fs.Path
+import org.json4s._
+import org.json4s.JsonDSL._
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.ml.{Pipeline, PipelineModel}
+import org.apache.spark.ml.attribute.AttributeGroup
+import org.apache.spark.ml.classification.{BinaryLogisticRegressionSummary, LogisticRegression, LogisticRegressionModel}
+import org.apache.spark.ml.feature.RFormula
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.{DataFrame, Dataset}
+
+private[r] class LogisticRegressionWrapper private (
+ val pipeline: PipelineModel,
+ val features: Array[String],
+ val isLoaded: Boolean = false) extends MLWritable {
+
+ private val logisticRegressionModel: LogisticRegressionModel =
+ pipeline.stages(1).asInstanceOf[LogisticRegressionModel]
+
+ lazy val totalIterations: Int = logisticRegressionModel.summary.totalIterations
+
+ lazy val objectiveHistory: Array[Double] = logisticRegressionModel.summary.objectiveHistory
+
+ lazy val roc: DataFrame =
+ logisticRegressionModel.summary.asInstanceOf[BinaryLogisticRegressionSummary].roc
--- End diff --
Add a new `val blrSummary`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66407 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66407/consoleFull)** for PR 15365 at commit [`0f4f551`](https://github.com/apache/spark/commit/0f4f5510e595459b7eb399f5d32d8a9a9304ba84).
* This patch **fails R style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/15365
@felixcheung When run check-cran, there are errors:
Error : requireNamespace("e1071", quietly = TRUE) is not TRUE
Error : requireNamespace("e1071", quietly = TRUE) is not TRUE
Error : requireNamespace("e1071", quietly = TRUE) is not TRUE
Error : requireNamespace("e1071", quietly = TRUE) is not TRUE
Error : requireNamespace("e1071", quietly = TRUE) is not TRUE
I am trying to figure out what is the problem. Any hints?
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84988793
--- Diff: R/pkg/inst/tests/testthat/test_mllib.R ---
@@ -587,6 +587,59 @@ test_that("spark.isotonicRegression", {
unlink(modelPath)
})
+test_that("spark.logit", {
+ # test binary logistic regression
+ label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+ feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+ binary_data <- as.data.frame(cbind(label, feature))
+ binary_df <- createDataFrame(binary_data)
+
+ blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
--- End diff --
Updated tests and example.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84387432
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
+#' value may be 0. The class with largest value p/t is predicted, where p is the original probability
+#' of that class and t is the class's threshold. Note: When thresholds is set, any user-set
+#' value for threshold will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is NULL.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
+#' blr_model <- spark.logit(binary_df, label ~ feature, threshold = 1.0)
+#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
+#'
+#' # summary of binary logistic regression
+#' blr_summary <- summary(blr_model)
+#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold, "threshold", "F-Measure"))
+#' # save fitted model to input path
+#' path <- "path/to/model"
+#' write.ml(blr_model, path)
+#'
+#' # can also read back the saved model and print
+#' savedModel <- read.ml(path)
+#' summary(savedModel)
+#'
+#' # multinomial logistic regression
+#'
+#' label <- c(0.0, 1.0, 2.0, 0.0, 0.0)
+#' feature1 <- c(4.845940, 5.64480, 7.430381, 6.464263, 5.555667)
+#' feature2 <- c(2.941319, 2.614812, 2.162451, 3.339474, 2.970987)
+#' feature3 <- c(1.322733, 1.348044, 3.861237, 9.686976, 3.447130)
+#' feature4 <- c(1.3246388, 0.5510444, 0.9225810, 1.2147881, 1.6020842)
+#' data <- as.data.frame(cbind(label, feature1, feature2, feature3, feature4))
+#' df <- suppressWarnings(createDataFrame(data))
+#'
+#' model <- spark.logit(df, label ~ ., family = "multinomial", thresholds=c(0, 1, 1))
+#' predict1 <- collect(select(predict(model, df), "prediction"))
+#' }
+#' @note spark.logit since 2.1.0
+setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula"),
+ function(data, formula, regParam = 0.0, elasticNetParam = 0.0, maxIter = 100,
+ tol = 1E-6, fitIntercept = TRUE, family = "auto", standardization = TRUE,
+ threshold = 0.5, thresholds = NULL, weightCol = NULL, aggregationDepth = 2) {
--- End diff --
@felixcheung I don't find a native method to convert the `Vector` to `Array` when using select on R side.
Optionally, I can add a method `getProbability` in scala wrapper, which converts the Rows of Vectors to a dataframe. Then, on R side, we can call that method in `summary` method. At the same time, we remove the `probabilityCol` from the function definition.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15365
**[Test build #66428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66428/consoleFull)** for PR 15365 at commit [`a172bb4`](https://github.com/apache/spark/commit/a172bb4a5ff1ce0f6c5d1bd79c74dc8abc7a14a8).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r82528698
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
--- End diff --
this might be confusing, in particular with parameter matching in R, should this be `thresholds` when length == 1 for binary and > 1 for multiclass?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66857/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15365: [SPARK-17157][SPARKR]: Add multiclass logistic re...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15365#discussion_r84832359
--- Diff: R/pkg/R/mllib.R ---
@@ -647,6 +654,195 @@ setMethod("predict", signature(object = "KMeansModel"),
predict_internal(object, newData)
})
+#' Logistic Regression Model
+#'
+#' Fits an logistic regression model against a Spark DataFrame. It supports "binomial": Binary logistic regression
+#' with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet.
+#' Users can print, make predictions on the produced model and save the model to the input path.
+#'
+#' @param data SparkDataFrame for training
+#' @param formula A symbolic description of the model to be fitted. Currently only a few formula
+#' operators are supported, including '~', '.', ':', '+', and '-'.
+#' @param regParam the regularization parameter. Default is 0.0.
+#' @param elasticNetParam the ElasticNet mixing parameter. For alpha = 0, the penalty is an L2 penalty.
+#' For alpha = 1, it is an L1 penalty. For 0 < alpha < 1, the penalty is a combination
+#' of L1 and L2. Default is 0.0 which is an L2 penalty.
+#' @param maxIter maximum iteration number.
+#' @param tol convergence tolerance of iterations.
+#' @param fitIntercept whether to fit an intercept term. Default is TRUE.
+#' @param family the name of family which is a description of the label distribution to be used in the model.
+#' Supported options:
+#' - "auto": Automatically select the family based on the number of classes:
+#' If numClasses == 1 || numClasses == 2, set to "binomial".
+#' Else, set to "multinomial".
+#' - "binomial": Binary logistic regression with pivoting.
+#' - "multinomial": Multinomial logistic (softmax) regression without pivoting.
+#' Default is "auto".
+#' @param standardization whether to standardize the training features before fitting the model. The coefficients
+#' of models will be always returned on the original scale, so it will be transparent for
+#' users. Note that with/without standardization, the models should be always converged
+#' to the same solution when no regularization is applied. Default is TRUE, same as glmnet.
+#' @param threshold in binary classification, in range [0, 1]. If the estimated probability of class label 1
+#' is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0
+#' more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with
+#' threshold p is equivalent to setting thresholds (Array(1-p, p)). When threshold is set, any user-set
+#' value for thresholds will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is 0.5.
+#' @param thresholds in multiclass (or binary) classification to adjust the probability of predicting each class.
+#' Array must have length equal to the number of classes, with values > 0, excepting that at most one
+#' value may be 0. The class with largest value p/t is predicted, where p is the original probability
+#' of that class and t is the class's threshold. Note: When thresholds is set, any user-set
+#' value for threshold will be cleared. If both threshold and thresholds are set, then they must be
+#' equivalent. Default is NULL.
+#' @param weightCol The weight column name.
+#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions of features or the number of partitions
+#' are large, this param could be adjusted to a larger size. Default is 2.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.logit} returns a fitted logistic regression model
+#' @rdname spark.logit
+#' @aliases spark.logit,SparkDataFrame,formula-method
+#' @name spark.logit
+#' @export
+#' @examples
+#' \dontrun{
+#' sparkR.session()
+#' # binary logistic regression
+#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
+#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, feature))
+#' binary_df <- suppressWarnings(createDataFrame(binary_data))
--- End diff --
suppressWarnings still here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org