You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yanboliang <gi...@git.apache.org> on 2016/01/07 10:42:58 UTC
[GitHub] spark pull request: [SPARK-9835] [ML] Initial draft of Iteratively...
GitHub user yanboliang opened a pull request:
https://github.com/apache/spark/pull/10639
[SPARK-9835] [ML] Initial draft of IterativelyReweightedLeastSquares
Initial draft of ```IterativelyReweightedLeastSquares```.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yanboliang/spark spark-9835
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10639.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10639
----
commit 86492bb707635a2da904e9d4ffb2cd7e2257d533
Author: Yanbo Liang <yb...@gmail.com>
Date: 2016-01-07T09:41:59Z
Initial draft of IterativelyReweightedLeastSquares
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170497846
**[Test build #49126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49126/consoleFull)** for PR 10639 at commit [`65133ad`](https://github.com/apache/spark/commit/65133add9fdc2a4612fe481954b951eaac40a1a7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50838729
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and other optimization problems.
+ *
+ * @param initialModel the initial guess model.
+ * @param reweightFunc the reweight function which is used to update offsets and weights
+ * at each iteration.
+ * @param fitIntercept whether to fit intercept.
+ * @param regParam L2 regularization parameter used by WLS.
+ * @param maxIter maximum number of iterations.
+ * @param tol the convergence tolerance.
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val initialModel: WeightedLeastSquaresModel,
+ val reweightFunc: (Instance, WeightedLeastSquaresModel) => (Double, Double),
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ var converged = false
+ var iter = 0
+
+ var offsetsAndWeights: RDD[(Double, Double)] = null
+ var model: WeightedLeastSquaresModel = initialModel
+ var oldModel: WeightedLeastSquaresModel = initialModel
+
+ while (iter < maxIter && !converged) {
+
+ oldModel = model
+
+ // Update offsets and weights using reweightFunc
+ offsetsAndWeights = instances.map { instance => reweightFunc(instance, oldModel) }
+
+ // Estimate new model
+ val newInstances = instances.zip(offsetsAndWeights).map {
+ case (instance, (offset, weight)) => Instance(offset, weight, instance.features)
+ }
+ model = new WeightedLeastSquares(fitIntercept, regParam, false, false).fit(newInstances)
+
+ val oldParameters = Array.concat(Array(oldModel.intercept), oldModel.coefficients.toArray)
+ val parameters = Array.concat(Array(model.intercept), model.coefficients.toArray)
+ val deltaArray = oldParameters.zip(parameters).map { case (x: Double, y: Double) =>
+ math.abs(x - y)
+ }
+ if (!deltaArray.exists(_ > tol)) {
--- End diff --
I found R [```l1linreg```](https://github.com/cran/pracma/blob/master/R/l1linreg.R) use infinity norm (Chebyshev distance between the model vectors) as the tolerance. R GLM use ```deviance``` as the tolerance which is family related, that means we need pass another argument.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169627791
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169951469
**[Test build #49016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49016/consoleFull)** for PR 10639 at commit [`5b26d2a`](https://github.com/apache/spark/commit/5b26d2a2f01916eeebdf81da3fe4fbbcaf3cfe20).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-173327510
@yanboliang The proposal looks okay to me. Some minor comments:
1. `WeightedLeastSquares` model contains extra content that we don't need in IRLS iterations, e.g., `diagInvAWA`. This could be a follow-up work because the API is currently private.
2. `reweightedFunction` -> `reweightFunc`. Does it need to operate on RDDs? We can simplify the API to `(Instance, Model) => (Double, Double)`.
3. `standardizeFeatures` and `standardizeLabel` only affect the regularization term. We don't need it to match `glm` in R. So let's remove them from the API. Turning them on would change the convergence of IRLS algorithm, which needs further discussion.
QR is more stable than the normal equation solver for WLS. We should switch to it in the future. For general Lp regression, there is an interior-point method implemented in R's `quantreg` package.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-173545758
**[Test build #49875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49875/consoleFull)** for PR 10639 at commit [`2191d2a`](https://github.com/apache/spark/commit/2191d2a8ee1a8def5dc942ce03718826da2f5813).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50801100
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and other optimization problems.
+ *
+ * @param initialModel the initial guess model.
+ * @param reweightFunc the reweight function which is used to update offsets and weights
+ * at each iteration.
+ * @param fitIntercept whether to fit intercept.
+ * @param regParam L2 regularization parameter used by WLS.
+ * @param maxIter maximum number of iterations.
+ * @param tol the convergence tolerance.
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val initialModel: WeightedLeastSquaresModel,
+ val reweightFunc: (Instance, WeightedLeastSquaresModel) => (Double, Double),
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ var converged = false
+ var iter = 0
+
+ var offsetsAndWeights: RDD[(Double, Double)] = null
+ var model: WeightedLeastSquaresModel = initialModel
+ var oldModel: WeightedLeastSquaresModel = initialModel
+
+ while (iter < maxIter && !converged) {
+
+ oldModel = model
+
+ // Update offsets and weights using reweightFunc
+ offsetsAndWeights = instances.map { instance => reweightFunc(instance, oldModel) }
+
+ // Estimate new model
+ val newInstances = instances.zip(offsetsAndWeights).map {
+ case (instance, (offset, weight)) => Instance(offset, weight, instance.features)
+ }
+ model = new WeightedLeastSquares(fitIntercept, regParam, false, false).fit(newInstances)
+
+ val oldParameters = Array.concat(Array(oldModel.intercept), oldModel.coefficients.toArray)
+ val parameters = Array.concat(Array(model.intercept), model.coefficients.toArray)
+ val deltaArray = oldParameters.zip(parameters).map { case (x: Double, y: Double) =>
+ math.abs(x - y)
+ }
--- End diff --
This is inefficient because it allocates several temporary vectors. We can compute `(intercept - oldIntercept)^2 + ||coefficients - oldCoefficients||_2^2` and then take the square root, without allocating new vectors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170939575
**[Test build #49240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49240/consoleFull)** for PR 10639 at commit [`446b1cb`](https://github.com/apache/spark/commit/446b1cb6b7697fdb64aa8a100960cbdc4dfed3d6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169627518
**[Test build #48930 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48930/consoleFull)** for PR 10639 at commit [`86492bb`](https://github.com/apache/spark/commit/86492bb707635a2da904e9d4ffb2cd7e2257d533).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-176036772
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50261/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-173554879
**[Test build #49875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49875/consoleFull)** for PR 10639 at commit [`2191d2a`](https://github.com/apache/spark/commit/2191d2a8ee1a8def5dc942ce03718826da2f5813).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-175046541
@mengxr Thanks for your comments, I will update other issues soon. To the convergence criterion issue, please see my inline comments and I want to here your thoughts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10639
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-173548548
@mengxr Thanks for your comments. For the issue that ```WeightedLeastSquares``` contains extra content such as ```diagInvAWA```, it will be used to generate statistic summary of IRLS or GLM. We can discuss them in the follow-up work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169959350
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170469026
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170479905
**[Test build #49126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49126/consoleFull)** for PR 10639 at commit [`65133ad`](https://github.com/apache/spark/commit/65133add9fdc2a4612fe481954b951eaac40a1a7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r49400552
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.linalg.BLAS._
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage.StorageLevel
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Fits a generalized linear model (GLM) for a given family using
+ * iteratively reweighted least squares (IRLS).
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val family: Family,
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val standardizeFeatures: Boolean,
+ val standardizeLabel: Boolean,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ val y = instances.map(_.label).persist(StorageLevel.MEMORY_AND_DISK)
+ val yMean = y.reduce(_ + _) / y.count()
--- End diff --
Is there a reason to prefer `y.reduce(_ + _) / y.count()` over `y.mean()` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r49144794
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.linalg.BLAS._
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage.StorageLevel
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Fits a generalized linear model (GLM) for a given family using
+ * iteratively reweighted least squares (IRLS).
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val family: Family,
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val standardizeFeatures: Boolean,
+ val standardizeLabel: Boolean,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ val y = instances.map(_.label).persist(StorageLevel.MEMORY_AND_DISK)
+ val yMean = y.reduce(_ + _) / y.count()
+ var mu = y.map { yi => family.startingMu(yi, yMean) }
+ var eta = mu.map { mu => family.link.link(mu) }
--- End diff --
Pre-computing `eta` here seems unnecessary since it is re-assigned within the while loop before it is used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170477998
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169896655
**[Test build #49000 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49000/consoleFull)** for PR 10639 at commit [`6d809d0`](https://github.com/apache/spark/commit/6d809d0c935b42434e5b69f31561d9e307bf191b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169627794
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48930/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170498123
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49126/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170478867
**[Test build #49125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49125/consoleFull)** for PR 10639 at commit [`65133ad`](https://github.com/apache/spark/commit/65133add9fdc2a4612fe481954b951eaac40a1a7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170952075
**[Test build #49240 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49240/consoleFull)** for PR 10639 at commit [`446b1cb`](https://github.com/apache/spark/commit/446b1cb6b7697fdb64aa8a100960cbdc4dfed3d6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170497105
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169896700
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170952311
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r51050739
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and other optimization problems.
+ *
+ * @param initialModel the initial guess model.
+ * @param reweightFunc the reweight function which is used to update offsets and weights
+ * at each iteration.
+ * @param fitIntercept whether to fit intercept.
+ * @param regParam L2 regularization parameter used by WLS.
+ * @param maxIter maximum number of iterations.
+ * @param tol the convergence tolerance.
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val initialModel: WeightedLeastSquaresModel,
+ val reweightFunc: (Instance, WeightedLeastSquaresModel) => (Double, Double),
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ var converged = false
+ var iter = 0
+
+ var offsetsAndWeights: RDD[(Double, Double)] = null
+ var model: WeightedLeastSquaresModel = initialModel
+ var oldModel: WeightedLeastSquaresModel = initialModel
+
+ while (iter < maxIter && !converged) {
+
+ oldModel = model
+
+ // Update offsets and weights using reweightFunc
+ offsetsAndWeights = instances.map { instance => reweightFunc(instance, oldModel) }
+
+ // Estimate new model
+ val newInstances = instances.zip(offsetsAndWeights).map {
+ case (instance, (offset, weight)) => Instance(offset, weight, instance.features)
+ }
+ model = new WeightedLeastSquares(fitIntercept, regParam, false, false).fit(newInstances)
+
+ val oldParameters = Array.concat(Array(oldModel.intercept), oldModel.coefficients.toArray)
+ val parameters = Array.concat(Array(model.intercept), model.coefficients.toArray)
+ val deltaArray = oldParameters.zip(parameters).map { case (x: Double, y: Double) =>
+ math.abs(x - y)
+ }
--- End diff --
Either norm is fine for now. They are more or less equivalent. We can have a separate discussion about it. For this PR, let's just make the computation more efficient.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r49618064
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/GLMFamilies.scala ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.rdd.RDD
+
+/**
+ * A description of the error distribution and link function to be used in the model.
+ * @param link a link function instance
+ */
+private[ml] abstract class Family(val link: Link) extends Serializable {
+
+ /**
+ * Starting value for mu in the IRLS algorithm.
+ */
+ def startingMu(y: Double, yMean: Double): Double = (y + yMean) / 2.0
+
+ /**
+ * Deviance of (y, mu) pair.
+ * Deviance is usually defined as twice the loglikelihood ratio.
+ */
+ def deviance(y: RDD[Double], mu: RDD[Double]): Double
+
+ /** The variance of mu to its mean. */
+ def variance(mu: Double): Double = 1.0
+
+ /** Weights for IRLS steps. */
+ def weights(mu: Double): Double = {
+ 1.0 / (math.pow(this.link.deriv(mu), 2.0) * this.variance(mu))
+ }
+
+ /** The adjusted response variable. */
+ def adjusted(y: Double, mu: Double, eta: Double): Double = {
+ eta + (y - mu) * link.deriv(mu)
+ }
+
+ /** Linear predictors based on given mu. */
+ def predict(mu: Double): Double = this.link.link(mu)
+
+ /** Fitted values based on linear predictors eta. */
+ def fitted(eta: Double): Double = this.link.unlink(eta)
+}
+
+/**
+ * Binomial exponential family distribution.
+ * The default link for the Binomial family is the logit link.
+ * @param link a link function instance
+ */
+private[ml] class Binomial(link: Link = Logit) extends Family(link) {
+
+ override def startingMu(y: Double, yMean: Double): Double = (y + 0.5) / 2.0
+
+ override def deviance(y: RDD[Double], mu: RDD[Double]): Double = {
+ mu.zip(y).map { case (mu, y) =>
+ val my = 1.0 - y
+ y * math.log(math.max(y, 1.0) / mu) +
+ my * math.log(math.max(my, 1.0) / (1.0 - mu))
+ }.sum() * 2
+ }
+
+ override def variance(mu: Double): Double = {
+ mu * (1 - mu)
+ }
+}
+
+private[ml] object Binomial {
+
+ def apply(link: Link): Binomial = new Binomial(link)
+}
+
+/**
+ * Poisson exponential family.
+ * The default link for the Poisson family is the log link.
+ * @param link a link function instance
+ */
+private[ml] class Poisson(link: Link = Log) extends Family(link) {
+
+ override def deviance(y: RDD[Double], mu: RDD[Double]): Double = {
+ mu.zip(y).map { case (mu, y) =>
+ y * math.log(math.max(y, 1.0) / mu) - (y - mu)
--- End diff --
Is there a reference you can point to for this exact form of the deviance equation (other than SparkGLM package)? I haven't seen the max function in other places.
This may or may not be related, but how will we guarantee that the endogenous variable does not contain invalid values (negative for poisson, outside [0, 1] for binomial, etc...?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50801097
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and other optimization problems.
+ *
+ * @param initialModel the initial guess model.
+ * @param reweightFunc the reweight function which is used to update offsets and weights
+ * at each iteration.
+ * @param fitIntercept whether to fit intercept.
+ * @param regParam L2 regularization parameter used by WLS.
+ * @param maxIter maximum number of iterations.
+ * @param tol the convergence tolerance.
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val initialModel: WeightedLeastSquaresModel,
+ val reweightFunc: (Instance, WeightedLeastSquaresModel) => (Double, Double),
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ var converged = false
+ var iter = 0
+
+ var offsetsAndWeights: RDD[(Double, Double)] = null
+ var model: WeightedLeastSquaresModel = initialModel
+ var oldModel: WeightedLeastSquaresModel = initialModel
+
+ while (iter < maxIter && !converged) {
+
+ oldModel = model
+
+ // Update offsets and weights using reweightFunc
+ offsetsAndWeights = instances.map { instance => reweightFunc(instance, oldModel) }
+
+ // Estimate new model
+ val newInstances = instances.zip(offsetsAndWeights).map {
+ case (instance, (offset, weight)) => Instance(offset, weight, instance.features)
+ }
+ model = new WeightedLeastSquares(fitIntercept, regParam, false, false).fit(newInstances)
--- End diff --
use named arguments for booleans
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50842817
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and other optimization problems.
+ *
+ * @param initialModel the initial guess model.
+ * @param reweightFunc the reweight function which is used to update offsets and weights
+ * at each iteration.
+ * @param fitIntercept whether to fit intercept.
+ * @param regParam L2 regularization parameter used by WLS.
+ * @param maxIter maximum number of iterations.
+ * @param tol the convergence tolerance.
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val initialModel: WeightedLeastSquaresModel,
+ val reweightFunc: (Instance, WeightedLeastSquaresModel) => (Double, Double),
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ var converged = false
+ var iter = 0
+
+ var offsetsAndWeights: RDD[(Double, Double)] = null
+ var model: WeightedLeastSquaresModel = initialModel
+ var oldModel: WeightedLeastSquaresModel = initialModel
+
+ while (iter < maxIter && !converged) {
+
+ oldModel = model
+
+ // Update offsets and weights using reweightFunc
+ offsetsAndWeights = instances.map { instance => reweightFunc(instance, oldModel) }
+
+ // Estimate new model
+ val newInstances = instances.zip(offsetsAndWeights).map {
+ case (instance, (offset, weight)) => Instance(offset, weight, instance.features)
+ }
+ model = new WeightedLeastSquares(fitIntercept, regParam, false, false).fit(newInstances)
+
+ val oldParameters = Array.concat(Array(oldModel.intercept), oldModel.coefficients.toArray)
+ val parameters = Array.concat(Array(model.intercept), model.coefficients.toArray)
+ val deltaArray = oldParameters.zip(parameters).map { case (x: Double, y: Double) =>
+ math.abs(x - y)
+ }
--- End diff --
I think Chebyshev distance is better than Euclidean distance if we compare between difference models rather than data. I can get efficient way to compute Chebyshev distance inspired by your comments, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-172656627
@yanboliang I made a pass on your implementation and read Green's paper "Iteratively Reweighted Least Squares for Maximum Likelihood Estimation, and some Robust and Resistant Alternatives". My main question is whether we should couple IRLS and GLM together. IRLS could be applied to models outside GLM family. Coupling them together makes it harder to extend and some concepts harder to understand, e.g., "Family.startingMu". I thought about decoupling the concepts but didn't find a satisfactory solution. One possible approach is to make IRLS accept the initial guess `x0` and a reweighting function that updates weights and offsets at each iteration. Then we can wrap GLM family and link functions to use that interface. It would be great if the IRLS implementation could be extended to handle: GLMs, Lp regression, LASSO, and maybe some non-convex loss functions. If this sounds good to you, we can think more about the refactoring.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169959354
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49016/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170952320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49240/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50801090
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and some other optimization problems.
+ *
+ * @param initialModel the initial guess model.
+ * @param reweightFunc the reweight function which is used to update offsets and weights
+ * at each iteration.
+ * @param fitIntercept whether to fit intercept.
+ * @param regParam L2 regularization parameter used by WLS.
+ * @param maxIter maximum number of iterations.
+ * @param tol the convergence tolerance.
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val initialModel: WeightedLeastSquaresModel,
+ val reweightFunc: (Instance, WeightedLeastSquaresModel) => (Double, Double),
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ var converged = false
+ var iter = 0
+
+ var offsetsAndWeights: RDD[(Double, Double)] = null
--- End diff --
It is fine for internal use.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-174421925
**[Test build #49981 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49981/consoleFull)** for PR 10639 at commit [`6074fd6`](https://github.com/apache/spark/commit/6074fd6d2c4b9cb181fe6dd4d2b83972bc39346f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50801341
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and other optimization problems.
--- End diff --
It would be good to provide a reference about IRLS. The IRLS page on Wikipedia is specialized for Lp regression. I would recommend Green's paper as a reference: http://www.jstor.org/stable/2345503
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50517513
--- Diff: mllib/src/test/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquaresSuite.scala ---
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.ml.glm._
+import org.apache.spark.mllib.linalg.BLAS._
+import org.apache.spark.mllib.linalg.Vectors
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.mllib.util.TestingUtils._
+import org.apache.spark.rdd.RDD
+
+class IterativelyReweightedLeastSquaresSuite extends SparkFunSuite with MLlibTestSparkContext {
+
+ private var instances1: RDD[Instance] = _
+ private var instances2: RDD[Instance] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ /*
+ R code:
+
+ A <- matrix(c(0, 1, 2, 3, 5, 2, 1, 3), 4, 2)
+ b <- c(1, 0, 1, 0)
+ w <- c(1, 2, 3, 4)
+ */
+ instances1 = sc.parallelize(Seq(
+ Instance(1.0, 1.0, Vectors.dense(0.0, 5.0).toSparse),
+ Instance(0.0, 2.0, Vectors.dense(1.0, 2.0)),
+ Instance(1.0, 3.0, Vectors.dense(2.0, 1.0)),
+ Instance(0.0, 4.0, Vectors.dense(3.0, 3.0))
+ ), 2)
+ /*
+ R code:
+
+ A <- matrix(c(0, 1, 2, 3, 5, 7, 11, 13), 4, 2)
+ b <- c(2, 8, 3, 9)
+ w <- c(1, 2, 3, 4)
+ */
+ instances2 = sc.parallelize(Seq(
+ Instance(2.0, 1.0, Vectors.dense(0.0, 5.0).toSparse),
+ Instance(8.0, 2.0, Vectors.dense(1.0, 7.0)),
+ Instance(3.0, 3.0, Vectors.dense(2.0, 11.0)),
+ Instance(9.0, 4.0, Vectors.dense(3.0, 13.0))
+ ), 2)
+ }
+
+ test("IRLS against GLM with Binomial") {
+ /*
+ R code:
+
+ df <- as.data.frame(cbind(A, b))
+ for (formula in c(b ~ . -1, b ~ .)) {
+ model <- glm(formula, family="binomial", data=df, weights=w)
+ print(as.vector(coef(model)))
+ }
+
+ [1] -0.30216651 -0.04452045
+ [1] 3.5651651 -1.2334085 -0.7348971
+ */
+ val expected = Seq(
+ Vectors.dense(0.0, -0.30216651, -0.04452045),
+ Vectors.dense(3.5651651, -1.2334085, -0.7348971))
+
+ import IterativelyReweightedLeastSquaresSuite._
+
+ val family = Binomial(Logit)
+ val regParam = 0.0
+ val maxIter = 25
+ val tol = 1e-8
+
+ var idx = 0
+ for (fitIntercept <- Seq(false, true)) {
+ val initial = initialModel(family, instances1, fitIntercept, regParam)
+ val irls = new IterativelyReweightedLeastSquares(initial, BinomialReweightFunc,
+ fitIntercept, regParam, maxIter, tol).fit(instances1)
+ val actual = Vectors.dense(irls.intercept, irls.coefficients(0), irls.coefficients(1))
+ assert(actual ~== expected(idx) absTol 1e-4)
+ idx += 1
+ }
+ }
+
+ test("IRLS against GLM with Poisson") {
+ /*
+ R code:
+
+ df <- as.data.frame(cbind(A, b))
+ for (formula in c(b ~ . -1, b ~ .)) {
+ model <- glm(formula, family="poisson", data=df, weights=w)
+ print(as.vector(coef(model)))
+ }
+
+ [1] -0.09607792 0.18375613
+ [1] 6.299947 3.324107 -1.081766
+ */
+ val expected = Seq(
+ Vectors.dense(0.0, -0.09607792, 0.18375613),
+ Vectors.dense(6.299947, 3.324107, -1.081766))
+
+ import IterativelyReweightedLeastSquaresSuite._
+
+ val family = Poisson(Log)
+ val regParam = 0.0
+ val maxIter = 25
+ val tol = 1e-8
+
+ var idx = 0
+ for (fitIntercept <- Seq(false, true)) {
+ val initial = initialModel(family, instances2, fitIntercept, regParam)
+ val irls = new IterativelyReweightedLeastSquares(initial, PoissonReweightFunc,
+ fitIntercept, regParam, maxIter, tol).fit(instances2)
+ val actual = Vectors.dense(irls.intercept, irls.coefficients(0), irls.coefficients(1))
+ assert(actual ~== expected(idx) absTol 1e-4)
+ idx += 1
+ }
+ }
+
+ test("IRLS against Quantile Regression / L1Regression") {
+ /*
+ R code:
+
+ library(quantreg)
+
+ df <- as.data.frame(cbind(A, b))
+ for (formula in c(b ~ . -1, b ~ .)) {
+ model <- rq(formula, data=df, weights=w)
+ print(as.vector(coef(model)))
+ }
+
+ [1] 1.266667 0.400000
+ [1] 29.5 17.0 -5.5
+ */
+ val expected = Seq(
+ Vectors.dense(0.0, 1.266667, 0.400000),
+ Vectors.dense(29.5, 17.0, -5.5))
+
+ import IterativelyReweightedLeastSquaresSuite._
+
+ val regParam = 0.0
+ val maxIter = 200
+ val tol = 1e-7
+
+ var idx = 0
+ for (fitIntercept <- Seq(false, true)) {
+ val wls = new WeightedLeastSquares(fitIntercept, regParam, false, false)
+ val initial = wls.fit(instances2)
+ val irls = new IterativelyReweightedLeastSquares(initial, L1RegressionReweightFunc,
+ fitIntercept, regParam, maxIter, tol).fit(instances2)
+ val actual = Vectors.dense(irls.intercept, irls.coefficients(0), irls.coefficients(1))
+ assert(actual ~== expected(idx) absTol 1e-4)
+ idx += 1
+ }
+ }
+}
+
+object IterativelyReweightedLeastSquaresSuite {
+
+ def initialModel(
+ family: Family,
+ instances: RDD[Instance],
+ fitIntercept: Boolean,
+ regParam: Double): WeightedLeastSquaresModel = {
+ val wls = new WeightedLeastSquares(fitIntercept, regParam, false, false)
+ val yMean = instances.map(_.label).mean
+ val newInstances = instances.map { instance =>
+ val eta = family.predict((instance.label + yMean) / 2.0)
+ Instance(eta, instance.weight, instance.features)
+ }
+ wls.fit(newInstances)
+ }
+
+ def BinomialReweightFunc(
+ instance: Instance,
+ model: WeightedLeastSquaresModel): (Double, Double) = {
+ val family = Binomial(Logit)
+ val eta = dot(instance.features, model.coefficients) + model.intercept
--- End diff --
Could we add a method ```predict``` for ```WeightedLeastSquaresModel``` that we can eliminate some code duplication?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170496760
**[Test build #49125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49125/consoleFull)** for PR 10639 at commit [`65133ad`](https://github.com/apache/spark/commit/65133add9fdc2a4612fe481954b951eaac40a1a7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-176454982
LGTM. Merged into master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169888565
@sethah Thanks for your comments. The mainly reference documents is http://data.princeton.edu/wws509/notes/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-171365945
@yanboliang The approach looks good, nice PR! It will be good to get others feedback, especially those with an idea of how the GLM package will fit into the larger picture. I think the exact form of the link functions and families will depend on what the plans are for future additions (e.g. what summary stats will we return?) but those can be done in a separate PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50390322
--- Diff: mllib/src/main/scala/org/apache/spark/ml/glm/Families.scala ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.glm
+
+import org.apache.spark.rdd.RDD
+
+/**
+ * A description of the error distribution and link function to be used in the model.
+ * @param link a link function instance
+ */
+private[ml] abstract class Family(val link: Link) extends Serializable {
--- End diff --
I think ```Families``` can be used by [SPARK-12811](https://issues.apache.org/jira/browse/SPARK-12811) which provide Estimator interface for GLMs, so I move it to a new folder named ```glm```.
Here we have two ways to support GLMs:
* Implement ```reweightFunc``` for each ```Family/Link``` directly based on mathematical formula.
* Implement the ```Family``` framework like what I have done and a factory method which can output ```reweightFunc``` according to argument.
The former one has better execution efficiency, the later one is more easy to understand. Looking forward your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-173554998
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49875/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170939993
@sethah Thanks for your comments! This is not a public GLM API but a resolver like ```WeightedLeastSquares```. It will be used to resolve ```GLM``` and other algorithms such as ```LogisticRegression```, so I marked all the classes as ```private[ml]```.
Just like what you side, some GLM implementations restrict the link functions that can be used with a given family, I think we should keep the restriction in Spark as well and this should further refinement when we make GLM public.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170476891
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49120/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-173554996
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170468883
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r49157038
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/GLMFamilies.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.rdd.RDD
+
+/**
+ * A description of the error distribution and link function to be used in the model.
+ * @param link a link function instance
+ */
+private[ml] abstract class Family(val link: Link) extends Serializable {
+
+ /**
+ * Starting value for mu in the IRLS algorithm.
+ */
+ def startingMu(y: Double, yMean: Double): Double = (y + yMean) / 2.0
+
+ /**
+ * Deviance of (y, mu) pair.
+ * Deviance is usually defined as twice the loglikelihood ratio.
+ */
+ def deviance(y: RDD[Double], mu: RDD[Double]): Double
+
+ /** Weights for IRLS steps. */
+ def weights(mu: Double): Double
+
+ /** The working dependent variable. */
+ def z(y: Double, mu: Double, eta: Double): Double
+}
+
+/**
+ * Binomial exponential family distribution.
+ * The default link for the Binomial family is the logit link.
+ * @param link a link function instance
+ */
+private[ml] class Binomial(link: Link = new Logit) extends Family(link) {
+
+ override def startingMu(y: Double, yMean: Double): Double = (y + 0.5) / 2.0
+
+ override def deviance(y: RDD[Double], mu: RDD[Double]): Double = {
+ mu.zip(y).map { case (mu, y) =>
+ val my = 1.0 - y
+ y * math.log(math.max(y, 1.0) / mu) +
+ my * math.log(math.max(my, 1.0) / (1.0 - mu))
+ }.sum() * 2
+ }
+
+ override def weights(mu: Double): Double = {
+ mu * (1 - mu)
+ }
+
+ override def z(y: Double, mu: Double, eta: Double): Double = {
+ eta + (y - mu) * link.deriv(mu)
+ }
+}
+
+/**
+ * Poisson exponential family.
+ * The default link for the Poisson family is the log link.
+ * @param link a link function instance
+ */
+private[ml] class Poisson(link: Link = new Logit) extends Family(link) {
--- End diff --
Yep, it's typo.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169888340
**[Test build #49000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49000/consoleFull)** for PR 10639 at commit [`6d809d0`](https://github.com/apache/spark/commit/6d809d0c935b42434e5b69f31561d9e307bf191b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-173219998
@mengxr Thanks for your comments! IRLS is not bound with GLM in essence, so it's make sense to decouple them. Based on your prompt, I propose the following API for ```IterativelyReweightedLeastSquares```:
```Scala
private[ml] class IterativelyReweightedLeastSquares(
val initialModel: WeightedLeastSquaresModel,
val reweightedFunction: (RDD[Instance], WeightedLeastSquaresModel) => RDD[(Double, Double)],
val fitIntercept: Boolean,
val regParam: Double,
val standardizeFeatures: Boolean,
val standardizeLabel: Boolean,
val maxIter: Int,
val tol: Double) extends Logging with Serializable {
......
}
```
where ```initialModel``` is the initial guess, ```reweightedFunction``` is used to update ```y/z```(adjusted response variable) and ```weights```. And the terminate condition is delta model not great than ```tolerance```.
This framework can fit GLMs, Lp regression and Lasso. I will update this PR following this idea if it's OK. And if I have some misunderstand, please correct me. Thanks!
BTW, Do you know which package is the most authoritative one for Lp regression? I found ```pracma:::l1linreg``` and ```L1pack:::l1fit```, but they use ```qr.solve()``` to solve WLS equation and produce different result compared with ML ```WeightedLeastSquares``` when ```weights``` is not equal to 1.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-176036767
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50801103
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and other optimization problems.
+ *
+ * @param initialModel the initial guess model.
+ * @param reweightFunc the reweight function which is used to update offsets and weights
+ * at each iteration.
+ * @param fitIntercept whether to fit intercept.
+ * @param regParam L2 regularization parameter used by WLS.
+ * @param maxIter maximum number of iterations.
+ * @param tol the convergence tolerance.
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val initialModel: WeightedLeastSquaresModel,
+ val reweightFunc: (Instance, WeightedLeastSquaresModel) => (Double, Double),
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ var converged = false
+ var iter = 0
+
+ var offsetsAndWeights: RDD[(Double, Double)] = null
+ var model: WeightedLeastSquaresModel = initialModel
+ var oldModel: WeightedLeastSquaresModel = initialModel
+
+ while (iter < maxIter && !converged) {
+
+ oldModel = model
+
+ // Update offsets and weights using reweightFunc
+ offsetsAndWeights = instances.map { instance => reweightFunc(instance, oldModel) }
+
+ // Estimate new model
+ val newInstances = instances.zip(offsetsAndWeights).map {
+ case (instance, (offset, weight)) => Instance(offset, weight, instance.features)
+ }
+ model = new WeightedLeastSquares(fitIntercept, regParam, false, false).fit(newInstances)
+
+ val oldParameters = Array.concat(Array(oldModel.intercept), oldModel.coefficients.toArray)
+ val parameters = Array.concat(Array(model.intercept), model.coefficients.toArray)
+ val deltaArray = oldParameters.zip(parameters).map { case (x: Double, y: Double) =>
+ math.abs(x - y)
+ }
+ if (!deltaArray.exists(_ > tol)) {
--- End diff --
This is the infinity norm. Any reference?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169896702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49000/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169848523
@yanboliang Could you post a link to a reference paper? I find documentation on IRLS scattered, so it would be nice to have something concrete to point to.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170476887
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-171838281
@yanboliang Thanks for working on this! I'll make a pass soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170498120
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50390974
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and some other optimization problems.
+ *
+ * @param initialModel the initial guess model.
+ * @param reweightFunc the reweight function which is used to update offsets and weights
+ * at each iteration.
+ * @param fitIntercept whether to fit intercept.
+ * @param regParam L2 regularization parameter used by WLS.
+ * @param maxIter maximum number of iterations.
+ * @param tol the convergence tolerance.
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val initialModel: WeightedLeastSquaresModel,
+ val reweightFunc: (Instance, WeightedLeastSquaresModel) => (Double, Double),
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ var converged = false
+ var iter = 0
+
+ var offsetsAndWeights: RDD[(Double, Double)] = null
--- End diff --
R glm has argument named ```offset```, but ```offsetsAndWeights``` is ```private```. I hope it won't confuse users, or should we rename to other better one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r49226861
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/GLMFamilies.scala ---
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.rdd.RDD
+
+/**
+ * A description of the error distribution and link function to be used in the model.
+ * @param link a link function instance
+ */
+private[ml] abstract class Family(val link: Link) extends Serializable {
+
+ /**
+ * Starting value for mu in the IRLS algorithm.
+ */
+ def startingMu(y: Double, yMean: Double): Double = (y + yMean) / 2.0
+
+ /**
+ * Deviance of (y, mu) pair.
+ * Deviance is usually defined as twice the loglikelihood ratio.
+ */
+ def deviance(y: RDD[Double], mu: RDD[Double]): Double
+
+ /** Weights for IRLS steps. */
+ def weights(mu: Double): Double
+
+ /** The working dependent variable. */
+ def z(y: Double, mu: Double, eta: Double): Double
+
+ /** Linear predictors based on given mu. */
+ def predict(mu: Double): Double = this.link.link(mu)
+
+ /** Fitted values based on linear predictors eta. */
+ def fitted(eta: Double): Double = this.link.unlink(eta)
+}
+
+/**
+ * Binomial exponential family distribution.
+ * The default link for the Binomial family is the logit link.
+ * @param link a link function instance
+ */
+private[ml] class Binomial(link: Link = Logit) extends Family(link) {
+
+ override def startingMu(y: Double, yMean: Double): Double = (y + 0.5) / 2.0
+
+ override def deviance(y: RDD[Double], mu: RDD[Double]): Double = {
+ mu.zip(y).map { case (mu, y) =>
+ val my = 1.0 - y
+ y * math.log(math.max(y, 1.0) / mu) +
+ my * math.log(math.max(my, 1.0) / (1.0 - mu))
+ }.sum() * 2
+ }
+
+ override def weights(mu: Double): Double = {
+ mu * (1 - mu)
+ }
+
+ override def z(y: Double, mu: Double, eta: Double): Double = {
+ eta + (y - mu) * link.deriv(mu)
--- End diff --
This is the generic form for the z update, but in the Poisson family you have hard coded the update specific to the family. I think we should stick to a single convention, and it makes most sense to me to use the generic update. That way, we can implement in the parent class and we don't need to implement in the base classes.
Additionally, I think it would be better to call this something other than `z` because it isn't very descriptive. I don't have a strong opinion on what it should be called, but I have seen it called _adjusted response_ in other places (among other names).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-176036615
**[Test build #50261 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50261/consoleFull)** for PR 10639 at commit [`cb2057e`](https://github.com/apache/spark/commit/cb2057e2eeadbb52c467667ad15e23f1723685cd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r49144625
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/GLMFamilies.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.rdd.RDD
+
+/**
+ * A description of the error distribution and link function to be used in the model.
+ * @param link a link function instance
+ */
+private[ml] abstract class Family(val link: Link) extends Serializable {
+
+ /**
+ * Starting value for mu in the IRLS algorithm.
+ */
+ def startingMu(y: Double, yMean: Double): Double = (y + yMean) / 2.0
+
+ /**
+ * Deviance of (y, mu) pair.
+ * Deviance is usually defined as twice the loglikelihood ratio.
+ */
+ def deviance(y: RDD[Double], mu: RDD[Double]): Double
+
+ /** Weights for IRLS steps. */
+ def weights(mu: Double): Double
+
+ /** The working dependent variable. */
+ def z(y: Double, mu: Double, eta: Double): Double
+}
+
+/**
+ * Binomial exponential family distribution.
+ * The default link for the Binomial family is the logit link.
+ * @param link a link function instance
+ */
+private[ml] class Binomial(link: Link = new Logit) extends Family(link) {
+
+ override def startingMu(y: Double, yMean: Double): Double = (y + 0.5) / 2.0
+
+ override def deviance(y: RDD[Double], mu: RDD[Double]): Double = {
+ mu.zip(y).map { case (mu, y) =>
+ val my = 1.0 - y
+ y * math.log(math.max(y, 1.0) / mu) +
+ my * math.log(math.max(my, 1.0) / (1.0 - mu))
+ }.sum() * 2
+ }
+
+ override def weights(mu: Double): Double = {
+ mu * (1 - mu)
+ }
+
+ override def z(y: Double, mu: Double, eta: Double): Double = {
+ eta + (y - mu) * link.deriv(mu)
+ }
+}
+
+/**
+ * Poisson exponential family.
+ * The default link for the Poisson family is the log link.
+ * @param link a link function instance
+ */
+private[ml] class Poisson(link: Link = new Logit) extends Family(link) {
--- End diff --
I believe the link function here should default to `Log` not `Logit`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-174430167
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50801095
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and other optimization problems.
+ *
+ * @param initialModel the initial guess model.
+ * @param reweightFunc the reweight function which is used to update offsets and weights
+ * at each iteration.
+ * @param fitIntercept whether to fit intercept.
+ * @param regParam L2 regularization parameter used by WLS.
+ * @param maxIter maximum number of iterations.
+ * @param tol the convergence tolerance.
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val initialModel: WeightedLeastSquaresModel,
+ val reweightFunc: (Instance, WeightedLeastSquaresModel) => (Double, Double),
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ var converged = false
+ var iter = 0
+
+ var offsetsAndWeights: RDD[(Double, Double)] = null
+ var model: WeightedLeastSquaresModel = initialModel
+ var oldModel: WeightedLeastSquaresModel = initialModel
+
+ while (iter < maxIter && !converged) {
+
+ oldModel = model
+
+ // Update offsets and weights using reweightFunc
+ offsetsAndWeights = instances.map { instance => reweightFunc(instance, oldModel) }
+
+ // Estimate new model
+ val newInstances = instances.zip(offsetsAndWeights).map {
--- End diff --
`zip` is not efficient. Generate `newInstances` directly:
~~~scala
val newInstances = instances.map { instance =>
val (newOffset, newWeight) = reweightFunc(instance, oldModel)
Instance(newOffset, newWeight, instance.features)
}
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-174430106
**[Test build #49981 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49981/consoleFull)** for PR 10639 at commit [`6074fd6`](https://github.com/apache/spark/commit/6074fd6d2c4b9cb181fe6dd4d2b83972bc39346f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-175857448
@yanboliang This PR should be ready to go after the comments are addressed. FYI, I assigned SPARK-12811 to you, which is along the line of your work. Thanks for working on GLM features!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170468886
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49113/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169612741
**[Test build #48930 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48930/consoleFull)** for PR 10639 at commit [`86492bb`](https://github.com/apache/spark/commit/86492bb707635a2da904e9d4ffb2cd7e2257d533).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170497112
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49125/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r50801093
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.Logging
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.rdd.RDD
+
+/**
+ * Model fitted by [[IterativelyReweightedLeastSquares]].
+ * @param coefficients model coefficients
+ * @param intercept model intercept
+ */
+private[ml] class IterativelyReweightedLeastSquaresModel(
+ val coefficients: DenseVector,
+ val intercept: Double) extends Serializable
+
+/**
+ * Implements the method of iteratively reweighted least squares (IRLS) which is used to solve
+ * certain optimization problems by an iterative method. In each step of the iterations, it
+ * involves solving a weighted lease squares (WLS) problem by [[WeightedLeastSquares]].
+ * It can be used to find maximum likelihood estimates of a generalized linear model (GLM),
+ * find M-estimator in robust regression and other optimization problems.
+ *
+ * @param initialModel the initial guess model.
+ * @param reweightFunc the reweight function which is used to update offsets and weights
+ * at each iteration.
+ * @param fitIntercept whether to fit intercept.
+ * @param regParam L2 regularization parameter used by WLS.
+ * @param maxIter maximum number of iterations.
+ * @param tol the convergence tolerance.
+ */
+private[ml] class IterativelyReweightedLeastSquares(
+ val initialModel: WeightedLeastSquaresModel,
+ val reweightFunc: (Instance, WeightedLeastSquaresModel) => (Double, Double),
+ val fitIntercept: Boolean,
+ val regParam: Double,
+ val maxIter: Int,
+ val tol: Double) extends Logging with Serializable {
+
+ def fit(instances: RDD[Instance]): IterativelyReweightedLeastSquaresModel = {
+
+ var converged = false
+ var iter = 0
+
+ var offsetsAndWeights: RDD[(Double, Double)] = null
+ var model: WeightedLeastSquaresModel = initialModel
+ var oldModel: WeightedLeastSquaresModel = initialModel
--- End diff --
`= null`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-174430169
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49981/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] Implement IterativelyReweigh...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-176021171
**[Test build #50261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50261/consoleFull)** for PR 10639 at commit [`cb2057e`](https://github.com/apache/spark/commit/cb2057e2eeadbb52c467667ad15e23f1723685cd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/10639#discussion_r49395627
--- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/GLMFamilies.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.optim
+
+import org.apache.spark.rdd.RDD
+
+/**
+ * A description of the error distribution and link function to be used in the model.
+ * @param link a link function instance
+ */
+private[ml] abstract class Family(val link: Link) extends Serializable {
+
+ /**
+ * Starting value for mu in the IRLS algorithm.
+ */
+ def startingMu(y: Double, yMean: Double): Double = (y + yMean) / 2.0
+
+ /**
+ * Deviance of (y, mu) pair.
+ * Deviance is usually defined as twice the loglikelihood ratio.
+ */
+ def deviance(y: RDD[Double], mu: RDD[Double]): Double
+
+ /** Weights for IRLS steps. */
+ def weights(mu: Double): Double
+
+ /** The adjusted response variable. */
+ def adjusted(y: Double, mu: Double, eta: Double): Double = {
+ eta + (y - mu) * link.deriv(mu)
+ }
+
+ /** Linear predictors based on given mu. */
+ def predict(mu: Double): Double = this.link.link(mu)
+
+ /** Fitted values based on linear predictors eta. */
+ def fitted(eta: Double): Double = this.link.unlink(eta)
+}
+
+/**
+ * Binomial exponential family distribution.
+ * The default link for the Binomial family is the logit link.
+ * @param link a link function instance
+ */
+private[ml] class Binomial(link: Link = Logit) extends Family(link) {
+
+ override def startingMu(y: Double, yMean: Double): Double = (y + 0.5) / 2.0
+
+ override def deviance(y: RDD[Double], mu: RDD[Double]): Double = {
+ mu.zip(y).map { case (mu, y) =>
+ val my = 1.0 - y
+ y * math.log(math.max(y, 1.0) / mu) +
+ my * math.log(math.max(my, 1.0) / (1.0 - mu))
+ }.sum() * 2
+ }
+
+ override def weights(mu: Double): Double = {
+ mu * (1 - mu)
--- End diff --
Hard coding the weights like this here won't be correct if anything other than the canonical link function is used, I believe. Since we aren't doing anything to restrict link functions to only the canonical ones, this should probably defined in terms of the link function's derivative. Statsmodels does it [here](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/families/family.py#L120).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-170731964
I'm still not exactly clear what the scope of this GLM implementation will be (e.g. if it will be a public API) and I'd be interested to hear more on that topic. But along those lines, some GLM implementations restrict the link functions that can be used with a given family, so I'm wondering if that should be done in Spark as well. I suppose this is only a problem if we expose it as an API. Curious to hear others' thoughts...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [ML] IterativelyReweightedLeastSq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-169958981
**[Test build #49016 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49016/consoleFull)** for PR 10639 at commit [`5b26d2a`](https://github.com/apache/spark/commit/5b26d2a2f01916eeebdf81da3fe4fbbcaf3cfe20).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org