[jira] [Commented] (SPARK-16638) The L2 regularization of LinearRegression seems wrong when standardization is false

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-16638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385200#comment-15385200 ] 

Apache Spark commented on SPARK-16638:
--------------------------------------

User 'WeichenXu123' has created a pull request for this issue:
https://github.com/apache/spark/pull/14276

> The L2 regularization of LinearRegression seems wrong when standardization is false
> -----------------------------------------------------------------------------------
>
>                 Key: SPARK-16638
>                 URL: https://issues.apache.org/jira/browse/SPARK-16638
>             Project: Spark
>          Issue Type: Bug
>          Components: ML, Optimizer
>            Reporter: Weichen Xu
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The original L2 is
> 0.5 * effectiveL2regParam * sigma( wi^2 )
> (wi is the coefficients we want to train)
> And in linearRegression code, when standardization == false, the code modify L2 into:
> 0.5 * effectiveL2regParam * sigma( ( wi / featuresStd(i) )^2 )
> It is obviously wrong, I think.
> As the purpose of author wrote in the code comment, the modification to L2 reg should be:
> 0.5 * effectiveL2regParam * sigma( ( wi * featuresStd(i) )^2 )
> wi should not be divided by featuresStd(i), but should be multiplied by featuresStd(i)
> We can simply think this problem in the following way:
> assume a training data, there is a dimension k with very large std ( the value of featuresStd(k) is very large), so, we hope the training result coefficient w(k) to be small to keep the numerical stability. As a way of that, we can add a penalty to the L2 reg on this dimension.
> so that it should be surely w(k) * featuresStd(k), not  w(k) / featuresStd(k)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org