You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by "Ulanov, Alexander" <al...@hp.com> on 2015/04/08 00:03:33 UTC
Regularization in MLlib
Hi,
Could anyone elaborate on the regularization in Spark? I've found that L1 and L2 are implemented with Updaters (L1Updater, SquaredL2Updater).
1)Why the loss reported by L2 is (0.5 * regParam * norm * norm) where norm is Norm(weights, 2.0)? It should be 0.5*regParam*norm (0.5 to disappear after differentiation). It seems that it is mixed up with mean squared error.
2)Why all weights are regularized? I think we should leave the bias weights (aka free or intercept) untouched if we don't assume that the data is centralized.
3)Are there any short-term plans to move regularization from updater to a more convenient place?
Best regards, Alexander
Re: Regularization in MLlib
Posted by DB Tsai <db...@dbtsai.com>.
Hi Theodore,
I'm currently working on elastic-net regression in ML framework, and I
decided not to have any extra layer of abstraction for now but focus
on accuracy and performance. We may come out with proper solution
later. Any idea is welcome.
Sincerely,
DB Tsai
-------------------------------------------------------
Blog: https://www.dbtsai.com
On Tue, Apr 14, 2015 at 6:54 AM, Theodore Vasiloudis
<th...@gmail.com> wrote:
> Hello DB,
>
> could you elaborate a bit on how you are currently fixing this for the new
> ML pipeline framework?
> Are there any JIRAs/PR we could follow?
>
> Regards,
> Theodore
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Regularization-in-MLlib-tp11457p11583.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
RE: Regularization in MLlib
Posted by Theodore Vasiloudis <th...@gmail.com>.
Hello DB,
could you elaborate a bit on how you are currently fixing this for the new
ML pipeline framework?
Are there any JIRAs/PR we could follow?
Regards,
Theodore
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Regularization-in-MLlib-tp11457p11583.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
RE: Regularization in MLlib
Posted by "Ulanov, Alexander" <al...@hp.com>.
Hi DB,
Thank you!
In general case (not only for regression), I think that Regularizer should be tightly coupled with Gradient otherwise it will have no idea which weights are bias (intercept).
Best regards, Alexander
-----Original Message-----
From: DB Tsai [mailto:dbtsai@dbtsai.com]
Sent: Tuesday, April 07, 2015 3:28 PM
To: Ulanov, Alexander
Cc: dev@spark.apache.org
Subject: Re: Regularization in MLlib
1) Norm(weights, N) will return (w_1^N + w_2^N +....)^(1/N), so norm
* norm is required.
2) This is bug as you said. I intend to fix this using weighted regularization, and intercept term will be regularized with weight zero. https://github.com/apache/spark/pull/1518 But I never actually have time to finish it. In the meantime, I'm fixing this without this framework in new ML pipeline framework.
3) I think in the long term, we need weighted regularizer instead of updater which couples regularization and adaptive step size update for GD which is not needed in other optimization package.
Sincerely,
DB Tsai
-------------------------------------------------------
Blog: https://www.dbtsai.com
On Tue, Apr 7, 2015 at 3:03 PM, Ulanov, Alexander <al...@hp.com> wrote:
> Hi,
>
> Could anyone elaborate on the regularization in Spark? I've found that L1 and L2 are implemented with Updaters (L1Updater, SquaredL2Updater).
> 1)Why the loss reported by L2 is (0.5 * regParam * norm * norm) where norm is Norm(weights, 2.0)? It should be 0.5*regParam*norm (0.5 to disappear after differentiation). It seems that it is mixed up with mean squared error.
> 2)Why all weights are regularized? I think we should leave the bias weights (aka free or intercept) untouched if we don't assume that the data is centralized.
> 3)Are there any short-term plans to move regularization from updater to a more convenient place?
>
> Best regards, Alexander
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Regularization in MLlib
Posted by DB Tsai <db...@dbtsai.com>.
1) Norm(weights, N) will return (w_1^N + w_2^N +....)^(1/N), so norm
* norm is required.
2) This is bug as you said. I intend to fix this using weighted
regularization, and intercept term will be regularized with weight
zero. https://github.com/apache/spark/pull/1518 But I never actually
have time to finish it. In the meantime, I'm fixing this without this
framework in new ML pipeline framework.
3) I think in the long term, we need weighted regularizer instead of
updater which couples regularization and adaptive step size update for
GD which is not needed in other optimization package.
Sincerely,
DB Tsai
-------------------------------------------------------
Blog: https://www.dbtsai.com
On Tue, Apr 7, 2015 at 3:03 PM, Ulanov, Alexander
<al...@hp.com> wrote:
> Hi,
>
> Could anyone elaborate on the regularization in Spark? I've found that L1 and L2 are implemented with Updaters (L1Updater, SquaredL2Updater).
> 1)Why the loss reported by L2 is (0.5 * regParam * norm * norm) where norm is Norm(weights, 2.0)? It should be 0.5*regParam*norm (0.5 to disappear after differentiation). It seems that it is mixed up with mean squared error.
> 2)Why all weights are regularized? I think we should leave the bias weights (aka free or intercept) untouched if we don't assume that the data is centralized.
> 3)Are there any short-term plans to move regularization from updater to a more convenient place?
>
> Best regards, Alexander
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org