You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Frank Scholten <fr...@frankscholten.nl> on 2014/01/13 13:14:38 UTC
Logistic Regression cost function
Hi,
I followed the Coursera Machine Learning course quite a while ago and I am
trying to find out how Mahout implements the Logistic Regression cost
function in the code surrounding AbstractOnlineLogisticRegression.
I am looking at the train method in AbstractOnlineLogisticRegression and I
see online gradient descent step where the beta matrix is updated but to me
its unclear how matches with the cost function described at:
http://www.holehouse.org/mlclass/06_Logistic_Regression.html
Perhaps Mahout uses an optimized approach for that does not directly map
into the formula at that link?
Cheers,
Frank
Re: Logistic Regression cost function
Posted by Ted Dunning <te...@gmail.com>.
I think that this is the link in the code:
http://leon.bottou.org/research/stochastic
On Mon, Jan 13, 2014 at 11:58 AM, Frank Scholten <fr...@frankscholten.nl>wrote:
> Do you know which paper it is? He has quite a few publications. I don't see
> any mention of one of his papers in the code. I only see
> www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdf in
> MixedGradient but this is something different.
>
>
>
> On Mon, Jan 13, 2014 at 1:27 PM, Suneel Marthi <suneel_marthi@yahoo.com
> >wrote:
>
> > Mahout's impl is based off of Leon Bottou's paper on this subject. I
> > don't gave the link handy but it's referenced in the code or try google
> > search
> >
> > Sent from my iPhone
> >
> > > On Jan 13, 2014, at 7:14 AM, Frank Scholten <fr...@frankscholten.nl>
> > wrote:
> > >
> > > Hi,
> > >
> > > I followed the Coursera Machine Learning course quite a while ago and I
> > am
> > > trying to find out how Mahout implements the Logistic Regression cost
> > > function in the code surrounding AbstractOnlineLogisticRegression.
> > >
> > > I am looking at the train method in AbstractOnlineLogisticRegression
> and
> > I
> > > see online gradient descent step where the beta matrix is updated but
> to
> > me
> > > its unclear how matches with the cost function described at:
> > > http://www.holehouse.org/mlclass/06_Logistic_Regression.html
> > >
> > > Perhaps Mahout uses an optimized approach for that does not directly
> map
> > > into the formula at that link?
> > >
> > > Cheers,
> > >
> > > Frank
> >
>
Re: Logistic Regression cost function
Posted by Ted Dunning <te...@gmail.com>.
IF you really want to re-organize the multiplication it is fine by me.
On Tue, Jan 14, 2014 at 1:20 AM, Frank Scholten <fr...@frankscholten.nl>wrote:
> I see the update rule to the beta matrix is derived from pseudo code in the
> innermost loop in 'Algorithm 1: Stochastic Gradient Descent' in
>
> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf
>
> In the paper the learning rate is put before the gradient of the error
> function and the instance value multiplication is put before the gradient.
>
> Perhaps we can rearrange the code like this so it matches with the paper
> and add a comment? The only difference then is the perTermAnnealingRate.
>
> // See 'Algorithm 1: SGD' in
>
> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf
> double newValue = beta.getQuick(i, j) + learningRate *
> perTermLearningRate(j) * instance.get(j) * gradientBase;
>
> Cheers,
>
> Frank
>
>
>
>
>
>
>
> On Mon, Jan 13, 2014 at 10:54 PM, Frank Scholten <frank@frankscholten.nl
> >wrote:
>
> > Thanks guys, I have some reading to do :-)
> >
> >
> > On Mon, Jan 13, 2014 at 10:45 PM, Ted Dunning <ted.dunning@gmail.com
> >wrote:
> >
> >> The reference is to the web site in general.
> >>
> >> If anything, this blog is closest:
> >>
> >>
> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf
> >>
> >>
> >> On Mon, Jan 13, 2014 at 1:14 PM, Suneel Marthi <suneel_marthi@yahoo.com
> >> >wrote:
> >>
> >> > I think this is the one. Yes, I don't see this paper referenced in the
> >> > code sorry about that.
> >> > http://leon.bottou.org/publications/pdf/compstat-2010.pdf
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Monday, January 13, 2014 3:51 PM, Frank Scholten <
> >> > frank@frankscholten.nl> wrote:
> >> >
> >> > Do you know which paper it is? He has quite a few publications. I
> don't
> >> see
> >> > any mention of one of his papers in the code. I only see
> >> >
> www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdfin
> >> > MixedGradient but this is something different.
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, Jan 13, 2014 at 1:27 PM, Suneel Marthi <
> suneel_marthi@yahoo.com
> >> > >wrote:
> >> >
> >> > > Mahout's impl is based off of Leon Bottou's paper on this subject.
> I
> >> > > don't gave the link handy but it's referenced in the code or try
> >> google
> >> > > search
> >> > >
> >> > > Sent from my iPhone
> >> > >
> >> > > > On Jan 13, 2014, at 7:14 AM, Frank Scholten <
> frank@frankscholten.nl
> >> >
> >> > > wrote:
> >> > > >
> >> > > > Hi,
> >> > > >
> >> > > > I followed the Coursera Machine Learning course quite a while ago
> >> and I
> >> > > am
> >> > > > trying to find out how Mahout implements the Logistic Regression
> >> cost
> >> > > > function in the code surrounding AbstractOnlineLogisticRegression.
> >> > > >
> >> > > > I am looking at the train method in
> AbstractOnlineLogisticRegression
> >> > and
> >> > > I
> >> > > > see online gradient descent step where the beta matrix is updated
> >> but
> >> > to
> >> > > me
> >> > > > its unclear how matches with the cost function described at:
> >> > > > http://www.holehouse.org/mlclass/06_Logistic_Regression.html
> >> > > >
> >> > > > Perhaps Mahout uses an optimized approach for that does not
> directly
> >> > map
> >> > > > into the formula at that link?
> >> > > >
> >> > > > Cheers,
> >> > > >
> >> > > > Frank
> >> > >
> >> >
> >>
> >
> >
>
Re: Logistic Regression cost function
Posted by Frank Scholten <fr...@frankscholten.nl>.
I see the update rule to the beta matrix is derived from pseudo code in the
innermost loop in 'Algorithm 1: Stochastic Gradient Descent' in
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf
In the paper the learning rate is put before the gradient of the error
function and the instance value multiplication is put before the gradient.
Perhaps we can rearrange the code like this so it matches with the paper
and add a comment? The only difference then is the perTermAnnealingRate.
// See 'Algorithm 1: SGD' in
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf
double newValue = beta.getQuick(i, j) + learningRate *
perTermLearningRate(j) * instance.get(j) * gradientBase;
Cheers,
Frank
On Mon, Jan 13, 2014 at 10:54 PM, Frank Scholten <fr...@frankscholten.nl>wrote:
> Thanks guys, I have some reading to do :-)
>
>
> On Mon, Jan 13, 2014 at 10:45 PM, Ted Dunning <te...@gmail.com>wrote:
>
>> The reference is to the web site in general.
>>
>> If anything, this blog is closest:
>>
>> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf
>>
>>
>> On Mon, Jan 13, 2014 at 1:14 PM, Suneel Marthi <suneel_marthi@yahoo.com
>> >wrote:
>>
>> > I think this is the one. Yes, I don't see this paper referenced in the
>> > code sorry about that.
>> > http://leon.bottou.org/publications/pdf/compstat-2010.pdf
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Monday, January 13, 2014 3:51 PM, Frank Scholten <
>> > frank@frankscholten.nl> wrote:
>> >
>> > Do you know which paper it is? He has quite a few publications. I don't
>> see
>> > any mention of one of his papers in the code. I only see
>> > www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdfin
>> > MixedGradient but this is something different.
>> >
>> >
>> >
>> >
>> > On Mon, Jan 13, 2014 at 1:27 PM, Suneel Marthi <suneel_marthi@yahoo.com
>> > >wrote:
>> >
>> > > Mahout's impl is based off of Leon Bottou's paper on this subject. I
>> > > don't gave the link handy but it's referenced in the code or try
>> google
>> > > search
>> > >
>> > > Sent from my iPhone
>> > >
>> > > > On Jan 13, 2014, at 7:14 AM, Frank Scholten <frank@frankscholten.nl
>> >
>> > > wrote:
>> > > >
>> > > > Hi,
>> > > >
>> > > > I followed the Coursera Machine Learning course quite a while ago
>> and I
>> > > am
>> > > > trying to find out how Mahout implements the Logistic Regression
>> cost
>> > > > function in the code surrounding AbstractOnlineLogisticRegression.
>> > > >
>> > > > I am looking at the train method in AbstractOnlineLogisticRegression
>> > and
>> > > I
>> > > > see online gradient descent step where the beta matrix is updated
>> but
>> > to
>> > > me
>> > > > its unclear how matches with the cost function described at:
>> > > > http://www.holehouse.org/mlclass/06_Logistic_Regression.html
>> > > >
>> > > > Perhaps Mahout uses an optimized approach for that does not directly
>> > map
>> > > > into the formula at that link?
>> > > >
>> > > > Cheers,
>> > > >
>> > > > Frank
>> > >
>> >
>>
>
>
Re: Logistic Regression cost function
Posted by Frank Scholten <fr...@frankscholten.nl>.
Thanks guys, I have some reading to do :-)
On Mon, Jan 13, 2014 at 10:45 PM, Ted Dunning <te...@gmail.com> wrote:
> The reference is to the web site in general.
>
> If anything, this blog is closest:
>
> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf
>
>
> On Mon, Jan 13, 2014 at 1:14 PM, Suneel Marthi <suneel_marthi@yahoo.com
> >wrote:
>
> > I think this is the one. Yes, I don't see this paper referenced in the
> > code sorry about that.
> > http://leon.bottou.org/publications/pdf/compstat-2010.pdf
> >
> >
> >
> >
> >
> >
> >
> > On Monday, January 13, 2014 3:51 PM, Frank Scholten <
> > frank@frankscholten.nl> wrote:
> >
> > Do you know which paper it is? He has quite a few publications. I don't
> see
> > any mention of one of his papers in the code. I only see
> > www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdfin
> > MixedGradient but this is something different.
> >
> >
> >
> >
> > On Mon, Jan 13, 2014 at 1:27 PM, Suneel Marthi <suneel_marthi@yahoo.com
> > >wrote:
> >
> > > Mahout's impl is based off of Leon Bottou's paper on this subject. I
> > > don't gave the link handy but it's referenced in the code or try google
> > > search
> > >
> > > Sent from my iPhone
> > >
> > > > On Jan 13, 2014, at 7:14 AM, Frank Scholten <fr...@frankscholten.nl>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > I followed the Coursera Machine Learning course quite a while ago
> and I
> > > am
> > > > trying to find out how Mahout implements the Logistic Regression cost
> > > > function in the code surrounding AbstractOnlineLogisticRegression.
> > > >
> > > > I am looking at the train method in AbstractOnlineLogisticRegression
> > and
> > > I
> > > > see online gradient descent step where the beta matrix is updated but
> > to
> > > me
> > > > its unclear how matches with the cost function described at:
> > > > http://www.holehouse.org/mlclass/06_Logistic_Regression.html
> > > >
> > > > Perhaps Mahout uses an optimized approach for that does not directly
> > map
> > > > into the formula at that link?
> > > >
> > > > Cheers,
> > > >
> > > > Frank
> > >
> >
>
Re: Logistic Regression cost function
Posted by Ted Dunning <te...@gmail.com>.
The reference is to the web site in general.
If anything, this blog is closest:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf
On Mon, Jan 13, 2014 at 1:14 PM, Suneel Marthi <su...@yahoo.com>wrote:
> I think this is the one. Yes, I don't see this paper referenced in the
> code sorry about that.
> http://leon.bottou.org/publications/pdf/compstat-2010.pdf
>
>
>
>
>
>
>
> On Monday, January 13, 2014 3:51 PM, Frank Scholten <
> frank@frankscholten.nl> wrote:
>
> Do you know which paper it is? He has quite a few publications. I don't see
> any mention of one of his papers in the code. I only see
> www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdf in
> MixedGradient but this is something different.
>
>
>
>
> On Mon, Jan 13, 2014 at 1:27 PM, Suneel Marthi <suneel_marthi@yahoo.com
> >wrote:
>
> > Mahout's impl is based off of Leon Bottou's paper on this subject. I
> > don't gave the link handy but it's referenced in the code or try google
> > search
> >
> > Sent from my iPhone
> >
> > > On Jan 13, 2014, at 7:14 AM, Frank Scholten <fr...@frankscholten.nl>
> > wrote:
> > >
> > > Hi,
> > >
> > > I followed the Coursera Machine Learning course quite a while ago and I
> > am
> > > trying to find out how Mahout implements the Logistic Regression cost
> > > function in the code surrounding AbstractOnlineLogisticRegression.
> > >
> > > I am looking at the train method in AbstractOnlineLogisticRegression
> and
> > I
> > > see online gradient descent step where the beta matrix is updated but
> to
> > me
> > > its unclear how matches with the cost function described at:
> > > http://www.holehouse.org/mlclass/06_Logistic_Regression.html
> > >
> > > Perhaps Mahout uses an optimized approach for that does not directly
> map
> > > into the formula at that link?
> > >
> > > Cheers,
> > >
> > > Frank
> >
>
Re: Logistic Regression cost function
Posted by Suneel Marthi <su...@yahoo.com>.
I think this is the one. Yes, I don't see this paper referenced in the code sorry about that.
http://leon.bottou.org/publications/pdf/compstat-2010.pdf
On Monday, January 13, 2014 3:51 PM, Frank Scholten <fr...@frankscholten.nl> wrote:
Do you know which paper it is? He has quite a few publications. I don't see
any mention of one of his papers in the code. I only see
www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdf in
MixedGradient but this is something different.
On Mon, Jan 13, 2014 at 1:27 PM, Suneel Marthi <su...@yahoo.com>wrote:
> Mahout's impl is based off of Leon Bottou's paper on this subject. I
> don't gave the link handy but it's referenced in the code or try google
> search
>
> Sent from my iPhone
>
> > On Jan 13, 2014, at 7:14 AM, Frank Scholten <fr...@frankscholten.nl>
> wrote:
> >
> > Hi,
> >
> > I followed the Coursera Machine Learning course quite a while ago and I
> am
> > trying to find out how Mahout implements the Logistic Regression cost
> > function in the code surrounding AbstractOnlineLogisticRegression.
> >
> > I am looking at the train method in AbstractOnlineLogisticRegression and
> I
> > see online gradient descent step where the beta matrix is updated but to
> me
> > its unclear how matches with the cost function described at:
> > http://www.holehouse.org/mlclass/06_Logistic_Regression.html
> >
> > Perhaps Mahout uses an optimized approach for that does not directly map
> > into the formula at that link?
> >
> > Cheers,
> >
> > Frank
>
Re: Logistic Regression cost function
Posted by Suneel Marthi <su...@yahoo.com>.
Komarek paper is not what the implementation is based off of. I once was chided by Ted on the LinkedIn Mahout forums for quoting the Komarek paper (more than 2 years ago).
Here's the link to leon bottou's paper - http://leon.bottou.org/publications/pdf/compstat-2010.pdf
On Monday, January 13, 2014 4:27 PM, Tim Smith <ti...@hotmail.com> wrote:
There is a link on http://mahout.apache.org/users/classification/logistic-regression.html to the following paper: http://www.autonlab.org/autonweb/14709/version/4/part/5/data/komarek:lr_thesis.pdf?branch=main&language=en
> Date: Mon, 13 Jan 2014 20:58:28 +0100
> Subject: Re: Logistic Regression cost function
> From: frank@frankscholten.nl
> To: user@mahout.apache.org
>
> Do you know which paper it is? He has quite a few publications. I don't see
> any mention of one of his papers in the code. I only see
> www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdf in
> MixedGradient but this is something different.
>
>
>
> On Mon, Jan 13, 2014 at 1:27 PM, Suneel Marthi <su...@yahoo.com>wrote:
>
> > Mahout's impl is based off of Leon Bottou's paper on this subject. I
> > don't gave the link handy but it's referenced in the code or try google
> > search
> >
> > Sent from my iPhone
> >
> > > On Jan 13, 2014, at 7:14 AM, Frank Scholten <fr...@frankscholten.nl>
> > wrote:
> > >
> > > Hi,
> > >
> > > I followed the Coursera Machine Learning course quite a while ago and I
> > am
> > > trying to find out how Mahout implements the Logistic Regression cost
> > > function in the code surrounding AbstractOnlineLogisticRegression.
> > >
> > > I am looking at the train method in AbstractOnlineLogisticRegression and
> > I
> > > see online gradient descent step where the beta matrix is updated but to
> > me
> > > its unclear how matches with the cost function described at:
> > > http://www.holehouse.org/mlclass/06_Logistic_Regression.html
> > >
> > > Perhaps Mahout uses an optimized approach for that does not directly map
> > > into the formula at that link?
> > >
> > > Cheers,
> > >
> > > Frank
> >
RE: Logistic Regression cost function
Posted by Tim Smith <ti...@hotmail.com>.
There is a link on http://mahout.apache.org/users/classification/logistic-regression.html to the following paper: http://www.autonlab.org/autonweb/14709/version/4/part/5/data/komarek:lr_thesis.pdf?branch=main&language=en
> Date: Mon, 13 Jan 2014 20:58:28 +0100
> Subject: Re: Logistic Regression cost function
> From: frank@frankscholten.nl
> To: user@mahout.apache.org
>
> Do you know which paper it is? He has quite a few publications. I don't see
> any mention of one of his papers in the code. I only see
> www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdf in
> MixedGradient but this is something different.
>
>
>
> On Mon, Jan 13, 2014 at 1:27 PM, Suneel Marthi <su...@yahoo.com>wrote:
>
> > Mahout's impl is based off of Leon Bottou's paper on this subject. I
> > don't gave the link handy but it's referenced in the code or try google
> > search
> >
> > Sent from my iPhone
> >
> > > On Jan 13, 2014, at 7:14 AM, Frank Scholten <fr...@frankscholten.nl>
> > wrote:
> > >
> > > Hi,
> > >
> > > I followed the Coursera Machine Learning course quite a while ago and I
> > am
> > > trying to find out how Mahout implements the Logistic Regression cost
> > > function in the code surrounding AbstractOnlineLogisticRegression.
> > >
> > > I am looking at the train method in AbstractOnlineLogisticRegression and
> > I
> > > see online gradient descent step where the beta matrix is updated but to
> > me
> > > its unclear how matches with the cost function described at:
> > > http://www.holehouse.org/mlclass/06_Logistic_Regression.html
> > >
> > > Perhaps Mahout uses an optimized approach for that does not directly map
> > > into the formula at that link?
> > >
> > > Cheers,
> > >
> > > Frank
> >
Re: Logistic Regression cost function
Posted by Frank Scholten <fr...@frankscholten.nl>.
Do you know which paper it is? He has quite a few publications. I don't see
any mention of one of his papers in the code. I only see
www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdf in
MixedGradient but this is something different.
On Mon, Jan 13, 2014 at 1:27 PM, Suneel Marthi <su...@yahoo.com>wrote:
> Mahout's impl is based off of Leon Bottou's paper on this subject. I
> don't gave the link handy but it's referenced in the code or try google
> search
>
> Sent from my iPhone
>
> > On Jan 13, 2014, at 7:14 AM, Frank Scholten <fr...@frankscholten.nl>
> wrote:
> >
> > Hi,
> >
> > I followed the Coursera Machine Learning course quite a while ago and I
> am
> > trying to find out how Mahout implements the Logistic Regression cost
> > function in the code surrounding AbstractOnlineLogisticRegression.
> >
> > I am looking at the train method in AbstractOnlineLogisticRegression and
> I
> > see online gradient descent step where the beta matrix is updated but to
> me
> > its unclear how matches with the cost function described at:
> > http://www.holehouse.org/mlclass/06_Logistic_Regression.html
> >
> > Perhaps Mahout uses an optimized approach for that does not directly map
> > into the formula at that link?
> >
> > Cheers,
> >
> > Frank
>
Re: Logistic Regression cost function
Posted by Suneel Marthi <su...@yahoo.com>.
Mahout's impl is based off of Leon Bottou's paper on this subject. I don't gave the link handy but it's referenced in the code or try google search
Sent from my iPhone
> On Jan 13, 2014, at 7:14 AM, Frank Scholten <fr...@frankscholten.nl> wrote:
>
> Hi,
>
> I followed the Coursera Machine Learning course quite a while ago and I am
> trying to find out how Mahout implements the Logistic Regression cost
> function in the code surrounding AbstractOnlineLogisticRegression.
>
> I am looking at the train method in AbstractOnlineLogisticRegression and I
> see online gradient descent step where the beta matrix is updated but to me
> its unclear how matches with the cost function described at:
> http://www.holehouse.org/mlclass/06_Logistic_Regression.html
>
> Perhaps Mahout uses an optimized approach for that does not directly map
> into the formula at that link?
>
> Cheers,
>
> Frank