You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opennlp.apache.org by Saurabh Jain <sa...@gmail.com> on 2017/08/21 14:08:26 UTC
Early stopping NameFinderME
Hi All
How can we use early stopping while training/crossvalidating custom data
with NameFinder ? What I want if change in likelihood value or accuracy of
model is less than 0.05 between two steps (differ by 5 i.e compare x+5 step
output with x step) then training should stop. I could not find anything
regarding this in documentation. Can some one please help ?
--
*Thanks & Regards*
*Saurabh Jain *
*AI Developer*
*Active Intelligence *
*"*
*To do a thing yesterday was the best time . Second best time is today .” *
Re: Early stopping NameFinderME
Posted by Dan Russ <da...@gmail.com>.
can't you set the number of iterations in the training properties
On Aug 24, 2017 4:48 AM, "Joern Kottmann" <ko...@gmail.com> wrote:
> You are the first one who ever asked this question. I think we have this as
> an option already on the gis trainer but it is not exposed all the way
> through.
>
> Please open a jira and I can look at it next week.
>
> Jörn
>
> On Aug 21, 2017 5:11 PM, "Saurabh Jain" <sa...@gmail.com> wrote:
>
> > Hi All
> >
> > How can we use early stopping while training/crossvalidating custom data
> > with NameFinder ? What I want if change in likelihood value or accuracy
> of
> > model is less than 0.05 between two steps (differ by 5 i.e compare x+5
> step
> > output with x step) then training should stop. I could not find anything
> > regarding this in documentation. Can some one please help ?
> >
> > --
> > *Thanks & Regards*
> >
> >
> > *Saurabh Jain *
> > *AI Developer*
> >
> > *Active Intelligence *
> >
> > *"*
> > *To do a thing yesterday was the best time . Second best time is today
> .” *
> >
>
Re: Early stopping NameFinderME
Posted by Saurabh Jain <sa...@gmail.com>.
Hi Jorn
Let me explain you what I am trying to do. I have 3 categories and need to
create 3 different NameFinder model for each category. I train 3 models in
sequence (although each one is independent) with training step 1000. While
training on a specific category, lets say first category, if change in
likelihood values between two steps (differ by 5) is less than a threshold
then I would like stop training at that step and start training next
category and so on.
Your question: *can't you set the number of iterations in the training
properties*
No, I can't
@Joern
Issue has been created on Jira
https://issues.apache.org/jira/browse/OPENNLP-1123
On Sat, Aug 26, 2017 at 4:37 AM, Daniel Russ <da...@gmail.com> wrote:
> Jörn,
>
> Currently, GISTrainer has a private static final variable LLThreshold,
> which controls if the change in the log likelihood between two iterations
> is too small. We could make this parameter. I am concerned about using the
> accuracy to train the model. If we use accuracy, the weight space may be
> flat.
>
> Saurabh, you use the term “early stopping”. In deep learning, early
> stopping is used to prevent overtraining and improve generalization to
> unseen data. I am not sure early stopping serves the same purpose with GIS
> training. Does anyone know if early stopping improves generalization for a
> maxent problem?
>
> Daniel
>
> > On Aug 24, 2017, at 4:48 AM, Joern Kottmann <ko...@gmail.com> wrote:
> >
> > You are the first one who ever asked this question. I think we have this
> as
> > an option already on the gis trainer but it is not exposed all the way
> > through.
> >
> > Please open a jira and I can look at it next week.
> >
> > Jörn
> >
> > On Aug 21, 2017 5:11 PM, "Saurabh Jain" <sa...@gmail.com>
> wrote:
> >
> >> Hi All
> >>
> >> How can we use early stopping while training/crossvalidating custom data
> >> with NameFinder ? What I want if change in likelihood value or accuracy
> of
> >> model is less than 0.05 between two steps (differ by 5 i.e compare x+5
> step
> >> output with x step) then training should stop. I could not find anything
> >> regarding this in documentation. Can some one please help ?
> >>
> >> --
> >> *Thanks & Regards*
> >>
> >>
> >> *Saurabh Jain *
> >> *AI Developer*
> >>
> >> *Active Intelligence *
> >>
> >> *"*
> >> *To do a thing yesterday was the best time . Second best time is today
> .” *
> >>
>
>
--
*Thanks & Regards*
*Saurabh Jain *
*AI Developer*
*Active Intelligence *
*"*
*To do a thing yesterday was the best time . Second best time is today .” *
Re: Early stopping NameFinderME
Posted by Joern Kottmann <ko...@gmail.com>.
Go ahead and do the change. Otherwise I can work on it tomorrow.
Jörn
On Tue, Aug 29, 2017 at 4:38 PM, Dan Russ <da...@gmail.com> wrote:
> Hi Jörn,
> I don’t see a problem with it. Make sure the default is set to the current value. Are you making the fix? I could get to it later tonight.
> Daniel
>
>> On Aug 29, 2017, at 10:32 AM, Joern Kottmann <ko...@gmail.com> wrote:
>>
>> Hi Daniel,
>>
>> do you see any issue if we expose LLThreshold and allow the user to
>> change it via training parameters?
>>
>> Jörn
>>
>> On Sat, Aug 26, 2017 at 1:07 AM, Daniel Russ <da...@gmail.com> wrote:
>>> Jörn,
>>>
>>> Currently, GISTrainer has a private static final variable LLThreshold, which controls if the change in the log likelihood between two iterations is too small. We could make this parameter. I am concerned about using the accuracy to train the model. If we use accuracy, the weight space may be flat.
>>>
>>> Saurabh, you use the term “early stopping”. In deep learning, early stopping is used to prevent overtraining and improve generalization to unseen data. I am not sure early stopping serves the same purpose with GIS training. Does anyone know if early stopping improves generalization for a maxent problem?
>>>
>>> Daniel
>>>
>>>> On Aug 24, 2017, at 4:48 AM, Joern Kottmann <ko...@gmail.com> wrote:
>>>>
>>>> You are the first one who ever asked this question. I think we have this as
>>>> an option already on the gis trainer but it is not exposed all the way
>>>> through.
>>>>
>>>> Please open a jira and I can look at it next week.
>>>>
>>>> Jörn
>>>>
>>>> On Aug 21, 2017 5:11 PM, "Saurabh Jain" <sa...@gmail.com> wrote:
>>>>
>>>>> Hi All
>>>>>
>>>>> How can we use early stopping while training/crossvalidating custom data
>>>>> with NameFinder ? What I want if change in likelihood value or accuracy of
>>>>> model is less than 0.05 between two steps (differ by 5 i.e compare x+5 step
>>>>> output with x step) then training should stop. I could not find anything
>>>>> regarding this in documentation. Can some one please help ?
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>>
>>>>> *Saurabh Jain *
>>>>> *AI Developer*
>>>>>
>>>>> *Active Intelligence *
>>>>>
>>>>> *"*
>>>>> *To do a thing yesterday was the best time . Second best time is today .” *
>>>>>
>>>
>
Re: Early stopping NameFinderME
Posted by Dan Russ <da...@gmail.com>.
Hi Jörn,
I don’t see a problem with it. Make sure the default is set to the current value. Are you making the fix? I could get to it later tonight.
Daniel
> On Aug 29, 2017, at 10:32 AM, Joern Kottmann <ko...@gmail.com> wrote:
>
> Hi Daniel,
>
> do you see any issue if we expose LLThreshold and allow the user to
> change it via training parameters?
>
> Jörn
>
> On Sat, Aug 26, 2017 at 1:07 AM, Daniel Russ <da...@gmail.com> wrote:
>> Jörn,
>>
>> Currently, GISTrainer has a private static final variable LLThreshold, which controls if the change in the log likelihood between two iterations is too small. We could make this parameter. I am concerned about using the accuracy to train the model. If we use accuracy, the weight space may be flat.
>>
>> Saurabh, you use the term “early stopping”. In deep learning, early stopping is used to prevent overtraining and improve generalization to unseen data. I am not sure early stopping serves the same purpose with GIS training. Does anyone know if early stopping improves generalization for a maxent problem?
>>
>> Daniel
>>
>>> On Aug 24, 2017, at 4:48 AM, Joern Kottmann <ko...@gmail.com> wrote:
>>>
>>> You are the first one who ever asked this question. I think we have this as
>>> an option already on the gis trainer but it is not exposed all the way
>>> through.
>>>
>>> Please open a jira and I can look at it next week.
>>>
>>> Jörn
>>>
>>> On Aug 21, 2017 5:11 PM, "Saurabh Jain" <sa...@gmail.com> wrote:
>>>
>>>> Hi All
>>>>
>>>> How can we use early stopping while training/crossvalidating custom data
>>>> with NameFinder ? What I want if change in likelihood value or accuracy of
>>>> model is less than 0.05 between two steps (differ by 5 i.e compare x+5 step
>>>> output with x step) then training should stop. I could not find anything
>>>> regarding this in documentation. Can some one please help ?
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>>
>>>> *Saurabh Jain *
>>>> *AI Developer*
>>>>
>>>> *Active Intelligence *
>>>>
>>>> *"*
>>>> *To do a thing yesterday was the best time . Second best time is today .” *
>>>>
>>
Re: Early stopping NameFinderME
Posted by Joern Kottmann <ko...@gmail.com>.
Hi Daniel,
do you see any issue if we expose LLThreshold and allow the user to
change it via training parameters?
Jörn
On Sat, Aug 26, 2017 at 1:07 AM, Daniel Russ <da...@gmail.com> wrote:
> Jörn,
>
> Currently, GISTrainer has a private static final variable LLThreshold, which controls if the change in the log likelihood between two iterations is too small. We could make this parameter. I am concerned about using the accuracy to train the model. If we use accuracy, the weight space may be flat.
>
> Saurabh, you use the term “early stopping”. In deep learning, early stopping is used to prevent overtraining and improve generalization to unseen data. I am not sure early stopping serves the same purpose with GIS training. Does anyone know if early stopping improves generalization for a maxent problem?
>
> Daniel
>
>> On Aug 24, 2017, at 4:48 AM, Joern Kottmann <ko...@gmail.com> wrote:
>>
>> You are the first one who ever asked this question. I think we have this as
>> an option already on the gis trainer but it is not exposed all the way
>> through.
>>
>> Please open a jira and I can look at it next week.
>>
>> Jörn
>>
>> On Aug 21, 2017 5:11 PM, "Saurabh Jain" <sa...@gmail.com> wrote:
>>
>>> Hi All
>>>
>>> How can we use early stopping while training/crossvalidating custom data
>>> with NameFinder ? What I want if change in likelihood value or accuracy of
>>> model is less than 0.05 between two steps (differ by 5 i.e compare x+5 step
>>> output with x step) then training should stop. I could not find anything
>>> regarding this in documentation. Can some one please help ?
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>>
>>> *Saurabh Jain *
>>> *AI Developer*
>>>
>>> *Active Intelligence *
>>>
>>> *"*
>>> *To do a thing yesterday was the best time . Second best time is today .” *
>>>
>
Re: Early stopping NameFinderME
Posted by Daniel Russ <da...@gmail.com>.
Jörn,
Currently, GISTrainer has a private static final variable LLThreshold, which controls if the change in the log likelihood between two iterations is too small. We could make this parameter. I am concerned about using the accuracy to train the model. If we use accuracy, the weight space may be flat.
Saurabh, you use the term “early stopping”. In deep learning, early stopping is used to prevent overtraining and improve generalization to unseen data. I am not sure early stopping serves the same purpose with GIS training. Does anyone know if early stopping improves generalization for a maxent problem?
Daniel
> On Aug 24, 2017, at 4:48 AM, Joern Kottmann <ko...@gmail.com> wrote:
>
> You are the first one who ever asked this question. I think we have this as
> an option already on the gis trainer but it is not exposed all the way
> through.
>
> Please open a jira and I can look at it next week.
>
> Jörn
>
> On Aug 21, 2017 5:11 PM, "Saurabh Jain" <sa...@gmail.com> wrote:
>
>> Hi All
>>
>> How can we use early stopping while training/crossvalidating custom data
>> with NameFinder ? What I want if change in likelihood value or accuracy of
>> model is less than 0.05 between two steps (differ by 5 i.e compare x+5 step
>> output with x step) then training should stop. I could not find anything
>> regarding this in documentation. Can some one please help ?
>>
>> --
>> *Thanks & Regards*
>>
>>
>> *Saurabh Jain *
>> *AI Developer*
>>
>> *Active Intelligence *
>>
>> *"*
>> *To do a thing yesterday was the best time . Second best time is today .” *
>>
Re: Early stopping NameFinderME
Posted by Joern Kottmann <ko...@gmail.com>.
You are the first one who ever asked this question. I think we have this as
an option already on the gis trainer but it is not exposed all the way
through.
Please open a jira and I can look at it next week.
Jörn
On Aug 21, 2017 5:11 PM, "Saurabh Jain" <sa...@gmail.com> wrote:
> Hi All
>
> How can we use early stopping while training/crossvalidating custom data
> with NameFinder ? What I want if change in likelihood value or accuracy of
> model is less than 0.05 between two steps (differ by 5 i.e compare x+5 step
> output with x step) then training should stop. I could not find anything
> regarding this in documentation. Can some one please help ?
>
> --
> *Thanks & Regards*
>
>
> *Saurabh Jain *
> *AI Developer*
>
> *Active Intelligence *
>
> *"*
> *To do a thing yesterday was the best time . Second best time is today .” *
>