You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opennlp.apache.org by Joern Kottmann <ko...@gmail.com> on 2017/02/06 17:40:32 UTC
Name Finder trainer default settings
Hello all,
I would like to propose to switch the default training algorithm from
maxent gis to perceptron for the Name Finder. In all the data sets I
tried perceptron performs better than maxent gis and I believe that
would be a much more sensible default.
A user can always override the default by providing the algorithms
parameter for training.
What do you think?
J�rn
Re: Name Finder trainer default settings
Posted by Rodrigo Agerri <ro...@ehu.eus>.
+1
R
On Mon, Feb 6, 2017 at 7:17 PM, William Colen <wi...@gmail.com>
wrote:
> +1
> The times I tried, it was always better with Perceptron cutoff 0.
>
> 2017-02-06 15:40 GMT-02:00 Joern Kottmann <ko...@gmail.com>:
>
> > Hello all,
> >
> > I would like to propose to switch the default training algorithm from
> > maxent gis to perceptron for the Name Finder. In all the data sets I
> > tried perceptron performs better than maxent gis and I believe that
> > would be a much more sensible default.
> >
> > A user can always override the default by providing the algorithms
> > parameter for training.
> >
> > What do you think?
> >
> > Jörn
> >
> >
>
Re: Name Finder trainer default settings
Posted by William Colen <wi...@gmail.com>.
+1
The times I tried, it was always better with Perceptron cutoff 0.
2017-02-06 15:40 GMT-02:00 Joern Kottmann <ko...@gmail.com>:
> Hello all,
>
> I would like to propose to switch the default training algorithm from
> maxent gis to perceptron for the Name Finder. In all the data sets I
> tried perceptron performs better than maxent gis and I believe that
> would be a much more sensible default.
>
> A user can always override the default by providing the algorithms
> parameter for training.
>
> What do you think?
>
> Jörn
>
>
Re: Name Finder trainer default settings
Posted by "Russ, Daniel (NIH/CIT) [E]" <dr...@mail.nih.gov>.
Very interesting… the difference is a lot less than I thought.
1. Ratinov and Roth (2009): 90.57 F1 Perceptron
2. Passos et al. (2009): 90.90 F1 CRF
Seeing this numbers get my +1 on Jörn’s original comment. I was under the impressions that CRFs did a lot better (Myth busted).
I completely agree that the feature selection is the most important part of the machine learning process, and that deep learning is the long-term solution (if you have the data). But training with Neural Nets still requires an expert.
Thank you Rodrigo.
Daniel
On 2/7/17, 10:15 AM, "Rodrigo Agerri" <ro...@ehu.eus> wrote:
Hi Daniel,
Previous publications suggest features are more important than learning
methods. Before last year, the trend seemed to go towards CRFs, nowadays,
it goes towards deep learning (LSTM, CNN, RNN, etc. and so on).
However, if we do a very quick review of English results for CoNLL 2003
(the most used dataset for NER), published results show that it is possible
to get as good results in NER using Perceptron as those using other
methods. With the additional benefit of training in a few minutes at most
(the OpenNLP implementation at least) and being very easy to configure.
1. Ratinov and Roth (2009): 90.57 F1 Perceptron
2. Passos et al. (2009): 90.90 F1 CRF
3. Chiu and Nichols (2015): 90.77 LSTM-CNN
4. Lample et al. (2016): 90.94 LSTM
5. Luo et al. (2016): 91.20 Extended Semi-CRF
6. Agerri and Rigau (2016): 91.36 Perceptron
7. Yang et al (2016: 91.20 F1 RNN
8. Ma and Hovy (2016): 91.21 LSTM-CNN-CRF
Having said that, +1 to adding CRF and deep learning algorithms in the
future too :)
R
On Tue, Feb 7, 2017 at 3:42 PM, Russ, Daniel (NIH/CIT) [E] <
druss@mail.nih.gov> wrote:
> Hi Jörn,
>
>
>
> I think the best entity recognition systems use CRF’s. At some point
> we might want to consider adding them. As you know, ME classifiers suffer
> from label bias problem (see Lafferty et. al<http://repository.upenn.
> edu/cgi/viewcontent.cgi?article=1162&context=cis_papers>.) CRF’s deal
> with that issue. I believe that perceptrons suffer from the same problem.
> If you think the results are better, I have no problem. I think that our
> long-term goal should be to add a CRF, and make it the default for the
> NameFinder.
>
>
>
> Daniel
>
>
>
>
>
> On 2/6/17, 12:40 PM, "Joern Kottmann" <ko...@gmail.com> wrote:
>
>
>
> Hello all,
>
>
>
> I would like to propose to switch the default training algorithm from
>
> maxent gis to perceptron for the Name Finder. In all the data sets I
>
> tried perceptron performs better than maxent gis and I believe that
>
> would be a much more sensible default.
>
>
>
> A user can always override the default by providing the algorithms
>
> parameter for training.
>
>
>
> What do you think?
>
>
>
> Jörn
>
>
>
>
>
Re: Name Finder trainer default settings
Posted by Rodrigo Agerri <ro...@ehu.eus>.
Hi Daniel,
Previous publications suggest features are more important than learning
methods. Before last year, the trend seemed to go towards CRFs, nowadays,
it goes towards deep learning (LSTM, CNN, RNN, etc. and so on).
However, if we do a very quick review of English results for CoNLL 2003
(the most used dataset for NER), published results show that it is possible
to get as good results in NER using Perceptron as those using other
methods. With the additional benefit of training in a few minutes at most
(the OpenNLP implementation at least) and being very easy to configure.
1. Ratinov and Roth (2009): 90.57 F1 Perceptron
2. Passos et al. (2009): 90.90 F1 CRF
3. Chiu and Nichols (2015): 90.77 LSTM-CNN
4. Lample et al. (2016): 90.94 LSTM
5. Luo et al. (2016): 91.20 Extended Semi-CRF
6. Agerri and Rigau (2016): 91.36 Perceptron
7. Yang et al (2016: 91.20 F1 RNN
8. Ma and Hovy (2016): 91.21 LSTM-CNN-CRF
Having said that, +1 to adding CRF and deep learning algorithms in the
future too :)
R
On Tue, Feb 7, 2017 at 3:42 PM, Russ, Daniel (NIH/CIT) [E] <
druss@mail.nih.gov> wrote:
> Hi Jörn,
>
>
>
> I think the best entity recognition systems use CRF’s. At some point
> we might want to consider adding them. As you know, ME classifiers suffer
> from label bias problem (see Lafferty et. al<http://repository.upenn.
> edu/cgi/viewcontent.cgi?article=1162&context=cis_papers>.) CRF’s deal
> with that issue. I believe that perceptrons suffer from the same problem.
> If you think the results are better, I have no problem. I think that our
> long-term goal should be to add a CRF, and make it the default for the
> NameFinder.
>
>
>
> Daniel
>
>
>
>
>
> On 2/6/17, 12:40 PM, "Joern Kottmann" <ko...@gmail.com> wrote:
>
>
>
> Hello all,
>
>
>
> I would like to propose to switch the default training algorithm from
>
> maxent gis to perceptron for the Name Finder. In all the data sets I
>
> tried perceptron performs better than maxent gis and I believe that
>
> would be a much more sensible default.
>
>
>
> A user can always override the default by providing the algorithms
>
> parameter for training.
>
>
>
> What do you think?
>
>
>
> Jörn
>
>
>
>
>
Re: Name Finder trainer default settings
Posted by Joern Kottmann <ko...@gmail.com>.
We actually can train a CRF from Mallet with the existing infrastructure,
and the code should still work (maybe there are minor issues, who knows). I
tried that but just couldn't get better results. We should maybe try to get
this code (mallet-addon) into a good shape again and then see what the
issues could be.
There might be some folks out there who want to do this for research or
other cases, I believe one of my issues were bad hyper params.
With perceptron you can get good results, quickly out of the box.
Jörn
On Tue, Feb 7, 2017 at 3:59 PM, Russ, Daniel (NIH/CIT) [E] <
druss@mail.nih.gov> wrote:
> It would be interesting to compare the results of OpenNLP’s perceptron
> trained models, GIS trained models, and a vanilla CRF implementation (i.e.
> not specifically trained for a task). We can make a better decision on if
> we should spend the effort to implement a CRF. Every once in a while we
> see people ask “what can I do?”. Maybe the answer should be… given an
> ObjectStream<Event> or DataIndexer, train a CRFModel that extends
> AbstractModel. Your training class must extend AbstractEventTrainer and we
> serializable using AbstractModelWriter
>
> Just my 2 cents.
> Daniel
>
> On 2/7/17, 9:51 AM, "Damiano Porta" <da...@gmail.com> wrote:
>
> I have good results with perceptron, but +1 for CRF
>
> 2017-02-07 15:42 GMT+01:00 Russ, Daniel (NIH/CIT) [E] <
> druss@mail.nih.gov>:
>
> > Hi Jörn,
> >
> >
> >
> > I think the best entity recognition systems use CRF’s. At some
> point
> > we might want to consider adding them. As you know, ME classifiers
> suffer
> > from label bias problem (see Lafferty et. al<http://repository.upenn
> .
> > edu/cgi/viewcontent.cgi?article=1162&context=cis_papers>.) CRF’s
> deal
> > with that issue. I believe that perceptrons suffer from the same
> problem.
> > If you think the results are better, I have no problem. I think
> that our
> > long-term goal should be to add a CRF, and make it the default for
> the
> > NameFinder.
> >
> >
> >
> > Daniel
> >
> >
> >
> >
> >
> > On 2/6/17, 12:40 PM, "Joern Kottmann" <ko...@gmail.com> wrote:
> >
> >
> >
> > Hello all,
> >
> >
> >
> > I would like to propose to switch the default training algorithm
> from
> >
> > maxent gis to perceptron for the Name Finder. In all the data
> sets I
> >
> > tried perceptron performs better than maxent gis and I believe
> that
> >
> > would be a much more sensible default.
> >
> >
> >
> > A user can always override the default by providing the
> algorithms
> >
> > parameter for training.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Jörn
> >
> >
> >
> >
> >
>
>
>
Re: Name Finder trainer default settings
Posted by "Russ, Daniel (NIH/CIT) [E]" <dr...@mail.nih.gov>.
It would be interesting to compare the results of OpenNLP’s perceptron trained models, GIS trained models, and a vanilla CRF implementation (i.e. not specifically trained for a task). We can make a better decision on if we should spend the effort to implement a CRF. Every once in a while we see people ask “what can I do?”. Maybe the answer should be… given an ObjectStream<Event> or DataIndexer, train a CRFModel that extends AbstractModel. Your training class must extend AbstractEventTrainer and we serializable using AbstractModelWriter
Just my 2 cents.
Daniel
On 2/7/17, 9:51 AM, "Damiano Porta" <da...@gmail.com> wrote:
I have good results with perceptron, but +1 for CRF
2017-02-07 15:42 GMT+01:00 Russ, Daniel (NIH/CIT) [E] <dr...@mail.nih.gov>:
> Hi Jörn,
>
>
>
> I think the best entity recognition systems use CRF’s. At some point
> we might want to consider adding them. As you know, ME classifiers suffer
> from label bias problem (see Lafferty et. al<http://repository.upenn.
> edu/cgi/viewcontent.cgi?article=1162&context=cis_papers>.) CRF’s deal
> with that issue. I believe that perceptrons suffer from the same problem.
> If you think the results are better, I have no problem. I think that our
> long-term goal should be to add a CRF, and make it the default for the
> NameFinder.
>
>
>
> Daniel
>
>
>
>
>
> On 2/6/17, 12:40 PM, "Joern Kottmann" <ko...@gmail.com> wrote:
>
>
>
> Hello all,
>
>
>
> I would like to propose to switch the default training algorithm from
>
> maxent gis to perceptron for the Name Finder. In all the data sets I
>
> tried perceptron performs better than maxent gis and I believe that
>
> would be a much more sensible default.
>
>
>
> A user can always override the default by providing the algorithms
>
> parameter for training.
>
>
>
> What do you think?
>
>
>
> Jörn
>
>
>
>
>
Re: Name Finder trainer default settings
Posted by Damiano Porta <da...@gmail.com>.
I have good results with perceptron, but +1 for CRF
2017-02-07 15:42 GMT+01:00 Russ, Daniel (NIH/CIT) [E] <dr...@mail.nih.gov>:
> Hi Jörn,
>
>
>
> I think the best entity recognition systems use CRF’s. At some point
> we might want to consider adding them. As you know, ME classifiers suffer
> from label bias problem (see Lafferty et. al<http://repository.upenn.
> edu/cgi/viewcontent.cgi?article=1162&context=cis_papers>.) CRF’s deal
> with that issue. I believe that perceptrons suffer from the same problem.
> If you think the results are better, I have no problem. I think that our
> long-term goal should be to add a CRF, and make it the default for the
> NameFinder.
>
>
>
> Daniel
>
>
>
>
>
> On 2/6/17, 12:40 PM, "Joern Kottmann" <ko...@gmail.com> wrote:
>
>
>
> Hello all,
>
>
>
> I would like to propose to switch the default training algorithm from
>
> maxent gis to perceptron for the Name Finder. In all the data sets I
>
> tried perceptron performs better than maxent gis and I believe that
>
> would be a much more sensible default.
>
>
>
> A user can always override the default by providing the algorithms
>
> parameter for training.
>
>
>
> What do you think?
>
>
>
> Jörn
>
>
>
>
>
Re: Name Finder trainer default settings
Posted by "Russ, Daniel (NIH/CIT) [E]" <dr...@mail.nih.gov>.
Hi Jörn,
I think the best entity recognition systems use CRF’s. At some point we might want to consider adding them. As you know, ME classifiers suffer from label bias problem (see Lafferty et. al<http://repository.upenn.edu/cgi/viewcontent.cgi?article=1162&context=cis_papers>.) CRF’s deal with that issue. I believe that perceptrons suffer from the same problem. If you think the results are better, I have no problem. I think that our long-term goal should be to add a CRF, and make it the default for the NameFinder.
Daniel
On 2/6/17, 12:40 PM, "Joern Kottmann" <ko...@gmail.com> wrote:
Hello all,
I would like to propose to switch the default training algorithm from
maxent gis to perceptron for the Name Finder. In all the data sets I
tried perceptron performs better than maxent gis and I believe that
would be a much more sensible default.
A user can always override the default by providing the algorithms
parameter for training.
What do you think?
Jörn