You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by Raj Kiran <ra...@pb.com> on 2014/11/04 19:05:28 UTC
Corpora used for training OpenNLP english models
Hi All,
We want to use OpenNLP for NER and other capabilities in a commercial software (English only). It looks like existing OpenNLP english models available at sourceforge might have some license restriction. Is there any information available on the source corpora used for training existing OpenNLP english models ?
Apart from purchasing the source corpora, this information would help us to enhance the existing models by adding more training data.
Thanks and Regards,
Raj
________________________________
RE: Corpora used for training OpenNLP english models
Posted by Raj Kiran <ra...@pb.com>.
Thanks, Ismael , I checked this page too but no information is available for english models except for chunker.
For example the name finder , description field contains "<entity name> name finder model ", For sentence and token its "Trained on opennlp training data."
Raj
________________________________________
From: Ismael Hasan [ismael.hasan.romero@gmail.com]
Sent: Wednesday, November 05, 2014 12:57 AM
To: users@opennlp.apache.org
Subject: Re: Corpora used for training OpenNLP english models
The following page indicates the name of the different corpora used to
build the models:
http://opennlp.sourceforge.net/models-1.5/
I hope this helps,
Ismael.
2014-11-04 19:05 GMT+01:00 Raj Kiran <ra...@pb.com>:
> Hi All,
>
> We want to use OpenNLP for NER and other capabilities in a commercial
> software (English only). It looks like existing OpenNLP english models
> available at sourceforge might have some license restriction. Is there any
> information available on the source corpora used for training existing
> OpenNLP english models ?
>
> Apart from purchasing the source corpora, this information would help us
> to enhance the existing models by adding more training data.
>
> Thanks and Regards,
> Raj
>
>
>
> ________________________________
>
>
--
Ismael Hasan Romero
http://www.linkedin.com/in/ismaelhasan/en
________________________________
Re: Corpora used for training OpenNLP english models
Posted by Ismael Hasan <is...@gmail.com>.
The following page indicates the name of the different corpora used to
build the models:
http://opennlp.sourceforge.net/models-1.5/
I hope this helps,
Ismael.
2014-11-04 19:05 GMT+01:00 Raj Kiran <ra...@pb.com>:
> Hi All,
>
> We want to use OpenNLP for NER and other capabilities in a commercial
> software (English only). It looks like existing OpenNLP english models
> available at sourceforge might have some license restriction. Is there any
> information available on the source corpora used for training existing
> OpenNLP english models ?
>
> Apart from purchasing the source corpora, this information would help us
> to enhance the existing models by adding more training data.
>
> Thanks and Regards,
> Raj
>
>
>
> ________________________________
>
>
--
Ismael Hasan Romero
http://www.linkedin.com/in/ismaelhasan/en
RE: Corpora used for training OpenNLP english models
Posted by Raj Kiran <ra...@pb.com>.
Thanks Jörn,
The corpora contains document from variety of sources and it was released last year. Should fit in our case , will check this too.
Raj
-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com]
Sent: Monday, November 10, 2014 2:10 PM
To: users@opennlp.apache.org
Subject: Re: Corpora used for training OpenNLP english models
On 11/05/2014 08:14 AM, Rodrigo Agerri wrote:
> Hi Raj,
>
> I believe that the NameFinder models were trained with MUC, but I am
> not sure. In any case, if you are going to annotate a different domain
> to that of MUC, you will better off annotating data for that domain
> because supervised approaches do not adapt well when used in other
> genres/domains.
>
The English name finder models are trained on MUC 6 / 7 plus some corrections to solve certain detection problems.
I suggest not to use MUC anymore because it is quite dated.
If you want to train name finder models which perform well I suggest to have a look at OntoNotes 4.0. We have support to train OpenNLP models directly on it.
The data is not free, we had to pay around 50 USD to get it.
There is now also a newer version 5.0:
https://catalog.ldc.upenn.edu/LDC2013T19
I guess the format of it didn't change to much, so there is a good chance it runs with the 4.0 parsing code.
HTH,
Jörn
________________________________
Re: Corpora used for training OpenNLP english models
Posted by Jörn Kottmann <ko...@gmail.com>.
On 11/05/2014 08:14 AM, Rodrigo Agerri wrote:
> Hi Raj,
>
> I believe that the NameFinder models were trained with MUC, but I am
> not sure. In any case, if you are going to annotate a different domain
> to that of MUC, you will better off annotating data for that domain
> because supervised approaches do not adapt well when used in other
> genres/domains.
>
The English name finder models are trained on MUC 6 / 7 plus some
corrections to solve
certain detection problems.
I suggest not to use MUC anymore because it is quite dated.
If you want to train name finder models which perform well I suggest to
have a look
at OntoNotes 4.0. We have support to train OpenNLP models directly on it.
The data is not free, we had to pay around 50 USD to get it.
There is now also a newer version 5.0:
https://catalog.ldc.upenn.edu/LDC2013T19
I guess the format of it didn't change to much, so there is a good
chance it runs
with the 4.0 parsing code.
HTH,
Jörn
RE: Corpora used for training OpenNLP english models
Posted by Raj Kiran <ra...@pb.com>.
Thanks for the information Rodrigo,
Will take care of this while training the new models.
Raj
-----Original Message-----
From: Rodrigo Agerri [mailto:rodrigo.agerri@ehu.es]
Sent: Wednesday, November 5, 2014 12:44 PM
To: users@opennlp.apache.org
Subject: Re: Corpora used for training OpenNLP english models
Hi Raj,
I believe that the NameFinder models were trained with MUC, but I am not sure. In any case, if you are going to annotate a different domain to that of MUC, you will better off annotating data for that domain because supervised approaches do not adapt well when used in other genres/domains.
HTH
R
On Wed, Nov 5, 2014 at 8:08 AM, Raj Kiran <ra...@pb.com> wrote:
> Hi Rodrigo,
>
> By extending model I meant, combining the base corpora (used to train existing model) with additional annotated text and retrain the model.
> Apart from licensing, this is one of the reason I am interested in knowing the source/base corpora used for training name finder models.
>
> Thanks,
> Raj
>
> -----Original Message-----
> From: Rodrigo Agerri [mailto:ragerri@apache.org]
> Sent: Wednesday, November 5, 2014 12:16 PM
> To: users@opennlp.apache.org
> Subject: Re: Corpora used for training OpenNLP english models
>
> Hi Raj,
>
> I do not know which license the models in sourceforge are distributed under. But you cannot extend the existing English models. You need to train new ones for your domain based on annotated data.
>
> Best,
>
> R
>
> On Tue, Nov 4, 2014 at 7:05 PM, Raj Kiran <ra...@pb.com> wrote:
>> Hi All,
>>
>> We want to use OpenNLP for NER and other capabilities in a commercial software (English only). It looks like existing OpenNLP english models available at sourceforge might have some license restriction. Is there any information available on the source corpora used for training existing OpenNLP english models ?
>>
>> Apart from purchasing the source corpora, this information would help us to enhance the existing models by adding more training data.
>>
>> Thanks and Regards,
>> Raj
>>
>>
>>
>> ________________________________
>>
>
> ________________________________
>
________________________________
Re: Corpora used for training OpenNLP english models
Posted by Rodrigo Agerri <ro...@ehu.es>.
Hi Raj,
I believe that the NameFinder models were trained with MUC, but I am
not sure. In any case, if you are going to annotate a different domain
to that of MUC, you will better off annotating data for that domain
because supervised approaches do not adapt well when used in other
genres/domains.
HTH
R
On Wed, Nov 5, 2014 at 8:08 AM, Raj Kiran <ra...@pb.com> wrote:
> Hi Rodrigo,
>
> By extending model I meant, combining the base corpora (used to train existing model) with additional annotated text and retrain the model.
> Apart from licensing, this is one of the reason I am interested in knowing the source/base corpora used for training name finder models.
>
> Thanks,
> Raj
>
> -----Original Message-----
> From: Rodrigo Agerri [mailto:ragerri@apache.org]
> Sent: Wednesday, November 5, 2014 12:16 PM
> To: users@opennlp.apache.org
> Subject: Re: Corpora used for training OpenNLP english models
>
> Hi Raj,
>
> I do not know which license the models in sourceforge are distributed under. But you cannot extend the existing English models. You need to train new ones for your domain based on annotated data.
>
> Best,
>
> R
>
> On Tue, Nov 4, 2014 at 7:05 PM, Raj Kiran <ra...@pb.com> wrote:
>> Hi All,
>>
>> We want to use OpenNLP for NER and other capabilities in a commercial software (English only). It looks like existing OpenNLP english models available at sourceforge might have some license restriction. Is there any information available on the source corpora used for training existing OpenNLP english models ?
>>
>> Apart from purchasing the source corpora, this information would help us to enhance the existing models by adding more training data.
>>
>> Thanks and Regards,
>> Raj
>>
>>
>>
>> ________________________________
>>
>
> ________________________________
>
RE: Corpora used for training OpenNLP english models
Posted by Raj Kiran <ra...@pb.com>.
Hi Rodrigo,
By extending model I meant, combining the base corpora (used to train existing model) with additional annotated text and retrain the model.
Apart from licensing, this is one of the reason I am interested in knowing the source/base corpora used for training name finder models.
Thanks,
Raj
-----Original Message-----
From: Rodrigo Agerri [mailto:ragerri@apache.org]
Sent: Wednesday, November 5, 2014 12:16 PM
To: users@opennlp.apache.org
Subject: Re: Corpora used for training OpenNLP english models
Hi Raj,
I do not know which license the models in sourceforge are distributed under. But you cannot extend the existing English models. You need to train new ones for your domain based on annotated data.
Best,
R
On Tue, Nov 4, 2014 at 7:05 PM, Raj Kiran <ra...@pb.com> wrote:
> Hi All,
>
> We want to use OpenNLP for NER and other capabilities in a commercial software (English only). It looks like existing OpenNLP english models available at sourceforge might have some license restriction. Is there any information available on the source corpora used for training existing OpenNLP english models ?
>
> Apart from purchasing the source corpora, this information would help us to enhance the existing models by adding more training data.
>
> Thanks and Regards,
> Raj
>
>
>
> ________________________________
>
________________________________
Re: Corpora used for training OpenNLP english models
Posted by Rodrigo Agerri <ra...@apache.org>.
Hi Raj,
I do not know which license the models in sourceforge are distributed
under. But you cannot extend the existing English models. You need to
train new ones for your domain based on annotated data.
Best,
R
On Tue, Nov 4, 2014 at 7:05 PM, Raj Kiran <ra...@pb.com> wrote:
> Hi All,
>
> We want to use OpenNLP for NER and other capabilities in a commercial software (English only). It looks like existing OpenNLP english models available at sourceforge might have some license restriction. Is there any information available on the source corpora used for training existing OpenNLP english models ?
>
> Apart from purchasing the source corpora, this information would help us to enhance the existing models by adding more training data.
>
> Thanks and Regards,
> Raj
>
>
>
> ________________________________
>