You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by PA...@homedepot.com on 2014/10/24 18:20:00 UTC
Getting started with OpenNLP
Hello all,
First off, thanks to all who contribute to this project! I've been tasked with doing some research on Apache Stanbol, which uses OpenNLP, to see if it can fill some roles in a few company projects. I've been reading about how to train a model for named entity recognition and it seems like the simplest case of "I have a list of n proper nouns, please just recognize them directly and nothing else" isn't addressed in the documentation. Is this too simple a use case? Would I be doing better to just use a simple substring match on a phrase then? I would later like to extend the model to recognize things other than just simple proper nouns, but for now, that is the simplest case I can think of.
Patrick Baggett
Online Engineer - Search Team
e: patrick_t_baggett@homedepot.com<ma...@homedepot.com>
p: +1 (214) 202-8964
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: Getting started with OpenNLP
Posted by Jörn Kottmann <ko...@gmail.com>.
The classifier will probably make mistakes from time to time. I would
try to use a threshold which
indicates that a choice is reasonable confident.
If all categories fall below that threshold you can label that item as
unkown.
Jörn
On 10/28/2014 07:19 PM, PATRICK_T_BAGGETT@homedepot.com wrote:
> Yeah, it looks like they get the same 1/n score...seems kind of hacky to depend on that, but so far does appear to work. I think I need a confidence score system that is [0...1] per category rather than normalized such that the sum of all weights is 1.0.
>
> Is there any option to do this? If not, where does one dig to make such changes?
>
> Patrick Baggett
> Online Engineer - Search Team
> e: patrick_t_baggett@homedepot.com
> p: +1 (214) 202-8964
>
> -----Original Message-----
> From: Mark G [mailto:giaconiamark@gmail.com]
> Sent: Tuesday, October 28, 2014 9:55 AM
> To: users@opennlp.apache.org
> Subject: Re: Getting started with OpenNLP
>
> I think if nothing matches the model at all each cat will have the same score associated.
>
>
>
>
> ________________________________
>
> The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
RE: Getting started with OpenNLP
Posted by PA...@homedepot.com.
Yeah, it looks like they get the same 1/n score...seems kind of hacky to depend on that, but so far does appear to work. I think I need a confidence score system that is [0...1] per category rather than normalized such that the sum of all weights is 1.0.
Is there any option to do this? If not, where does one dig to make such changes?
Patrick Baggett
Online Engineer - Search Team
e: patrick_t_baggett@homedepot.com
p: +1 (214) 202-8964
-----Original Message-----
From: Mark G [mailto:giaconiamark@gmail.com]
Sent: Tuesday, October 28, 2014 9:55 AM
To: users@opennlp.apache.org
Subject: Re: Getting started with OpenNLP
I think if nothing matches the model at all each cat will have the same score associated.
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: Getting started with OpenNLP
Posted by Mark G <gi...@gmail.com>.
I think if nothing matches the model at all each cat will have the same score associated.
> On Oct 28, 2014, at 10:03 AM, <PA...@homedepot.com> wrote:
>
> Hello All,
>
> I appreciate the advice. I did try training a larger model (800ish samples, 8 categories) and it performed better. Still, if type absolute non-sense like "asdfasdfsadf", the evaluation `opennlp Doccat <model>` must return a category -- so I'd like to be able to programmatically determine some confidence level. Perhaps I can reject all categories in the app if the confidence score is below a threshold? Is that possible right now?
>
> Thanks again for your help!
>
> Patrick Baggett
> Online Engineer - Search Team
> e: patrick_t_baggett@homedepot.com
> p: +1 (214) 202-8964
>
> -----Original Message-----
> From: Mark G [mailto:giaconiamark@gmail.com]
> Sent: Monday, October 27, 2014 7:47 PM
> To: users@opennlp.apache.org
> Subject: Re: Getting started with OpenNLP
>
> I think you bring up a good point inadvertently, I have run into this before, my use case was that I wanted a probability that the input text matched my samples for one class...sometimes you just need one.... I ended up just using a simple feature generator and just using a similarity measure. I can see a use case for a fuzzy scorer against a set of samples for only one category. I believe right now in the Doccat if you only have one category you always get a score of 1 for anything you pass in...regardless of how it matches any of the samples simply because it's the only one, which is really not so good.
>
>> On Mon, Oct 27, 2014 at 6:03 PM, Joern Kottmann <ko...@gmail.com> wrote:
>>
>> On Mon, 2014-10-27 at 19:26 +0000, PATRICK_T_BAGGETT@homedepot.com
>> wrote:
>>> So in other words, for this model, there is just one class (in a
>>> more complex example, there would be a number of classes). I trained
>>> the model and did some testing, but everything is classified as "MyClass".
>>
>> The model can only assign the classes it sees in the training data. If
>> you only have one class in your training data, then that is the only
>> class the model can assign. Actually the model always computes the
>> probability for each class, and many applications then just look for
>> the best class.
>>
>> We should probably add a warning to the trainer which says that
>> training with only one class doesn't make sense.
>>
>> I suggest that you try to train with a couple of classes, but at least
>> two.
>>
>> Here are two tips on how to create a model, maybe they are useful.
>>
>> - Make sure to use a good amount of training data. You probably need a
>> few hundred samples to get a model that somehow works.
>>
>> - And to determine how well the model works you should prepare some
>> test data to be able to evaluate on many samples and not just a few
>> hand picked ones. This can be done with the evaluation tool.
>>
>> HTH,
>> Jörn
>
> ________________________________
>
> The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
RE: Getting started with OpenNLP
Posted by PA...@homedepot.com.
Hello All,
I appreciate the advice. I did try training a larger model (800ish samples, 8 categories) and it performed better. Still, if type absolute non-sense like "asdfasdfsadf", the evaluation `opennlp Doccat <model>` must return a category -- so I'd like to be able to programmatically determine some confidence level. Perhaps I can reject all categories in the app if the confidence score is below a threshold? Is that possible right now?
Thanks again for your help!
Patrick Baggett
Online Engineer - Search Team
e: patrick_t_baggett@homedepot.com
p: +1 (214) 202-8964
-----Original Message-----
From: Mark G [mailto:giaconiamark@gmail.com]
Sent: Monday, October 27, 2014 7:47 PM
To: users@opennlp.apache.org
Subject: Re: Getting started with OpenNLP
I think you bring up a good point inadvertently, I have run into this before, my use case was that I wanted a probability that the input text matched my samples for one class...sometimes you just need one.... I ended up just using a simple feature generator and just using a similarity measure. I can see a use case for a fuzzy scorer against a set of samples for only one category. I believe right now in the Doccat if you only have one category you always get a score of 1 for anything you pass in...regardless of how it matches any of the samples simply because it's the only one, which is really not so good.
On Mon, Oct 27, 2014 at 6:03 PM, Joern Kottmann <ko...@gmail.com> wrote:
> On Mon, 2014-10-27 at 19:26 +0000, PATRICK_T_BAGGETT@homedepot.com
> wrote:
> > So in other words, for this model, there is just one class (in a
> > more complex example, there would be a number of classes). I trained
> > the model and did some testing, but everything is classified as "MyClass".
>
> The model can only assign the classes it sees in the training data. If
> you only have one class in your training data, then that is the only
> class the model can assign. Actually the model always computes the
> probability for each class, and many applications then just look for
> the best class.
>
> We should probably add a warning to the trainer which says that
> training with only one class doesn't make sense.
>
> I suggest that you try to train with a couple of classes, but at least
> two.
>
> Here are two tips on how to create a model, maybe they are useful.
>
> - Make sure to use a good amount of training data. You probably need a
> few hundred samples to get a model that somehow works.
>
> - And to determine how well the model works you should prepare some
> test data to be able to evaluate on many samples and not just a few
> hand picked ones. This can be done with the evaluation tool.
>
> HTH,
> Jörn
>
>
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: Getting started with OpenNLP
Posted by Mark G <gi...@gmail.com>.
I think you bring up a good point inadvertently, I have run into this
before, my use case was that I wanted a probability that the input text
matched my samples for one class...sometimes you just need one.... I ended
up just using a simple feature generator and just using a similarity
measure. I can see a use case for a fuzzy scorer against a set of samples
for only one category. I believe right now in the Doccat if you only have
one category you always get a score of 1 for anything you pass
in...regardless of how it matches any of the samples simply because it's
the only one, which is really not so good.
On Mon, Oct 27, 2014 at 6:03 PM, Joern Kottmann <ko...@gmail.com> wrote:
> On Mon, 2014-10-27 at 19:26 +0000, PATRICK_T_BAGGETT@homedepot.com
> wrote:
> > So in other words, for this model, there is just one class (in a more
> > complex example, there would be a number of classes). I trained the
> > model and did some testing, but everything is classified as "MyClass".
>
> The model can only assign the classes it sees in the training data. If
> you only have one class in your training data, then that is the only
> class the model can assign. Actually the model always computes the
> probability for each class, and many applications then just look for the
> best class.
>
> We should probably add a warning to the trainer which says that training
> with only one class doesn't make sense.
>
> I suggest that you try to train with a couple of classes, but at least
> two.
>
> Here are two tips on how to create a model, maybe they are useful.
>
> - Make sure to use a good amount of training data. You probably need a
> few hundred samples to get a model that somehow works.
>
> - And to determine how well the model works you should prepare some test
> data to be able to evaluate on many samples and not just a few hand
> picked ones. This can be done with the evaluation tool.
>
> HTH,
> Jörn
>
>
Re: Getting started with OpenNLP
Posted by Joern Kottmann <ko...@gmail.com>.
On Mon, 2014-10-27 at 19:26 +0000, PATRICK_T_BAGGETT@homedepot.com
wrote:
> So in other words, for this model, there is just one class (in a more
> complex example, there would be a number of classes). I trained the
> model and did some testing, but everything is classified as "MyClass".
The model can only assign the classes it sees in the training data. If
you only have one class in your training data, then that is the only
class the model can assign. Actually the model always computes the
probability for each class, and many applications then just look for the
best class.
We should probably add a warning to the trainer which says that training
with only one class doesn't make sense.
I suggest that you try to train with a couple of classes, but at least
two.
Here are two tips on how to create a model, maybe they are useful.
- Make sure to use a good amount of training data. You probably need a
few hundred samples to get a model that somehow works.
- And to determine how well the model works you should prepare some test
data to be able to evaluate on many samples and not just a few hand
picked ones. This can be done with the evaluation tool.
HTH,
Jörn
RE: Getting started with OpenNLP
Posted by PA...@homedepot.com.
Hi Jörn et al,
I've been looking at the Doccat module and I can see how it can be useful, but I'm having a bit of a hard time understanding a small detail. It seems to be unable to map to "no category". I've prepared a training file as follows:
MyClass Some Proper Noun 1
MyClass Some Proper Noun 2
MyClass Some Proper Noun 3
....
MyClass Some Proper Noun [n]
So in other words, for this model, there is just one class (in a more complex example, there would be a number of classes). I trained the model and did some testing, but everything is classified as "MyClass". Does it not have the ability to just say, "I don't know". For example, if I have a set of words w[0]- w[n], and I ask it to classify some word w* that is not equal to any words in w[0..n], must it return MyClass instead of "none of the above?"
With more categories, I expect this to be less of a problem, but ideally I'd like to be able return any of the categories found _OR_ "none of the above". Any thoughts?
Patrick Baggett
Online Engineer - Search Team
e: patrick_t_baggett@homedepot.com
p: +1 (214) 202-8964
-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com]
Sent: Monday, October 27, 2014 3:24 AM
To: users@opennlp.apache.org
Subject: Re: Getting started with OpenNLP
On 10/24/2014 06:20 PM, PATRICK_T_BAGGETT@homedepot.com wrote:
> Hello all,
>
> First off, thanks to all who contribute to this project! I've been tasked with doing some research on Apache Stanbol, which uses OpenNLP, to see if it can fill some roles in a few company projects. I've been reading about how to train a model for named entity recognition and it seems like the simplest case of "I have a list of n proper nouns, please just recognize them directly and nothing else" isn't addressed in the documentation. Is this too simple a use case? Would I be doing better to just use a simple substring match on a phrase then? I would later like to extend the model to recognize things other than just simple proper nouns, but for now, that is the simplest case I can think of.
The name finder is intended to find entities which are embedded in a text, e.g. a news articles, medical records or company filings. It can even recognize names which it hasn't seen before by evaluating the context the entity appears in.
If you just have a list of proper nouns you might be better of using the doccat package instead of the name finder. The doccat component tries to assign the categories for the entire input text, compared to the name finder which labels each input token.
HTH,
Jörn
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: Getting started with OpenNLP
Posted by Jörn Kottmann <ko...@gmail.com>.
On 10/24/2014 06:20 PM, PATRICK_T_BAGGETT@homedepot.com wrote:
> Hello all,
>
> First off, thanks to all who contribute to this project! I've been tasked with doing some research on Apache Stanbol, which uses OpenNLP, to see if it can fill some roles in a few company projects. I've been reading about how to train a model for named entity recognition and it seems like the simplest case of "I have a list of n proper nouns, please just recognize them directly and nothing else" isn't addressed in the documentation. Is this too simple a use case? Would I be doing better to just use a simple substring match on a phrase then? I would later like to extend the model to recognize things other than just simple proper nouns, but for now, that is the simplest case I can think of.
The name finder is intended to find entities which are embedded in a
text, e.g. a news articles, medical records or company filings. It can
even recognize names
which it hasn't seen before by evaluating the context the entity appears in.
If you just have a list of proper nouns you might be better of using the
doccat package instead of the name finder. The doccat component tries to
assign the categories
for the entire input text, compared to the name finder which labels each
input token.
HTH,
Jörn