You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by Benedict Holland <be...@gmail.com> on 2018/10/17 17:14:23 UTC

maxent produces very high probabilties for texts without overlap

Hello all,

I can't quite figure out how the Doccat MaxEnt modeling works. Here is my
setup:

I have a set of training texts split into is_cat_1 and is_not_cat_1. I
train my model using the default bag of words model. I have a document
without any overlapping text with texts that are in is_cat_1. They might
overlap with text in is_not_cat_1. Meaning, every single word in the
document I want to categorize does not appear in any of the model training
data in the is_cat_1 category.

The result of the MaxEnt model for my document is a probability over 90%
that it fits into the is_cat_1. Why is that?

Thanks,
~Ben

Re: maxent produces very high probabilties for texts without overlap

Posted by Benedict Holland <be...@gmail.com>.
Sure. Let me get a serialized version of the training dataset put together.
It shouldn't be too big. Thank you! This one is really confusing. Also, I
updated to the UIMA 3.0 and opennlp 1.9. Maybe that changes things. I will
get that put together tonight. I will try and export it to a tag <space>
<text> format for training. That shouldn't be too hard.

Thank you!
~Ben

On Wed, Oct 17, 2018 at 4:33 PM Dan Russ <da...@gmail.com> wrote:

> Ok, so you are using the API. If you send out a small training example,
> I’ll train a DocumentModel and see what I get.
> Also send a few examples to test on.  We can compare the results.  It’s
> late in the day on the US East coast, so I may not be able to get to it
> until tomorrow.
> Daniel
>
>
> > On Oct 17, 2018, at 4:27 PM, Benedict Holland <
> benedict.m.holland@gmail.com> wrote:
> >
> > I mean... not really? I store everything in a database. I created a
> stream
> > that reads my training data from the database, splits it up into tokens,
> > and creates DocumentSample objects. The DocumentSampleStream interface is
> > really easy to work with and I really didn't have to implement much.
> >
> > I am quite confident that I created the DocumentSample objects and the
> > stream properly. I mean, it is reading in the tokens and I am training
> > models. For me, this was basically a unit test. I created a very simple
> > model with a single observation in is_cat_1,  a bunch of observations in
> > is_not_cat_1, ran a document without any overlap against the model and it
> > predicted a probability over 90% of it belonging to is_cat_1.
> >
> > At that point, I posted this to the group. I really cann't figure out how
> > that would be possible. I could even email the training data to someone
> if
> > they would like.
> >
> > Before that point though, I think I am using the 1.8.4. I can try
> upgrading
> > to the 1.9.0 release.
> >
> > Thanks,
> > ~Ben
> >
> > On Wed, Oct 17, 2018 at 4:00 PM Dan Russ <da...@gmail.com> wrote:
> >
> >> Really surprised.  Looking at the documentation, your training data
> should
> >> be in the following format. See (
> >>
> https://opennlp.apache.org/docs/1.9.0/manual/opennlp.html#tools.doccat.training
> >> )
> >>
> >> Is_cat_1 <text>
> >> Is_not_cat_1 <text>
> >>
> >> Is that how you formatted your data?
> >> Daniel
> >>
> >>> On Oct 17, 2018, at 3:50 PM, Benedict Holland <
> >> benedict.m.holland@gmail.com> wrote:
> >>>
> >>> Hi! Thanks for the reply.
> >>>
> >>> Yes. There is a massive imbalance.  Out of the thousands of
> observations
> >> I
> >>> have, only a small handful are actually positive observations in the
> >>> is_cat_1. The rest are in the is_not_cat_1. In some cases, the number
> of
> >>> positives are 1.
> >>>
> >>> For example:
> >>>
> >>> In one category, the only observation in is_cat_1 is:
> >>> assault use reckless force or vi
> >>>
> >>> I have a bunch of observations in the is_not_cat_1. This model placed
> >> this
> >>> text
> >>>
> >>> 0099 usc 18 usc 2
> >>>
> >>> has a probability match over 90%. Mind you, I expected this setup to be
> >>> horrible. I actually expected this sort of text to get close to a 100%
> in
> >>> the is_not_cat_1 but what I really cannot explain is the overlap. I did
> >>> verify that the BoW produces the following features:
> >>> ["bow=assault", "bow=use", "bow=recklass", "bow=force", "bow=or",
> >> "bow="vi"]
> >>>
> >>> The only thing I could come up with is something like each string being
> >>> broken apart into individual letters but that wouldn't make sense. Or
> >> would
> >>> it?
> >>>
> >>> Thanks,
> >>> ~Ben
> >>>
> >>> On Wed, Oct 17, 2018 at 3:26 PM Dan Russ <da...@gmail.com> wrote:
> >>>
> >>>> Hi Ben,
> >>>>  Are you sure that your training documents are formatted
> appropriately?
> >>>> Also, do you have a large imbalance in the # of training documents?
> If
> >> the
> >>>> text in the testing document is not in either CAT_1 or the OTHER_CAT,
> >> there
> >>>> will be a .5 assignment to each category (assuming equal documents so
> >> the
> >>>> prior doesn’t change the value).  A .5 assignment is like “I can’t
> tell
> >> the
> >>>> two categories”.  You probably don’t want to think of it as “You don’t
> >> look
> >>>> like CAT_1 so you are NOT_CAT_1”.
> >>>> Daniel
> >>>>
> >>>>> On Oct 17, 2018, at 1:14 PM, Benedict Holland <
> >>>> benedict.m.holland@gmail.com> wrote:
> >>>>>
> >>>>> Hello all,
> >>>>>
> >>>>> I can't quite figure out how the Doccat MaxEnt modeling works. Here
> is
> >> my
> >>>>> setup:
> >>>>>
> >>>>> I have a set of training texts split into is_cat_1 and is_not_cat_1.
> I
> >>>>> train my model using the default bag of words model. I have a
> document
> >>>>> without any overlapping text with texts that are in is_cat_1. They
> >> might
> >>>>> overlap with text in is_not_cat_1. Meaning, every single word in the
> >>>>> document I want to categorize does not appear in any of the model
> >>>> training
> >>>>> data in the is_cat_1 category.
> >>>>>
> >>>>> The result of the MaxEnt model for my document is a probability over
> >> 90%
> >>>>> that it fits into the is_cat_1. Why is that?
> >>>>>
> >>>>> Thanks,
> >>>>> ~Ben
> >>>>
> >>>>
> >>
> >>
>
>

Re: maxent produces very high probabilties for texts without overlap

Posted by Dan Russ <da...@gmail.com>.
Ok, so you are using the API. If you send out a small training example, I’ll train a DocumentModel and see what I get. 
Also send a few examples to test on.  We can compare the results.  It’s late in the day on the US East coast, so I may not be able to get to it until tomorrow.
Daniel


> On Oct 17, 2018, at 4:27 PM, Benedict Holland <be...@gmail.com> wrote:
> 
> I mean... not really? I store everything in a database. I created a stream
> that reads my training data from the database, splits it up into tokens,
> and creates DocumentSample objects. The DocumentSampleStream interface is
> really easy to work with and I really didn't have to implement much.
> 
> I am quite confident that I created the DocumentSample objects and the
> stream properly. I mean, it is reading in the tokens and I am training
> models. For me, this was basically a unit test. I created a very simple
> model with a single observation in is_cat_1,  a bunch of observations in
> is_not_cat_1, ran a document without any overlap against the model and it
> predicted a probability over 90% of it belonging to is_cat_1.
> 
> At that point, I posted this to the group. I really cann't figure out how
> that would be possible. I could even email the training data to someone if
> they would like.
> 
> Before that point though, I think I am using the 1.8.4. I can try upgrading
> to the 1.9.0 release.
> 
> Thanks,
> ~Ben
> 
> On Wed, Oct 17, 2018 at 4:00 PM Dan Russ <da...@gmail.com> wrote:
> 
>> Really surprised.  Looking at the documentation, your training data should
>> be in the following format. See (
>> https://opennlp.apache.org/docs/1.9.0/manual/opennlp.html#tools.doccat.training
>> )
>> 
>> Is_cat_1 <text>
>> Is_not_cat_1 <text>
>> 
>> Is that how you formatted your data?
>> Daniel
>> 
>>> On Oct 17, 2018, at 3:50 PM, Benedict Holland <
>> benedict.m.holland@gmail.com> wrote:
>>> 
>>> Hi! Thanks for the reply.
>>> 
>>> Yes. There is a massive imbalance.  Out of the thousands of observations
>> I
>>> have, only a small handful are actually positive observations in the
>>> is_cat_1. The rest are in the is_not_cat_1. In some cases, the number of
>>> positives are 1.
>>> 
>>> For example:
>>> 
>>> In one category, the only observation in is_cat_1 is:
>>> assault use reckless force or vi
>>> 
>>> I have a bunch of observations in the is_not_cat_1. This model placed
>> this
>>> text
>>> 
>>> 0099 usc 18 usc 2
>>> 
>>> has a probability match over 90%. Mind you, I expected this setup to be
>>> horrible. I actually expected this sort of text to get close to a 100% in
>>> the is_not_cat_1 but what I really cannot explain is the overlap. I did
>>> verify that the BoW produces the following features:
>>> ["bow=assault", "bow=use", "bow=recklass", "bow=force", "bow=or",
>> "bow="vi"]
>>> 
>>> The only thing I could come up with is something like each string being
>>> broken apart into individual letters but that wouldn't make sense. Or
>> would
>>> it?
>>> 
>>> Thanks,
>>> ~Ben
>>> 
>>> On Wed, Oct 17, 2018 at 3:26 PM Dan Russ <da...@gmail.com> wrote:
>>> 
>>>> Hi Ben,
>>>>  Are you sure that your training documents are formatted appropriately?
>>>> Also, do you have a large imbalance in the # of training documents?  If
>> the
>>>> text in the testing document is not in either CAT_1 or the OTHER_CAT,
>> there
>>>> will be a .5 assignment to each category (assuming equal documents so
>> the
>>>> prior doesn’t change the value).  A .5 assignment is like “I can’t tell
>> the
>>>> two categories”.  You probably don’t want to think of it as “You don’t
>> look
>>>> like CAT_1 so you are NOT_CAT_1”.
>>>> Daniel
>>>> 
>>>>> On Oct 17, 2018, at 1:14 PM, Benedict Holland <
>>>> benedict.m.holland@gmail.com> wrote:
>>>>> 
>>>>> Hello all,
>>>>> 
>>>>> I can't quite figure out how the Doccat MaxEnt modeling works. Here is
>> my
>>>>> setup:
>>>>> 
>>>>> I have a set of training texts split into is_cat_1 and is_not_cat_1. I
>>>>> train my model using the default bag of words model. I have a document
>>>>> without any overlapping text with texts that are in is_cat_1. They
>> might
>>>>> overlap with text in is_not_cat_1. Meaning, every single word in the
>>>>> document I want to categorize does not appear in any of the model
>>>> training
>>>>> data in the is_cat_1 category.
>>>>> 
>>>>> The result of the MaxEnt model for my document is a probability over
>> 90%
>>>>> that it fits into the is_cat_1. Why is that?
>>>>> 
>>>>> Thanks,
>>>>> ~Ben
>>>> 
>>>> 
>> 
>> 


Re: maxent produces very high probabilties for texts without overlap

Posted by Benedict Holland <be...@gmail.com>.
I mean... not really? I store everything in a database. I created a stream
that reads my training data from the database, splits it up into tokens,
and creates DocumentSample objects. The DocumentSampleStream interface is
really easy to work with and I really didn't have to implement much.

I am quite confident that I created the DocumentSample objects and the
stream properly. I mean, it is reading in the tokens and I am training
models. For me, this was basically a unit test. I created a very simple
model with a single observation in is_cat_1,  a bunch of observations in
is_not_cat_1, ran a document without any overlap against the model and it
predicted a probability over 90% of it belonging to is_cat_1.

At that point, I posted this to the group. I really cann't figure out how
that would be possible. I could even email the training data to someone if
they would like.

Before that point though, I think I am using the 1.8.4. I can try upgrading
to the 1.9.0 release.

Thanks,
~Ben

On Wed, Oct 17, 2018 at 4:00 PM Dan Russ <da...@gmail.com> wrote:

> Really surprised.  Looking at the documentation, your training data should
> be in the following format. See (
> https://opennlp.apache.org/docs/1.9.0/manual/opennlp.html#tools.doccat.training
> )
>
> Is_cat_1 <text>
> Is_not_cat_1 <text>
>
> Is that how you formatted your data?
> Daniel
>
> > On Oct 17, 2018, at 3:50 PM, Benedict Holland <
> benedict.m.holland@gmail.com> wrote:
> >
> > Hi! Thanks for the reply.
> >
> > Yes. There is a massive imbalance.  Out of the thousands of observations
> I
> > have, only a small handful are actually positive observations in the
> > is_cat_1. The rest are in the is_not_cat_1. In some cases, the number of
> > positives are 1.
> >
> > For example:
> >
> > In one category, the only observation in is_cat_1 is:
> > assault use reckless force or vi
> >
> > I have a bunch of observations in the is_not_cat_1. This model placed
> this
> > text
> >
> > 0099 usc 18 usc 2
> >
> > has a probability match over 90%. Mind you, I expected this setup to be
> > horrible. I actually expected this sort of text to get close to a 100% in
> > the is_not_cat_1 but what I really cannot explain is the overlap. I did
> > verify that the BoW produces the following features:
> > ["bow=assault", "bow=use", "bow=recklass", "bow=force", "bow=or",
> "bow="vi"]
> >
> > The only thing I could come up with is something like each string being
> > broken apart into individual letters but that wouldn't make sense. Or
> would
> > it?
> >
> > Thanks,
> > ~Ben
> >
> > On Wed, Oct 17, 2018 at 3:26 PM Dan Russ <da...@gmail.com> wrote:
> >
> >> Hi Ben,
> >>   Are you sure that your training documents are formatted appropriately?
> >> Also, do you have a large imbalance in the # of training documents?  If
> the
> >> text in the testing document is not in either CAT_1 or the OTHER_CAT,
> there
> >> will be a .5 assignment to each category (assuming equal documents so
> the
> >> prior doesn’t change the value).  A .5 assignment is like “I can’t tell
> the
> >> two categories”.  You probably don’t want to think of it as “You don’t
> look
> >> like CAT_1 so you are NOT_CAT_1”.
> >> Daniel
> >>
> >>> On Oct 17, 2018, at 1:14 PM, Benedict Holland <
> >> benedict.m.holland@gmail.com> wrote:
> >>>
> >>> Hello all,
> >>>
> >>> I can't quite figure out how the Doccat MaxEnt modeling works. Here is
> my
> >>> setup:
> >>>
> >>> I have a set of training texts split into is_cat_1 and is_not_cat_1. I
> >>> train my model using the default bag of words model. I have a document
> >>> without any overlapping text with texts that are in is_cat_1. They
> might
> >>> overlap with text in is_not_cat_1. Meaning, every single word in the
> >>> document I want to categorize does not appear in any of the model
> >> training
> >>> data in the is_cat_1 category.
> >>>
> >>> The result of the MaxEnt model for my document is a probability over
> 90%
> >>> that it fits into the is_cat_1. Why is that?
> >>>
> >>> Thanks,
> >>> ~Ben
> >>
> >>
>
>

Re: maxent produces very high probabilties for texts without overlap

Posted by Dan Russ <da...@gmail.com>.
Really surprised.  Looking at the documentation, your training data should be in the following format. See (https://opennlp.apache.org/docs/1.9.0/manual/opennlp.html#tools.doccat.training)

Is_cat_1 <text>
Is_not_cat_1 <text>

Is that how you formatted your data?
Daniel

> On Oct 17, 2018, at 3:50 PM, Benedict Holland <be...@gmail.com> wrote:
> 
> Hi! Thanks for the reply.
> 
> Yes. There is a massive imbalance.  Out of the thousands of observations I
> have, only a small handful are actually positive observations in the
> is_cat_1. The rest are in the is_not_cat_1. In some cases, the number of
> positives are 1.
> 
> For example:
> 
> In one category, the only observation in is_cat_1 is:
> assault use reckless force or vi
> 
> I have a bunch of observations in the is_not_cat_1. This model placed this
> text
> 
> 0099 usc 18 usc 2
> 
> has a probability match over 90%. Mind you, I expected this setup to be
> horrible. I actually expected this sort of text to get close to a 100% in
> the is_not_cat_1 but what I really cannot explain is the overlap. I did
> verify that the BoW produces the following features:
> ["bow=assault", "bow=use", "bow=recklass", "bow=force", "bow=or", "bow="vi"]
> 
> The only thing I could come up with is something like each string being
> broken apart into individual letters but that wouldn't make sense. Or would
> it?
> 
> Thanks,
> ~Ben
> 
> On Wed, Oct 17, 2018 at 3:26 PM Dan Russ <da...@gmail.com> wrote:
> 
>> Hi Ben,
>>   Are you sure that your training documents are formatted appropriately?
>> Also, do you have a large imbalance in the # of training documents?  If the
>> text in the testing document is not in either CAT_1 or the OTHER_CAT, there
>> will be a .5 assignment to each category (assuming equal documents so the
>> prior doesn’t change the value).  A .5 assignment is like “I can’t tell the
>> two categories”.  You probably don’t want to think of it as “You don’t look
>> like CAT_1 so you are NOT_CAT_1”.
>> Daniel
>> 
>>> On Oct 17, 2018, at 1:14 PM, Benedict Holland <
>> benedict.m.holland@gmail.com> wrote:
>>> 
>>> Hello all,
>>> 
>>> I can't quite figure out how the Doccat MaxEnt modeling works. Here is my
>>> setup:
>>> 
>>> I have a set of training texts split into is_cat_1 and is_not_cat_1. I
>>> train my model using the default bag of words model. I have a document
>>> without any overlapping text with texts that are in is_cat_1. They might
>>> overlap with text in is_not_cat_1. Meaning, every single word in the
>>> document I want to categorize does not appear in any of the model
>> training
>>> data in the is_cat_1 category.
>>> 
>>> The result of the MaxEnt model for my document is a probability over 90%
>>> that it fits into the is_cat_1. Why is that?
>>> 
>>> Thanks,
>>> ~Ben
>> 
>> 


Re: maxent produces very high probabilties for texts without overlap

Posted by Benedict Holland <be...@gmail.com>.
Hi! Thanks for the reply.

Yes. There is a massive imbalance.  Out of the thousands of observations I
have, only a small handful are actually positive observations in the
is_cat_1. The rest are in the is_not_cat_1. In some cases, the number of
positives are 1.

For example:

In one category, the only observation in is_cat_1 is:
assault use reckless force or vi

I have a bunch of observations in the is_not_cat_1. This model placed this
text

0099 usc 18 usc 2

has a probability match over 90%. Mind you, I expected this setup to be
horrible. I actually expected this sort of text to get close to a 100% in
the is_not_cat_1 but what I really cannot explain is the overlap. I did
verify that the BoW produces the following features:
["bow=assault", "bow=use", "bow=recklass", "bow=force", "bow=or", "bow="vi"]

The only thing I could come up with is something like each string being
broken apart into individual letters but that wouldn't make sense. Or would
it?

Thanks,
~Ben

On Wed, Oct 17, 2018 at 3:26 PM Dan Russ <da...@gmail.com> wrote:

> Hi Ben,
>    Are you sure that your training documents are formatted appropriately?
> Also, do you have a large imbalance in the # of training documents?  If the
> text in the testing document is not in either CAT_1 or the OTHER_CAT, there
> will be a .5 assignment to each category (assuming equal documents so the
> prior doesn’t change the value).  A .5 assignment is like “I can’t tell the
> two categories”.  You probably don’t want to think of it as “You don’t look
> like CAT_1 so you are NOT_CAT_1”.
> Daniel
>
> > On Oct 17, 2018, at 1:14 PM, Benedict Holland <
> benedict.m.holland@gmail.com> wrote:
> >
> > Hello all,
> >
> > I can't quite figure out how the Doccat MaxEnt modeling works. Here is my
> > setup:
> >
> > I have a set of training texts split into is_cat_1 and is_not_cat_1. I
> > train my model using the default bag of words model. I have a document
> > without any overlapping text with texts that are in is_cat_1. They might
> > overlap with text in is_not_cat_1. Meaning, every single word in the
> > document I want to categorize does not appear in any of the model
> training
> > data in the is_cat_1 category.
> >
> > The result of the MaxEnt model for my document is a probability over 90%
> > that it fits into the is_cat_1. Why is that?
> >
> > Thanks,
> > ~Ben
>
>

Re: maxent produces very high probabilties for texts without overlap

Posted by Dan Russ <da...@gmail.com>.
Hi Ben,
   Are you sure that your training documents are formatted appropriately?  Also, do you have a large imbalance in the # of training documents?  If the text in the testing document is not in either CAT_1 or the OTHER_CAT, there will be a .5 assignment to each category (assuming equal documents so the prior doesn’t change the value).  A .5 assignment is like “I can’t tell the two categories”.  You probably don’t want to think of it as “You don’t look like CAT_1 so you are NOT_CAT_1”.
Daniel

> On Oct 17, 2018, at 1:14 PM, Benedict Holland <be...@gmail.com> wrote:
> 
> Hello all,
> 
> I can't quite figure out how the Doccat MaxEnt modeling works. Here is my
> setup:
> 
> I have a set of training texts split into is_cat_1 and is_not_cat_1. I
> train my model using the default bag of words model. I have a document
> without any overlapping text with texts that are in is_cat_1. They might
> overlap with text in is_not_cat_1. Meaning, every single word in the
> document I want to categorize does not appear in any of the model training
> data in the is_cat_1 category.
> 
> The result of the MaxEnt model for my document is a probability over 90%
> that it fits into the is_cat_1. Why is that?
> 
> Thanks,
> ~Ben