You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by Danica Damljanovic <da...@gmail.com> on 2013/09/02 19:10:33 UTC
Exception: The maxent model is not compatible with the sentence
detector! when training MaxEnt Sentence Detector
Hi everyone
I am trying to retrain the Sentence Detector, however, I keep getting an
exception. I get the same result with the command line and
programmatically. Below is the command I ran and the output.
I use apache-opennlp-1.5.3, Mac OS Lion, and I tried with this sample text
I found in one of the online tutorials:
"Being at the polls was just like being at church.
I didn't smell a drop of liquor, and we didn't have a bit of trouble.
The campaign leading to the election was not so quiet.
It was marked by controversy, anonymous midnight phone calls and veiled
threats of violence.
During the election campaign, both candidates, Davis and Bush, reportedly
received anonymous telephone calls.
Ordinary Williams said he , too , was subjected to anonymous calls soon
after he scheduled the election.
Many local citizens feared that there would be irregularities at the polls.
Williams got himself a permit to carry a gun and promised an orderly
election.
He attended New York University before switching to Georgetown University
in Washington."
Any hint much appreciated.
bin/opennlp SentenceDetectorTrainer -encoding UTF-8 -lang en -data
en-sent.train -model en-sent.bin
Indexing events using cutoff of 5
Computing event counts... done. 9 events
Indexing... done.
Sorting and merging events... done. Reduced 9 events to 2.
Done indexing.
Incorporating indexed data for training...
done.
Number of Event Tokens: 2
Number of Outcomes: 1
Number of Predicates: 4
...done.
Computing model parameters ...
Performing 100 iterations.
1: ... loglikelihood=0.0 1.0
2: ... loglikelihood=0.0 1.0
Exception in thread "main" java.lang.IllegalArgumentException:
opennlp.tools.util.InvalidFormatException: The maxent model is not
compatible with the sentence detector!
at opennlp.tools.util.model.BaseModel.checkArtifactMap(BaseModel.java:476)
at opennlp.tools.sentdetect.SentenceModel.<init>(SentenceModel.java:54)
at
opennlp.tools.sentdetect.SentenceDetectorME.train(SentenceDetectorME.java:315)
at
opennlp.tools.cmdline.sentdetect.SentenceDetectorTrainerTool.run(SentenceDetectorTrainerTool.java:88)
at opennlp.tools.cmdline.CLI.main(CLI.java:222)
Caused by: opennlp.tools.util.InvalidFormatException: The maxent model is
not compatible with the sentence detector!
at
opennlp.tools.sentdetect.SentenceModel.validateArtifactMap(SentenceModel.java:117)
at opennlp.tools.util.model.BaseModel.checkArtifactMap(BaseModel.java:474)
... 4 more
Thanks in advance!
Danica
Re: Exception: The maxent model is not compatible with the sentence
detector! when training MaxEnt Sentence Detector
Posted by "Jim - FooBar();" <ji...@gmail.com>.
On a slightly different note, what you're trying to do is NOT
unreasonable... I'm thinking about the wider topic of
'probabilistic-classifiers' where 'training' essentially boils down to
gathering a bunch of probabilities. I can't think of any technical
reasons that would prohibit you to train an already trained model on
some extra data, other than the implementation of a given system. In
other words, if all you've got is a bunch of probabilities, why not be
able to add to them in the future? Nothing can stop you from doing that
but the implementation specifics.
To make things more concrete, consider an HMM POS-tagger. When you
'train' it, all you're doing is 'observing' what tag appears more
frequently before the tag you’re currently looking. Based on those
frequencies you built a probability matrix which you consult later on in
order to make predictions. Now, consider this...let's say you chose to
represent your matrix as a HashMap. So the entire model is a
HashMap...there is absolutely no problem retraining that whenever you
get some more data and without losing the original training
probabilities. Of course now you're gonna say that the tag-set might be
different, but what if the first time you trained you had access to the
first half of the Brown corpus and after a couple of months you managed
to find the other half...the tag-set doesn't change within the same
corpus so you can retrain your HMM without introducing noise...you just
'merge-with +' the maps at the end and voila! You should have the exact
same model as you would had if you'd trained on the entire Brown corpus.
Having said all that, I'll admit that I've never encountered any
machine-learning implementations that allow you to do that and I'm
wondering why...It's easy to implement and provides a ton of
flexibility. I've proven to myself that it can be done my implementing
what I described in the previous paragraph (an HMM POS-Tagger) and it
works, but generally speaking libraries and frameworks don't allow it...
I'm saying all that so you don't go thinking that what you tried is just
plain wrong...Well, it is the way you tried to but your underlying
thought is perfectly valid.
hope that helps, :)
Jim
On 03/09/13 17:42, Jim - FooBar(); wrote:
> On 03/09/13 17:25, Danica Damljanovic wrote:
>> I was trying to find the original opennlp corpora used for training, but
>> could not get anything apart from the binary model...
>>
>> Anyone has any idea on whether it is possible to get this and how?
>
> If I'm not mistaken the original corpora cannot be re-distributed due
> to licensing issues...However, don't take my word for it - someone
> with the appropriate authority should answer this (someone from the
> dev-team)...
>
> Also, if I remember correctly, you can get a pretty decent
> sentence-detecting model with less than 100 sentences, whereas for the
> rest of the components (Tokenizer ,POSTagger, NER etc etc) you need
> thousands of sentences!
>
> Jim
Re: Exception: The maxent model is not compatible with the sentence
detector! when training MaxEnt Sentence Detector
Posted by "Jim - FooBar();" <ji...@gmail.com>.
On 03/09/13 17:25, Danica Damljanovic wrote:
> I was trying to find the original opennlp corpora used for training, but
> could not get anything apart from the binary model...
>
> Anyone has any idea on whether it is possible to get this and how?
If I'm not mistaken the original corpora cannot be re-distributed due to
licensing issues...However, don't take my word for it - someone with the
appropriate authority should answer this (someone from the dev-team)...
Also, if I remember correctly, you can get a pretty decent
sentence-detecting model with less than 100 sentences, whereas for the
rest of the components (Tokenizer ,POSTagger, NER etc etc) you need
thousands of sentences!
Jim
Re: Exception: The maxent model is not compatible with the sentence
detector! when training MaxEnt Sentence Detector
Posted by Danica Damljanovic <da...@gmail.com>.
Ah, I was wondering about this, but I misinterpreted what is written on the
wiki page! Thanks Jim.
I was trying to find the original opennlp corpora used for training, but
could not get anything apart from the binary model...
Anyone has any idea on whether it is possible to get this and how?
Thanks
Danica
On 3 September 2013 17:16, Jim - FooBar(); <ji...@gmail.com> wrote:
> You can't do that...there is no way to get the original corpus from the
> model! You seem to think that the trained model is just a bunch of
> sentences whereas the model is a bunch of binary features. If you want to
> improve the already trained sentence-detector you need to get your hands on
> the original corpus Jorn and the rest of the dev-team used...
>
> make sure you understand what the models consist of...
>
> HTH,
> Jim
>
>
>
> On 03/09/13 16:27, Danica Damljanovic wrote:
>
>> Jork: Yes, but that's if I want to train the model using input.txt and
>> then
>> read it again into output.txt.
>>
>> What I am trying to do is read out the corpus of sentences that are used
>> to
>> create en-sent.bin,
>> so that I can improve it. Any hints on how can I do that?
>>
>> In other words, I am trying to read
>> http://opennlp.sourceforge.**net/models-1.5/en-sent.bin<http://opennlp.sourceforge.net/models-1.5/en-sent.bin>into en-sent.train
>> txt file that I can open and update (and then retrain the sentence
>> splitter).
>>
>>
>> On 3 September 2013 16:19, Jörn Kottmann <ko...@gmail.com> wrote:
>>
>> On 09/03/2013 04:57 PM, Danica Damljanovic wrote:
>>>
>>> If I may ask one more question about retraining the sentence detector. I
>>>> created a corpus that I want to use for training, but I would rather
>>>> improve on the existing sentence splitter, so this is what I did to get
>>>> the
>>>> initial corpus:
>>>>
>>>> opennlp SentenceDetector en-sent.bin > output.txt
>>>>
>>>> However, although I gave the process 4GB of memory, it seems to be
>>>> running
>>>> for a while, and the only output I see is:
>>>>
>>>>
>>>> Loading Sentence Detector model ... done (0.031s)
>>>>
>>>> What I expect is to see the list of sentences used for training, so
>>>> that I
>>>> can merge output.txt with my corpus, and retrain the parser. But after
>>>> more
>>>> than one hour, it still did not start to write into output.txt.
>>>>
>>>>
>>> You need to provide some input text to the Sentence Detector, otherwise
>>> it
>>> just waits forever for it,
>>> have a look at the manual for a sample.
>>>
>>> Jörn
>>>
>>>
>
Re: Exception: The maxent model is not compatible with the sentence
detector! when training MaxEnt Sentence Detector
Posted by "Jim - FooBar();" <ji...@gmail.com>.
You can't do that...there is no way to get the original corpus from the
model! You seem to think that the trained model is just a bunch of
sentences whereas the model is a bunch of binary features. If you want
to improve the already trained sentence-detector you need to get your
hands on the original corpus Jorn and the rest of the dev-team used...
make sure you understand what the models consist of...
HTH,
Jim
On 03/09/13 16:27, Danica Damljanovic wrote:
> Jork: Yes, but that's if I want to train the model using input.txt and then
> read it again into output.txt.
>
> What I am trying to do is read out the corpus of sentences that are used to
> create en-sent.bin,
> so that I can improve it. Any hints on how can I do that?
>
> In other words, I am trying to read
> http://opennlp.sourceforge.net/models-1.5/en-sent.bin into en-sent.train
> txt file that I can open and update (and then retrain the sentence
> splitter).
>
>
> On 3 September 2013 16:19, Jörn Kottmann <ko...@gmail.com> wrote:
>
>> On 09/03/2013 04:57 PM, Danica Damljanovic wrote:
>>
>>> If I may ask one more question about retraining the sentence detector. I
>>> created a corpus that I want to use for training, but I would rather
>>> improve on the existing sentence splitter, so this is what I did to get
>>> the
>>> initial corpus:
>>>
>>> opennlp SentenceDetector en-sent.bin > output.txt
>>>
>>> However, although I gave the process 4GB of memory, it seems to be running
>>> for a while, and the only output I see is:
>>>
>>>
>>> Loading Sentence Detector model ... done (0.031s)
>>>
>>> What I expect is to see the list of sentences used for training, so that I
>>> can merge output.txt with my corpus, and retrain the parser. But after
>>> more
>>> than one hour, it still did not start to write into output.txt.
>>>
>>
>> You need to provide some input text to the Sentence Detector, otherwise it
>> just waits forever for it,
>> have a look at the manual for a sample.
>>
>> Jörn
>>
Re: Exception: The maxent model is not compatible with the sentence
detector! when training MaxEnt Sentence Detector
Posted by Danica Damljanovic <da...@gmail.com>.
Jork: Yes, but that's if I want to train the model using input.txt and then
read it again into output.txt.
What I am trying to do is read out the corpus of sentences that are used to
create en-sent.bin,
so that I can improve it. Any hints on how can I do that?
In other words, I am trying to read
http://opennlp.sourceforge.net/models-1.5/en-sent.bin into en-sent.train
txt file that I can open and update (and then retrain the sentence
splitter).
On 3 September 2013 16:19, Jörn Kottmann <ko...@gmail.com> wrote:
> On 09/03/2013 04:57 PM, Danica Damljanovic wrote:
>
>> If I may ask one more question about retraining the sentence detector. I
>> created a corpus that I want to use for training, but I would rather
>> improve on the existing sentence splitter, so this is what I did to get
>> the
>> initial corpus:
>>
>> opennlp SentenceDetector en-sent.bin > output.txt
>>
>> However, although I gave the process 4GB of memory, it seems to be running
>> for a while, and the only output I see is:
>>
>>
>> Loading Sentence Detector model ... done (0.031s)
>>
>> What I expect is to see the list of sentences used for training, so that I
>> can merge output.txt with my corpus, and retrain the parser. But after
>> more
>> than one hour, it still did not start to write into output.txt.
>>
>
>
> You need to provide some input text to the Sentence Detector, otherwise it
> just waits forever for it,
> have a look at the manual for a sample.
>
> Jörn
>
Re: Exception: The maxent model is not compatible with the sentence
detector! when training MaxEnt Sentence Detector
Posted by Jörn Kottmann <ko...@gmail.com>.
On 09/03/2013 04:57 PM, Danica Damljanovic wrote:
> If I may ask one more question about retraining the sentence detector. I
> created a corpus that I want to use for training, but I would rather
> improve on the existing sentence splitter, so this is what I did to get the
> initial corpus:
>
> opennlp SentenceDetector en-sent.bin > output.txt
>
> However, although I gave the process 4GB of memory, it seems to be running
> for a while, and the only output I see is:
>
>
> Loading Sentence Detector model ... done (0.031s)
>
> What I expect is to see the list of sentences used for training, so that I
> can merge output.txt with my corpus, and retrain the parser. But after more
> than one hour, it still did not start to write into output.txt.
You need to provide some input text to the Sentence Detector, otherwise
it just waits forever for it,
have a look at the manual for a sample.
Jörn
Re: Exception: The maxent model is not compatible with the sentence
detector! when training MaxEnt Sentence Detector
Posted by Danica Damljanovic <da...@gmail.com>.
Jorn, all,
If I may ask one more question about retraining the sentence detector. I
created a corpus that I want to use for training, but I would rather
improve on the existing sentence splitter, so this is what I did to get the
initial corpus:
opennlp SentenceDetector en-sent.bin > output.txt
However, although I gave the process 4GB of memory, it seems to be running
for a while, and the only output I see is:
Loading Sentence Detector model ... done (0.031s)
What I expect is to see the list of sentences used for training, so that I
can merge output.txt with my corpus, and retrain the parser. But after more
than one hour, it still did not start to write into output.txt.
Is this the right way to go about it? If so, is it normal for this to take
a long time or is there something else I need to do?
Thanks
Danica
On 3 September 2013 12:15, Danica Damljanovic
<da...@gmail.com>wrote:
> Thanks Jorn, that solved the problem!
>
>
> On 2 September 2013 19:32, Jörn Kottmann <ko...@gmail.com> wrote:
>
>> The error seems to occur when you don't have enough training data (and
>> train with cutoff 5). Try to train with
>> more data.
>>
>> Jörn
>>
>>
>>
>> On 09/02/2013 07:10 PM, Danica Damljanovic wrote:
>>
>>> Hi everyone
>>>
>>> I am trying to retrain the Sentence Detector, however, I keep getting an
>>> exception. I get the same result with the command line and
>>> programmatically. Below is the command I ran and the output.
>>>
>>> I use apache-opennlp-1.5.3, Mac OS Lion, and I tried with this sample
>>> text
>>> I found in one of the online tutorials:
>>>
>>> "Being at the polls was just like being at church.
>>> I didn't smell a drop of liquor, and we didn't have a bit of trouble.
>>> The campaign leading to the election was not so quiet.
>>> It was marked by controversy, anonymous midnight phone calls and veiled
>>> threats of violence.
>>> During the election campaign, both candidates, Davis and Bush, reportedly
>>> received anonymous telephone calls.
>>> Ordinary Williams said he , too , was subjected to anonymous calls soon
>>> after he scheduled the election.
>>> Many local citizens feared that there would be irregularities at the
>>> polls.
>>> Williams got himself a permit to carry a gun and promised an orderly
>>> election.
>>> He attended New York University before switching to Georgetown University
>>> in Washington."
>>>
>>>
>>> Any hint much appreciated.
>>>
>>> bin/opennlp SentenceDetectorTrainer -encoding UTF-8 -lang en -data
>>> en-sent.train -model en-sent.bin
>>> Indexing events using cutoff of 5
>>>
>>> Computing event counts... done. 9 events
>>> Indexing... done.
>>> Sorting and merging events... done. Reduced 9 events to 2.
>>> Done indexing.
>>> Incorporating indexed data for training...
>>> done.
>>> Number of Event Tokens: 2
>>> Number of Outcomes: 1
>>> Number of Predicates: 4
>>> ...done.
>>> Computing model parameters ...
>>> Performing 100 iterations.
>>> 1: ... loglikelihood=0.0 1.0
>>> 2: ... loglikelihood=0.0 1.0
>>> Exception in thread "main" java.lang.**IllegalArgumentException:
>>> opennlp.tools.util.**InvalidFormatException: The maxent model is not
>>> compatible with the sentence detector!
>>> at opennlp.tools.util.model.**BaseModel.checkArtifactMap(**
>>> BaseModel.java:476)
>>> at opennlp.tools.sentdetect.**SentenceModel.<init>(**
>>> SentenceModel.java:54)
>>> at
>>> opennlp.tools.sentdetect.**SentenceDetectorME.train(**
>>> SentenceDetectorME.java:315)
>>> at
>>> opennlp.tools.cmdline.**sentdetect.**SentenceDetectorTrainerTool.**run(*
>>> *SentenceDetectorTrainerTool.**java:88)
>>> at opennlp.tools.cmdline.CLI.**main(CLI.java:222)
>>> Caused by: opennlp.tools.util.**InvalidFormatException: The maxent
>>> model is
>>> not compatible with the sentence detector!
>>> at
>>> opennlp.tools.sentdetect.**SentenceModel.**validateArtifactMap(**
>>> SentenceModel.java:117)
>>> at opennlp.tools.util.model.**BaseModel.checkArtifactMap(**
>>> BaseModel.java:474)
>>> ... 4 more
>>>
>>> Thanks in advance!
>>> Danica
>>>
>>>
>>
>
Re: Exception: The maxent model is not compatible with the sentence
detector! when training MaxEnt Sentence Detector
Posted by Danica Damljanovic <da...@gmail.com>.
Thanks Jorn, that solved the problem!
On 2 September 2013 19:32, Jörn Kottmann <ko...@gmail.com> wrote:
> The error seems to occur when you don't have enough training data (and
> train with cutoff 5). Try to train with
> more data.
>
> Jörn
>
>
>
> On 09/02/2013 07:10 PM, Danica Damljanovic wrote:
>
>> Hi everyone
>>
>> I am trying to retrain the Sentence Detector, however, I keep getting an
>> exception. I get the same result with the command line and
>> programmatically. Below is the command I ran and the output.
>>
>> I use apache-opennlp-1.5.3, Mac OS Lion, and I tried with this sample text
>> I found in one of the online tutorials:
>>
>> "Being at the polls was just like being at church.
>> I didn't smell a drop of liquor, and we didn't have a bit of trouble.
>> The campaign leading to the election was not so quiet.
>> It was marked by controversy, anonymous midnight phone calls and veiled
>> threats of violence.
>> During the election campaign, both candidates, Davis and Bush, reportedly
>> received anonymous telephone calls.
>> Ordinary Williams said he , too , was subjected to anonymous calls soon
>> after he scheduled the election.
>> Many local citizens feared that there would be irregularities at the
>> polls.
>> Williams got himself a permit to carry a gun and promised an orderly
>> election.
>> He attended New York University before switching to Georgetown University
>> in Washington."
>>
>>
>> Any hint much appreciated.
>>
>> bin/opennlp SentenceDetectorTrainer -encoding UTF-8 -lang en -data
>> en-sent.train -model en-sent.bin
>> Indexing events using cutoff of 5
>>
>> Computing event counts... done. 9 events
>> Indexing... done.
>> Sorting and merging events... done. Reduced 9 events to 2.
>> Done indexing.
>> Incorporating indexed data for training...
>> done.
>> Number of Event Tokens: 2
>> Number of Outcomes: 1
>> Number of Predicates: 4
>> ...done.
>> Computing model parameters ...
>> Performing 100 iterations.
>> 1: ... loglikelihood=0.0 1.0
>> 2: ... loglikelihood=0.0 1.0
>> Exception in thread "main" java.lang.**IllegalArgumentException:
>> opennlp.tools.util.**InvalidFormatException: The maxent model is not
>> compatible with the sentence detector!
>> at opennlp.tools.util.model.**BaseModel.checkArtifactMap(**
>> BaseModel.java:476)
>> at opennlp.tools.sentdetect.**SentenceModel.<init>(**
>> SentenceModel.java:54)
>> at
>> opennlp.tools.sentdetect.**SentenceDetectorME.train(**
>> SentenceDetectorME.java:315)
>> at
>> opennlp.tools.cmdline.**sentdetect.**SentenceDetectorTrainerTool.**run(**
>> SentenceDetectorTrainerTool.**java:88)
>> at opennlp.tools.cmdline.CLI.**main(CLI.java:222)
>> Caused by: opennlp.tools.util.**InvalidFormatException: The maxent model
>> is
>> not compatible with the sentence detector!
>> at
>> opennlp.tools.sentdetect.**SentenceModel.**validateArtifactMap(**
>> SentenceModel.java:117)
>> at opennlp.tools.util.model.**BaseModel.checkArtifactMap(**
>> BaseModel.java:474)
>> ... 4 more
>>
>> Thanks in advance!
>> Danica
>>
>>
>
Re: Exception: The maxent model is not compatible with the sentence
detector! when training MaxEnt Sentence Detector
Posted by Jörn Kottmann <ko...@gmail.com>.
The error seems to occur when you don't have enough training data (and
train with cutoff 5). Try to train with
more data.
Jörn
On 09/02/2013 07:10 PM, Danica Damljanovic wrote:
> Hi everyone
>
> I am trying to retrain the Sentence Detector, however, I keep getting an
> exception. I get the same result with the command line and
> programmatically. Below is the command I ran and the output.
>
> I use apache-opennlp-1.5.3, Mac OS Lion, and I tried with this sample text
> I found in one of the online tutorials:
>
> "Being at the polls was just like being at church.
> I didn't smell a drop of liquor, and we didn't have a bit of trouble.
> The campaign leading to the election was not so quiet.
> It was marked by controversy, anonymous midnight phone calls and veiled
> threats of violence.
> During the election campaign, both candidates, Davis and Bush, reportedly
> received anonymous telephone calls.
> Ordinary Williams said he , too , was subjected to anonymous calls soon
> after he scheduled the election.
> Many local citizens feared that there would be irregularities at the polls.
> Williams got himself a permit to carry a gun and promised an orderly
> election.
> He attended New York University before switching to Georgetown University
> in Washington."
>
>
> Any hint much appreciated.
>
> bin/opennlp SentenceDetectorTrainer -encoding UTF-8 -lang en -data
> en-sent.train -model en-sent.bin
> Indexing events using cutoff of 5
>
> Computing event counts... done. 9 events
> Indexing... done.
> Sorting and merging events... done. Reduced 9 events to 2.
> Done indexing.
> Incorporating indexed data for training...
> done.
> Number of Event Tokens: 2
> Number of Outcomes: 1
> Number of Predicates: 4
> ...done.
> Computing model parameters ...
> Performing 100 iterations.
> 1: ... loglikelihood=0.0 1.0
> 2: ... loglikelihood=0.0 1.0
> Exception in thread "main" java.lang.IllegalArgumentException:
> opennlp.tools.util.InvalidFormatException: The maxent model is not
> compatible with the sentence detector!
> at opennlp.tools.util.model.BaseModel.checkArtifactMap(BaseModel.java:476)
> at opennlp.tools.sentdetect.SentenceModel.<init>(SentenceModel.java:54)
> at
> opennlp.tools.sentdetect.SentenceDetectorME.train(SentenceDetectorME.java:315)
> at
> opennlp.tools.cmdline.sentdetect.SentenceDetectorTrainerTool.run(SentenceDetectorTrainerTool.java:88)
> at opennlp.tools.cmdline.CLI.main(CLI.java:222)
> Caused by: opennlp.tools.util.InvalidFormatException: The maxent model is
> not compatible with the sentence detector!
> at
> opennlp.tools.sentdetect.SentenceModel.validateArtifactMap(SentenceModel.java:117)
> at opennlp.tools.util.model.BaseModel.checkArtifactMap(BaseModel.java:474)
> ... 4 more
>
> Thanks in advance!
> Danica
>