You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by Riccardo Tasso <ri...@gmail.com> on 2011/09/27 15:37:49 UTC
Problems training my own sentence splitter, with dictionary
I'm trying to use OpenNLP to train a sentence splitter model, following
the manual, and everything is ok with it.
I've noticed that the train method supports a Dictionary of
abbreviations, which tipically give me problems with my sentence
splitter and I wanted to try this strategy.
1) If I train a sentence splitter model passing a Dictionary I've build
and I serialize it (just as I serialize a simpler model) I have problems
in loading the model from file:
SentenceModel model = SentenceDetectorME.train(language, sampleStream,
true, abbreviations, 5, 100);
modelOut = new BufferedOutputStream(new
FileOutputStream("/home/lib/apache-opennlp-1.5.1-incubating/models/it/it-sent.bin"));
model.serialize(modelOut);
[...]
modelIn = new
FileInputStream("/home/lib/apache-opennlp-1.5.1-incubating/models/it/it-sent.bin");
final SentenceModel sentenceModel = new SentenceModel(modelIn);
i. e. the last instruction gives me the following exception:
java.io.IOException: Stream closed
at java.util.zip.ZipInputStream.ensureOpen(ZipInputStream.java:61)
at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:108)
at opennlp.tools.util.model.BaseModel.<init>(BaseModel.java:137)
at opennlp.tools.sentdetect.SentenceModel.<init>(SentenceModel.java:77)
at SentenceDetector.main(SentenceDetector.java:18)
2) I tried to skip the serialization/de-serialization phase, to go on
with my tests:
SentenceModel model = SentenceDetectorME.train(language, sampleStream,
true, abbreviations, 5, 100);
String[] sentences = sentenceDetector.sentDetect(document);
However sentences are splitted also on abbreviations which I declared in
my Dictionary, which isn't exactly what I expected. E.g. : "Sono Mr.
Brown" is uncorrectly splitted in "Sono Mr." and "Brown.".
Can you help me with these two problems, which seems to be different,
but may share the same issue.
Thanks,
Riccardo
Re: Problems training my own sentence splitter, with dictionary
Posted by Jörn Kottmann <ko...@gmail.com>.
On 9/28/11 2:43 PM, Jörn Kottmann wrote:
> On 9/28/11 2:36 PM, Riccardo Tasso wrote:
>> Why is the POSModel a final class?
>
> I guess because I thought that it is not a good idea to sub-class it when
> I wrote it.
>
> Any opinions about that?
If you look at the other model classes some or final and some are not.
These classes are not made to be sub-classed, but that is of course
necessary to
do things we did not "planned" people would do.
+1 to remove final from all model classes, because the current way of
doing it is
inconsistent, and people need to sub-class models to insert custom
resources.
We can do this change for our next release, do you want to take care of it?
Jörn
Re: Problems training my own sentence splitter, with dictionary
Posted by Jörn Kottmann <ko...@gmail.com>.
On 9/28/11 2:36 PM, Riccardo Tasso wrote:
> Why is the POSModel a final class?
I guess because I thought that it is not a good idea to sub-class it when
I wrote it.
Any opinions about that?
Jörn
Re: Problems training my own sentence splitter, with dictionary
Posted by Riccardo Tasso <ri...@gmail.com>.
On 28/09/2011 00:17, Jörn Kottmann wrote:
> As said in the other mail, you would need to sub-class the Dictionary
> class, and the POSModel class. If there are
> any issues with that please let us know so we can improve our code.
Why is the POSModel a final class?
Riccardo
Re: Problems training my own sentence splitter, with dictionary
Posted by Jörn Kottmann <ko...@gmail.com>.
On 9/30/11 12:23 AM, Riccardo Tasso wrote:
>
> On 28/09/2011 12:55, Jörn Kottmann wrote:
>> One of the issues we have with "custom" resources is that we don't
>> yet have a generic mechanism to create them.
>> Lets say you store a custom dictionary, in your data format int he
>> model, then the model loading code needs to
>> know how to parse this dictionary with the custom dictionary class.
>
> I sadly discovered what you mean after a whole day trying to implement
> the POSDictionary interface! At least I learned a lot on the actual
> implementation.
>
> Do you think that to handle the customization of POSDictionary is
> required a big re-organization of POSModel? Where would you start? I
> have the time to help you in the development as I decided to base my
> work on OpenNLP.
No, you need to sub-class it, and then at least override
createArtifactSerializers. There you register an Artifact Serializer
for your custom dictionary if it is based on a resource you want to
store in the model.
You then also need to overrride getTagDictionary to return your
instance, I guess we should change that so it returns a
TagDictionary and not a POSDictionary (please open a jira for that).
Hope that helps, don't hesitate to ask more questions.
To improve the situation we need to define a way to do this "custom"
resource loading in a more generic way,
e.g. placing a class name in a model. Maybe we should start to work on that.
Jörn
Re: Problems training my own sentence splitter, with dictionary
Posted by Riccardo Tasso <ri...@gmail.com>.
On 28/09/2011 12:55, Jörn Kottmann wrote:
> One of the issues we have with "custom" resources is that we don't yet
> have a generic mechanism to create them.
> Lets say you store a custom dictionary, in your data format int he
> model, then the model loading code needs to
> know how to parse this dictionary with the custom dictionary class.
I sadly discovered what you mean after a whole day trying to implement
the POSDictionary interface! At least I learned a lot on the actual
implementation.
Do you think that to handle the customization of POSDictionary is
required a big re-organization of POSModel? Where would you start? I
have the time to help you in the development as I decided to base my
work on OpenNLP.
Thanks,
Riccardo
Re: Problems training my own sentence splitter, with dictionary
Posted by Jörn Kottmann <ko...@gmail.com>.
On 9/28/11 12:44 PM, Riccardo Tasso wrote:
>
>> We once discussed to implement bloom filter based dictionaries, maybe
>> that would also be an option for you.
> Interesting, can you give me more insights about it?
Have a look here:
https://issues.apache.org/jira/browse/OPENNLP-88
>> Well, the current implementation cannot really be sub-classed. If you
>> need to replace it the way to go should be to implement
>> the TagDictionary interface yourself.
>
> Of course, but I still have a problem. The POSTaggerME.train method
> takes as a parameter a POSDictionary object, not a TagDictionary. What
> should I do?
Our current code doesn't support a custom TagDictionary. You would need
to implement this train method yourself, otherwise it cannot
create your custom POSModel implementation. I am sorry that things need
to be hacked here ...
One of the issues we have with "custom" resources is that we don't yet
have a generic mechanism to create them.
Lets say you store a custom dictionary, in your data format int he
model, then the model loading code needs to
know how to parse this dictionary with the custom dictionary class.
One way to solve this could be to place the class name of the dictionary
class in the model itself, then model loading code
could load the class to parse the model. A draw back of this approach is
that it might be tricky to get to work in an OSGi container,
but I guess there is a solution for this also. It just needs to be
investigated.
>> A List<String> or a String[] will not work as keys for a Map. We have
>> a StringList
>> object for this case. It contains a sequence of String objects.
>>
>> Can you explain the structure a little, I cannot follow here.
>> Is the key a sequence of tokens?
>
> Yes, I mean a sequence of tokens w.r.t. what you said in a previous
> message. There could be the need to assign a tag to a certain sequence
> of tokens.
A sequence of tokens could also have multiple tags per token. So we
would need something like this I believe:
Map<StringList, String[][]tags>
Jörn
Re: Problems training my own sentence splitter, with dictionary
Posted by Riccardo Tasso <ri...@gmail.com>.
On 28/09/2011 12:32, Jörn Kottmann wrote:
> We once discussed to implement bloom filter based dictionaries, maybe
> that would also be an option for you.
Interesting, can you give me more insights about it?
> Well, the current implementation cannot really be sub-classed. If you
> need to replace it the way to go should be to implement
> the TagDictionary interface yourself.
Of course, but I still have a problem. The POSTaggerME.train method
takes as a parameter a POSDictionary object, not a TagDictionary. What
should I do?
> A List<String> or a String[] will not work as keys for a Map. We have
> a StringList
> object for this case. It contains a sequence of String objects.
>
> Can you explain the structure a little, I cannot follow here.
> Is the key a sequence of tokens?
Yes, I mean a sequence of tokens w.r.t. what you said in a previous
message. There could be the need to assign a tag to a certain sequence
of tokens.
Riccardo
Re: Problems training my own sentence splitter, with dictionary
Posted by Jörn Kottmann <ko...@gmail.com>.
On 9/28/11 12:12 PM, Riccardo Tasso wrote:
>
> Hence at this moment the POSDictionary has the only aim to filter out
> invalid tags?
>
Yes. The POS Tagger takes an input token and gets back an array of valid
tags for this token,
if the token is not included in the dictionary, the POS Tagger can
assign any tag to it.
The POS Dictionary makes the POS Tagger also faster because it has to
consider less possible
sequences.
>> You are welcome to submit a patch to document our pos dict xml format.
>
> I'll look for the xml when I have enough time, and I woul be happy to
> contribute. By now I'm just trying to extend it, because I really
> can't take a full dictionary of possible tags in memory.
Why is that? Usually you only have around 100K words in a language. If
you cannot keep the dictionary in memory the
POS Tagger might get very slow.
We once discussed to implement bloom filter based dictionaries, maybe
that would also be an option for you.
> A first improvement, from my poin of view would be that of making the
> fields of the class protected, to make extension more clean.
>
Well, the current implementation cannot really be sub-classed. If you
need to replace it the way to go should be to implement
the TagDictionary interface yourself.
>
>>
>>> I would prefear a String[] get(String word) and a void put(String
>>> word, String[] tags) methods.
>>
>> For safety and thready safety reasons all our resources used during
>> tagging should be immutable,
>> well, that doesn't mean that we should not have an easy way to create
>> these resources.
>>
>> We have the get method, but it is called getTags.
>>
>> Jörn
>
> What do you think about this structure for the next version of
> POSDictionary?
> Map<List<String>, Set<String>>
> or the ligthweight version:
> Map<String[], String[]>
A List<String> or a String[] will not work as keys for a Map. We have a
StringList
object for this case. It contains a sequence of String objects.
Can you explain the structure a little, I cannot follow here.
Is the key a sequence of tokens?
Jörn
Re: Problems training my own sentence splitter, with dictionary
Posted by Riccardo Tasso <ri...@gmail.com>.
On 28/09/2011 11:46, Jörn Kottmann wrote:
> On 9/28/11 11:34 AM, Riccardo Tasso wrote:
>> This isn't a bug, but why can I load a POSDictionary from an xml
>> format which is undocumented?
>
> We previously had a plain/text format, which was replaced by this xml
> format. Because of
> encoding issues. I think we will do a couple of refactoring and
> redesign of the POS Tagger and
> then again improve the POS Dictionary and other dictionaries we
> currently have.
>
> There are a couple of things which can be done better, e.g. when the
> dictionary only allows one tag
> we do not need to call the classifier to make a decision, the
> dictionary should also support token sequences,
> etc.
Hence at this moment the POSDictionary has the only aim to filter out
invalid tags?
> You are welcome to submit a patch to document our pos dict xml format.
I'll look for the xml when I have enough time, and I woul be happy to
contribute. By now I'm just trying to extend it, because I really can't
take a full dictionary of possible tags in memory.
A first improvement, from my poin of view would be that of making the
fields of the class protected, to make extension more clean.
>
>> I would prefear a String[] get(String word) and a void put(String
>> word, String[] tags) methods.
>
> For safety and thready safety reasons all our resources used during
> tagging should be immutable,
> well, that doesn't mean that we should not have an easy way to create
> these resources.
>
> We have the get method, but it is called getTags.
>
> Jörn
What do you think about this structure for the next version of
POSDictionary?
Map<List<String>, Set<String>>
or the ligthweight version:
Map<String[], String[]>
My two cents,
Riccardo
Re: Problems training my own sentence splitter, with dictionary
Posted by Jörn Kottmann <ko...@gmail.com>.
On 9/28/11 11:34 AM, Riccardo Tasso wrote:
> This isn't a bug, but why can I load a POSDictionary from an xml
> format which is undocumented?
We previously had a plain/text format, which was replaced by this xml
format. Because of
encoding issues. I think we will do a couple of refactoring and redesign
of the POS Tagger and
then again improve the POS Dictionary and other dictionaries we
currently have.
There are a couple of things which can be done better, e.g. when the
dictionary only allows one tag
we do not need to call the classifier to make a decision, the dictionary
should also support token sequences,
etc.
You are welcome to submit a patch to document our pos dict xml format.
> I would prefear a String[] get(String word) and a void put(String
> word, String[] tags) methods.
For safety and thready safety reasons all our resources used during
tagging should be immutable,
well, that doesn't mean that we should not have an easy way to create
these resources.
We have the get method, but it is called getTags.
Jörn
Re: Problems training my own sentence splitter, with dictionary
Posted by Riccardo Tasso <ri...@gmail.com>.
This isn't a bug, but why can I load a POSDictionary from an xml format
which is undocumented? I would prefear a String[] get(String word) and a
void put(String word, String[] tags) methods.
Riccardo
On 28/09/2011 10:54, Jörn Kottmann wrote:
> On 9/28/11 3:23 AM, James Kosin wrote:
>> The POS Dictinary class really needs a shakedown, because right now a
>> lot of items are broken.
>
> What exactly is broken? I don't know of any bugs in the POS Dictionary.
>
> Jörn
Re: Problems training my own sentence splitter, with dictionary
Posted by Jörn Kottmann <ko...@gmail.com>.
On 9/28/11 3:23 AM, James Kosin wrote:
> The POS Dictinary class really needs a shakedown, because right now a
> lot of items are broken.
What exactly is broken? I don't know of any bugs in the POS Dictionary.
Jörn
Re: Problems training my own sentence splitter, with dictionary
Posted by James Kosin <ja...@gmail.com>.
On 9/27/2011 6:17 PM, Jörn Kottmann wrote:
> On 9/27/11 6:07 PM, Riccardo Tasso wrote:
>> Another question about dictionaries: I'm interested to implement my
>> own Dictionary classes (especially for POSDictionary) whith its own
>> backend (e.g. Redis instead of memory). It wouldn't be better if
>> train method had a Dictionary interface, instead of a class as a
>> parameter?
>
> Be aware of the fact that the POS Tagger is accessing the dictionary
> very frequently, the current memory
> dictionary does a couple of thousand look ups per second.
>
> As said in the other mail, you would need to sub-class the Dictionary
> class, and the POSModel class. If there are
> any issues with that please let us know so we can improve our code.
>
> Jörn
Jorn,
The POS Dictinary class really needs a shakedown, because right now a
lot of items are broken. I'm waiting until after this release to
re-vamp the class so... it both works correctly (with the case
sensitivity flag) and a consistent interface like the Dictionary class
already has.
James
Re: Problems training my own sentence splitter, with dictionary
Posted by Jörn Kottmann <ko...@gmail.com>.
On 9/27/11 6:07 PM, Riccardo Tasso wrote:
> Another question about dictionaries: I'm interested to implement my
> own Dictionary classes (especially for POSDictionary) whith its own
> backend (e.g. Redis instead of memory). It wouldn't be better if train
> method had a Dictionary interface, instead of a class as a parameter?
Be aware of the fact that the POS Tagger is accessing the dictionary
very frequently, the current memory
dictionary does a couple of thousand look ups per second.
As said in the other mail, you would need to sub-class the Dictionary
class, and the POSModel class. If there are
any issues with that please let us know so we can improve our code.
Jörn
Re: Problems training my own sentence splitter, with dictionary
Posted by Jörn Kottmann <ko...@gmail.com>.
On 9/28/11 12:07 AM, Riccardo Tasso wrote:
> In the meanwhile I think the simplest solution to implement my own
> dictionary is to extend the original one.
This will not work. You also need to subclass the model
class. Otherwise it doesn't know that the dictionary must be deserialized
with your dictionary implementation.
Jörn
Re: Problems training my own sentence splitter, with dictionary
Posted by Riccardo Tasso <ri...@gmail.com>.
In the meanwhile I think the simplest solution to implement my own
dictionary is to extend the original one.
Thanks,
Riccardo
Il giorno 28/set/2011 00:03, "william.colen@gmail.com" <
william.colen@gmail.com> ha scritto:
> Hi, Ricardo,
>
> On Tue, Sep 27, 2011 at 1:07 PM, Riccardo Tasso <riccardo.tasso@gmail.com
>wrote:
>
>> Hi William,
>> the upgrade solved my problem with the serialization of the model, thank
>> you.
>>
>> Another question about dictionaries: I'm interested to implement my own
>> Dictionary classes (especially for POSDictionary) whith its own backend
>> (e.g. Redis instead of memory). It wouldn't be better if train method had
a
>> Dictionary interface, instead of a class as a parameter?
>>
>
> +1 to include a Dictionary interface, but we will have to wait for the
next
> major release because we want 1.5.2 to be backward compatible with 1.5.x
>
> We already have the TagDictionary interface and the POSDictionary
implements
> it. You can use this interface to implement your own tag dictionary. We
have
> other constructors at POSTaggerME that takes a Dictionary, but they are
the
> n-gram dictionaries. N-gram dictionaries in POSTagger is an experimental
> feature and we was discussing if it should be removed or not.
>
> William
>
>
>>
>> Thank you,
>> Riccardo
>>
>>
>> On 27/09/2011 16:17, william.colen@gmail.com wrote:
>>
>>> Hi, Riccardo,
>>>
>>> We improved a little how the abbreviation dictionary is handled in
OpenNLP
>>> 1.5.2. You should try it using the current release candidate:
>>> http://people.apache.org/~****joern/releases/opennlp-1.5.2-***
>>> *incubating/rc1/<
http://people.apache.org/~**joern/releases/opennlp-1.5.2-**incubating/rc1/>
>>> <http://**people.apache.org/~joern/**releases/opennlp-1.5.2-**
>>> incubating/rc1/<
http://people.apache.org/~joern/releases/opennlp-1.5.2-incubating/rc1/>
>>> >
>>>
>>>
>>> 1.5.2 includes a command line tool named DictionaryBuilder:
>>>
>>> $ bin/opennlp DictionaryBuilder
>>> Usage: opennlp DictionaryBuilder -inputFile in -outputFile out
[-encoding
>>> charsetName]
>>>
>>> Arguments description:
>>> -inputFile in
>>> Plain file with one entry per line
>>> -outputFile out
>>> The dictionary file.
>>> -encoding charsetName
>>> specifies the encoding which should be used for reading and writing
text.
>>> If not specified the system default will be used.
>>>
>>> This tool should be used to create an abbreviation dictionary in XML
>>> format
>>> as expected by the SentenceDetector and Tokenizer tools.
>>>
>>> Regards,
>>> William
>>>
>>>
>>> On Tue, Sep 27, 2011 at 10:37 AM, Riccardo Tasso
>>> <ri...@gmail.com>**wrote:
>>>
>>> I'm trying to use OpenNLP to train a sentence splitter model, following
>>>> the
>>>> manual, and everything is ok with it.
>>>>
>>>> I've noticed that the train method supports a Dictionary of
>>>> abbreviations,
>>>> which tipically give me problems with my sentence splitter and I wanted
>>>> to
>>>> try this strategy.
>>>>
>>>> 1) If I train a sentence splitter model passing a Dictionary I've build
>>>> and
>>>> I serialize it (just as I serialize a simpler model) I have problems in
>>>> loading the model from file:
>>>>
>>>> SentenceModel model = SentenceDetectorME.train(****language,
>>>> sampleStream,
>>>> true, abbreviations, 5, 100);
>>>> modelOut = new BufferedOutputStream(new FileOutputStream("/home/lib/**
>>>> apache-opennlp-1.5.1-****incubating/models/it/it-sent.****bin"));
>>>> model.serialize(modelOut);
>>>> [...]
>>>> modelIn = new FileInputStream("/home/lib/****apache-opennlp-1.5.1-**
>>>> incubating/models/it/it-sent.****bin");
>>>> final SentenceModel sentenceModel = new SentenceModel(modelIn);
>>>>
>>>> i. e. the last instruction gives me the following exception:
>>>> java.io.IOException: Stream closed
>>>> at java.util.zip.ZipInputStream.****ensureOpen(ZipInputStream.****
>>>> java:61)
>>>> at java.util.zip.ZipInputStream.****closeEntry(ZipInputStream.****
>>>> java:108)
>>>> at opennlp.tools.util.model.****BaseModel.<init>(BaseModel.****
>>>> java:137)
>>>> at opennlp.tools.sentdetect.****SentenceModel.<init>(**
>>>> SentenceModel.java:77)
>>>> at SentenceDetector.main(****SentenceDetector.java:18)
>>>>
>>>> 2) I tried to skip the serialization/de-serialization phase, to go on
>>>> with
>>>> my tests:
>>>> SentenceModel model = SentenceDetectorME.train(****language,
>>>> sampleStream,
>>>> true, abbreviations, 5, 100);
>>>> String[] sentences = sentenceDetector.sentDetect(****document);
>>>>
>>>> However sentences are splitted also on abbreviations which I declared
in
>>>> my
>>>> Dictionary, which isn't exactly what I expected. E.g. : "Sono Mr.
Brown"
>>>> is
>>>> uncorrectly splitted in "Sono Mr." and "Brown.".
>>>>
>>>> Can you help me with these two problems, which seems to be different,
but
>>>> may share the same issue.
>>>>
>>>> Thanks,
>>>> Riccardo
>>>>
>>>>
>>
Re: Problems training my own sentence splitter, with dictionary
Posted by "william.colen@gmail.com" <wi...@gmail.com>.
Hi, Ricardo,
On Tue, Sep 27, 2011 at 1:07 PM, Riccardo Tasso <ri...@gmail.com>wrote:
> Hi William,
> the upgrade solved my problem with the serialization of the model, thank
> you.
>
> Another question about dictionaries: I'm interested to implement my own
> Dictionary classes (especially for POSDictionary) whith its own backend
> (e.g. Redis instead of memory). It wouldn't be better if train method had a
> Dictionary interface, instead of a class as a parameter?
>
+1 to include a Dictionary interface, but we will have to wait for the next
major release because we want 1.5.2 to be backward compatible with 1.5.x
We already have the TagDictionary interface and the POSDictionary implements
it. You can use this interface to implement your own tag dictionary. We have
other constructors at POSTaggerME that takes a Dictionary, but they are the
n-gram dictionaries. N-gram dictionaries in POSTagger is an experimental
feature and we was discussing if it should be removed or not.
William
>
> Thank you,
> Riccardo
>
>
> On 27/09/2011 16:17, william.colen@gmail.com wrote:
>
>> Hi, Riccardo,
>>
>> We improved a little how the abbreviation dictionary is handled in OpenNLP
>> 1.5.2. You should try it using the current release candidate:
>> http://people.apache.org/~****joern/releases/opennlp-1.5.2-***
>> *incubating/rc1/<http://people.apache.org/~**joern/releases/opennlp-1.5.2-**incubating/rc1/>
>> <http://**people.apache.org/~joern/**releases/opennlp-1.5.2-**
>> incubating/rc1/<http://people.apache.org/~joern/releases/opennlp-1.5.2-incubating/rc1/>
>> >
>>
>>
>> 1.5.2 includes a command line tool named DictionaryBuilder:
>>
>> $ bin/opennlp DictionaryBuilder
>> Usage: opennlp DictionaryBuilder -inputFile in -outputFile out [-encoding
>> charsetName]
>>
>> Arguments description:
>> -inputFile in
>> Plain file with one entry per line
>> -outputFile out
>> The dictionary file.
>> -encoding charsetName
>> specifies the encoding which should be used for reading and writing text.
>> If not specified the system default will be used.
>>
>> This tool should be used to create an abbreviation dictionary in XML
>> format
>> as expected by the SentenceDetector and Tokenizer tools.
>>
>> Regards,
>> William
>>
>>
>> On Tue, Sep 27, 2011 at 10:37 AM, Riccardo Tasso
>> <ri...@gmail.com>**wrote:
>>
>> I'm trying to use OpenNLP to train a sentence splitter model, following
>>> the
>>> manual, and everything is ok with it.
>>>
>>> I've noticed that the train method supports a Dictionary of
>>> abbreviations,
>>> which tipically give me problems with my sentence splitter and I wanted
>>> to
>>> try this strategy.
>>>
>>> 1) If I train a sentence splitter model passing a Dictionary I've build
>>> and
>>> I serialize it (just as I serialize a simpler model) I have problems in
>>> loading the model from file:
>>>
>>> SentenceModel model = SentenceDetectorME.train(****language,
>>> sampleStream,
>>> true, abbreviations, 5, 100);
>>> modelOut = new BufferedOutputStream(new FileOutputStream("/home/lib/**
>>> apache-opennlp-1.5.1-****incubating/models/it/it-sent.****bin"));
>>> model.serialize(modelOut);
>>> [...]
>>> modelIn = new FileInputStream("/home/lib/****apache-opennlp-1.5.1-**
>>> incubating/models/it/it-sent.****bin");
>>> final SentenceModel sentenceModel = new SentenceModel(modelIn);
>>>
>>> i. e. the last instruction gives me the following exception:
>>> java.io.IOException: Stream closed
>>> at java.util.zip.ZipInputStream.****ensureOpen(ZipInputStream.****
>>> java:61)
>>> at java.util.zip.ZipInputStream.****closeEntry(ZipInputStream.****
>>> java:108)
>>> at opennlp.tools.util.model.****BaseModel.<init>(BaseModel.****
>>> java:137)
>>> at opennlp.tools.sentdetect.****SentenceModel.<init>(**
>>> SentenceModel.java:77)
>>> at SentenceDetector.main(****SentenceDetector.java:18)
>>>
>>> 2) I tried to skip the serialization/de-serialization phase, to go on
>>> with
>>> my tests:
>>> SentenceModel model = SentenceDetectorME.train(****language,
>>> sampleStream,
>>> true, abbreviations, 5, 100);
>>> String[] sentences = sentenceDetector.sentDetect(****document);
>>>
>>> However sentences are splitted also on abbreviations which I declared in
>>> my
>>> Dictionary, which isn't exactly what I expected. E.g. : "Sono Mr. Brown"
>>> is
>>> uncorrectly splitted in "Sono Mr." and "Brown.".
>>>
>>> Can you help me with these two problems, which seems to be different, but
>>> may share the same issue.
>>>
>>> Thanks,
>>> Riccardo
>>>
>>>
>
Re: Problems training my own sentence splitter, with dictionary
Posted by Riccardo Tasso <ri...@gmail.com>.
Hi William,
the upgrade solved my problem with the serialization of the model,
thank you.
Another question about dictionaries: I'm interested to implement my own
Dictionary classes (especially for POSDictionary) whith its own backend
(e.g. Redis instead of memory). It wouldn't be better if train method
had a Dictionary interface, instead of a class as a parameter?
Thank you,
Riccardo
On 27/09/2011 16:17, william.colen@gmail.com wrote:
> Hi, Riccardo,
>
> We improved a little how the abbreviation dictionary is handled in OpenNLP
> 1.5.2. You should try it using the current release candidate:
> http://people.apache.org/~**joern/releases/opennlp-1.5.2-**incubating/rc1/<http://people.apache.org/~joern/releases/opennlp-1.5.2-incubating/rc1/>
>
> 1.5.2 includes a command line tool named DictionaryBuilder:
>
> $ bin/opennlp DictionaryBuilder
> Usage: opennlp DictionaryBuilder -inputFile in -outputFile out [-encoding
> charsetName]
>
> Arguments description:
> -inputFile in
> Plain file with one entry per line
> -outputFile out
> The dictionary file.
> -encoding charsetName
> specifies the encoding which should be used for reading and writing text.
> If not specified the system default will be used.
>
> This tool should be used to create an abbreviation dictionary in XML format
> as expected by the SentenceDetector and Tokenizer tools.
>
> Regards,
> William
>
>
> On Tue, Sep 27, 2011 at 10:37 AM, Riccardo Tasso
> <ri...@gmail.com>wrote:
>
>> I'm trying to use OpenNLP to train a sentence splitter model, following the
>> manual, and everything is ok with it.
>>
>> I've noticed that the train method supports a Dictionary of abbreviations,
>> which tipically give me problems with my sentence splitter and I wanted to
>> try this strategy.
>>
>> 1) If I train a sentence splitter model passing a Dictionary I've build and
>> I serialize it (just as I serialize a simpler model) I have problems in
>> loading the model from file:
>>
>> SentenceModel model = SentenceDetectorME.train(**language, sampleStream,
>> true, abbreviations, 5, 100);
>> modelOut = new BufferedOutputStream(new FileOutputStream("/home/lib/**
>> apache-opennlp-1.5.1-**incubating/models/it/it-sent.**bin"));
>> model.serialize(modelOut);
>> [...]
>> modelIn = new FileInputStream("/home/lib/**apache-opennlp-1.5.1-**
>> incubating/models/it/it-sent.**bin");
>> final SentenceModel sentenceModel = new SentenceModel(modelIn);
>>
>> i. e. the last instruction gives me the following exception:
>> java.io.IOException: Stream closed
>> at java.util.zip.ZipInputStream.**ensureOpen(ZipInputStream.**java:61)
>> at java.util.zip.ZipInputStream.**closeEntry(ZipInputStream.**java:108)
>> at opennlp.tools.util.model.**BaseModel.<init>(BaseModel.**java:137)
>> at opennlp.tools.sentdetect.**SentenceModel.<init>(**
>> SentenceModel.java:77)
>> at SentenceDetector.main(**SentenceDetector.java:18)
>>
>> 2) I tried to skip the serialization/de-serialization phase, to go on with
>> my tests:
>> SentenceModel model = SentenceDetectorME.train(**language, sampleStream,
>> true, abbreviations, 5, 100);
>> String[] sentences = sentenceDetector.sentDetect(**document);
>>
>> However sentences are splitted also on abbreviations which I declared in my
>> Dictionary, which isn't exactly what I expected. E.g. : "Sono Mr. Brown" is
>> uncorrectly splitted in "Sono Mr." and "Brown.".
>>
>> Can you help me with these two problems, which seems to be different, but
>> may share the same issue.
>>
>> Thanks,
>> Riccardo
>>
Re: Problems training my own sentence splitter, with dictionary
Posted by Jörn Kottmann <ko...@gmail.com>.
Hi,
one unrelated note, we also fixed quite a few white space handling issues
in this version which might increase the performance for you.
The dictionary is only used to generate features based on the abbreviations
and does not force the sentence detector not to split.
Jörn
On 9/27/11 4:17 PM, william.colen@gmail.com wrote:
> Hi, Riccardo,
>
> We improved a little how the abbreviation dictionary is handled in OpenNLP
> 1.5.2. You should try it using the current release candidate:
> http://people.apache.org/~**joern/releases/opennlp-1.5.2-**incubating/rc1/<http://people.apache.org/~joern/releases/opennlp-1.5.2-incubating/rc1/>
>
> 1.5.2 includes a command line tool named DictionaryBuilder:
>
> $ bin/opennlp DictionaryBuilder
> Usage: opennlp DictionaryBuilder -inputFile in -outputFile out [-encoding
> charsetName]
>
> Arguments description:
> -inputFile in
> Plain file with one entry per line
> -outputFile out
> The dictionary file.
> -encoding charsetName
> specifies the encoding which should be used for reading and writing text.
> If not specified the system default will be used.
>
> This tool should be used to create an abbreviation dictionary in XML format
> as expected by the SentenceDetector and Tokenizer tools.
>
> Regards,
> William
>
>
> On Tue, Sep 27, 2011 at 10:37 AM, Riccardo Tasso
> <ri...@gmail.com>wrote:
>
>> I'm trying to use OpenNLP to train a sentence splitter model, following the
>> manual, and everything is ok with it.
>>
>> I've noticed that the train method supports a Dictionary of abbreviations,
>> which tipically give me problems with my sentence splitter and I wanted to
>> try this strategy.
>>
>> 1) If I train a sentence splitter model passing a Dictionary I've build and
>> I serialize it (just as I serialize a simpler model) I have problems in
>> loading the model from file:
>>
>> SentenceModel model = SentenceDetectorME.train(**language, sampleStream,
>> true, abbreviations, 5, 100);
>> modelOut = new BufferedOutputStream(new FileOutputStream("/home/lib/**
>> apache-opennlp-1.5.1-**incubating/models/it/it-sent.**bin"));
>> model.serialize(modelOut);
>> [...]
>> modelIn = new FileInputStream("/home/lib/**apache-opennlp-1.5.1-**
>> incubating/models/it/it-sent.**bin");
>> final SentenceModel sentenceModel = new SentenceModel(modelIn);
>>
>> i. e. the last instruction gives me the following exception:
>> java.io.IOException: Stream closed
>> at java.util.zip.ZipInputStream.**ensureOpen(ZipInputStream.**java:61)
>> at java.util.zip.ZipInputStream.**closeEntry(ZipInputStream.**java:108)
>> at opennlp.tools.util.model.**BaseModel.<init>(BaseModel.**java:137)
>> at opennlp.tools.sentdetect.**SentenceModel.<init>(**
>> SentenceModel.java:77)
>> at SentenceDetector.main(**SentenceDetector.java:18)
>>
>> 2) I tried to skip the serialization/de-serialization phase, to go on with
>> my tests:
>> SentenceModel model = SentenceDetectorME.train(**language, sampleStream,
>> true, abbreviations, 5, 100);
>> String[] sentences = sentenceDetector.sentDetect(**document);
>>
>> However sentences are splitted also on abbreviations which I declared in my
>> Dictionary, which isn't exactly what I expected. E.g. : "Sono Mr. Brown" is
>> uncorrectly splitted in "Sono Mr." and "Brown.".
>>
>> Can you help me with these two problems, which seems to be different, but
>> may share the same issue.
>>
>> Thanks,
>> Riccardo
>>
Re: Problems training my own sentence splitter, with dictionary
Posted by "william.colen@gmail.com" <wi...@gmail.com>.
Hi, Riccardo,
We improved a little how the abbreviation dictionary is handled in OpenNLP
1.5.2. You should try it using the current release candidate:
http://people.apache.org/~**joern/releases/opennlp-1.5.2-**incubating/rc1/<http://people.apache.org/~joern/releases/opennlp-1.5.2-incubating/rc1/>
1.5.2 includes a command line tool named DictionaryBuilder:
$ bin/opennlp DictionaryBuilder
Usage: opennlp DictionaryBuilder -inputFile in -outputFile out [-encoding
charsetName]
Arguments description:
-inputFile in
Plain file with one entry per line
-outputFile out
The dictionary file.
-encoding charsetName
specifies the encoding which should be used for reading and writing text.
If not specified the system default will be used.
This tool should be used to create an abbreviation dictionary in XML format
as expected by the SentenceDetector and Tokenizer tools.
Regards,
William
On Tue, Sep 27, 2011 at 10:37 AM, Riccardo Tasso
<ri...@gmail.com>wrote:
> I'm trying to use OpenNLP to train a sentence splitter model, following the
> manual, and everything is ok with it.
>
> I've noticed that the train method supports a Dictionary of abbreviations,
> which tipically give me problems with my sentence splitter and I wanted to
> try this strategy.
>
> 1) If I train a sentence splitter model passing a Dictionary I've build and
> I serialize it (just as I serialize a simpler model) I have problems in
> loading the model from file:
>
> SentenceModel model = SentenceDetectorME.train(**language, sampleStream,
> true, abbreviations, 5, 100);
> modelOut = new BufferedOutputStream(new FileOutputStream("/home/lib/**
> apache-opennlp-1.5.1-**incubating/models/it/it-sent.**bin"));
> model.serialize(modelOut);
> [...]
> modelIn = new FileInputStream("/home/lib/**apache-opennlp-1.5.1-**
> incubating/models/it/it-sent.**bin");
> final SentenceModel sentenceModel = new SentenceModel(modelIn);
>
> i. e. the last instruction gives me the following exception:
> java.io.IOException: Stream closed
> at java.util.zip.ZipInputStream.**ensureOpen(ZipInputStream.**java:61)
> at java.util.zip.ZipInputStream.**closeEntry(ZipInputStream.**java:108)
> at opennlp.tools.util.model.**BaseModel.<init>(BaseModel.**java:137)
> at opennlp.tools.sentdetect.**SentenceModel.<init>(**
> SentenceModel.java:77)
> at SentenceDetector.main(**SentenceDetector.java:18)
>
> 2) I tried to skip the serialization/de-serialization phase, to go on with
> my tests:
> SentenceModel model = SentenceDetectorME.train(**language, sampleStream,
> true, abbreviations, 5, 100);
> String[] sentences = sentenceDetector.sentDetect(**document);
>
> However sentences are splitted also on abbreviations which I declared in my
> Dictionary, which isn't exactly what I expected. E.g. : "Sono Mr. Brown" is
> uncorrectly splitted in "Sono Mr." and "Brown.".
>
> Can you help me with these two problems, which seems to be different, but
> may share the same issue.
>
> Thanks,
> Riccardo
>