You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by Rodrigo Agerri <ro...@ehu.es> on 2013/04/10 11:27:57 UTC
English lemmatizer using wordnet
Hi,
I have put together a very simple English lemmatizer using JWNL as
part of a POS module based on opennlp API. The lemmatizer uses the
constructor of JWNLDictionary in opennlp coref package.
If there is interest in explicitly providing an English lemmatizer in
the project I would not mind providing the code.
You can try the thing in:
https://bitbucket.org/ragerri/ixa-pipe-pos
Cheers,
Rodrigo
Re: English lemmatizer using wordnet
Posted by Rodrigo Agerri <ro...@ehu.es>.
On Fri, Apr 12, 2013 at 3:31 PM, Aliaksandr Autayeu
<al...@autayeu.com> wrote:
>>
>
>> removes the dependence on a dictionary.
>>
> Impossible. There are always exceptions which need to be stored somewhere.
> WordNet stores the list of exceptions.
I meant dependency on a third-party maintained dictionary accessible
with a third-party API, e.g., the case of JWNL and Princeton WordNet.
Cheers,
Rodrigo
Re: English lemmatizer using wordnet
Posted by Aliaksandr Autayeu <al...@autayeu.com>.
>
> mappings) from the lemmatization functionality itself. For example,
> JWNL only works for English original WordNet. I am working on a
> Spanish pos + lemmatizer using this approach.
>
I did it for Italian.
> removes the dependence on a dictionary.
>
Impossible. There are always exceptions which need to be stored somewhere.
WordNet stores the list of exceptions.
Re: English lemmatizer using wordnet
Posted by Jörn Kottmann <ko...@gmail.com>.
They moved to github, here is the new link:
https://github.com/morfologik
Jörn
On 04/10/2013 02:34 PM, William Colen wrote:
> Hi,
>
> +1 for a lemmatizer API
>
> For my Master's project I created a lemma dictionary, which keys were the
> [token + POS tag] and the value one or more lemmas.
>
> To store and access the entries I used a very nice Java tool available
> under BSD license that is part of the Morfologik tool (
> http://sourceforge.net/projects/morfologik). This tool encodes the
> dictionary in a finite-state automata, allowing a very efficient access and
> a compact dictionary.
> The tool also provide a efficient way of encoding and accessing lexical
> dictionaries.
>
> The LanguageTools members wrote a tutorial on how to use Morfologik for
> this: http://wiki.languagetool.org/developing-a-tagger-dictionary
>
>
>
> On Wed, Apr 10, 2013 at 9:02 AM, Rodrigo Agerri <ro...@ehu.es>wrote:
>
>> On Wed, Apr 10, 2013 at 1:00 PM, Jörn Kottmann <ko...@gmail.com>
>> wrote:>
>>> +1, it would be nice to have control over the dictionary, maybe we can
>> come
>>> up with
>>> a format to store it in. That will allow us to easily include it in our
>>> models
>>> as a resource for feature generation and eliminates the dependency on
>>> external libraries.
>> I do not know yet which dictionary format will be best, but I can try
>> to come up with a proposal independent of WordNet or other third party
>> resources, when I have it working, and then discuss it.
>>
>>> +1
>>>
>>> We should define an interface which allows to use different
>> implementations
>>> like
>>> we did for the other components.
>> OK.
>>
>> Cheers,
>>
>> Rodrigo
>>
Re: English lemmatizer using wordnet
Posted by Rodrigo Agerri <ro...@ehu.es>.
On Wed, Apr 10, 2013 at 4:22 PM, Jörn Kottmann <ko...@gmail.com> wrote:
>
> The current dictionary package has an API to read and serialize a dictionary
> from and to the
> XML format. That could be changed to some binary based format which could be
> much faster.
> But as far as I understand is the main issue we have is the representation
> of the dictionary in memory
> and not the serialization of it.
Having a lemmatization API like this would be very nice.
Using JWNL just for a lemma lookup was quick but I actually think is
overkill. I have created now a "wordform pos lemma" dictionary from
WordNet and will perform hashmap lookups on it. If this could be done
using an adapted Dictionary API for a more efficient memory and speed
performance, that will be great.
Cheers,
Rodrigo
Re: English lemmatizer using wordnet
Posted by Jörn Kottmann <ko...@gmail.com>.
On 04/10/2013 10:43 PM, William Colen wrote:
> Yes, that would be a good step.
>
> But actually I was always talking about lexical dictionary of the POS
> Tagger, which default XML implementation relies on is called POSDictionary,
> the interface is named TagDictionary. Because it is an interface, to
> implement it using Morfologic FSA was easy. I also created a Featurizer
> component for my thesis, but since I was in a hurry, I did not follow
> OpenNLP structure for that.
Yes I know, I thought it might be a nice solution to have an additional
layer between
the component and the actual dictionary implementation, in that case we
would of course keep
the current xml dictionary, but from your comment below I now got the
impression that it
is not worth it, because it also makes things more complex.
It would probably be nice to have if the dictionary implementations we
could use would be
generic enough to be accessed via one common dictionary interface, but
with Morfologik its probably not
the case.
> I don't know if we can use the Morfologik FSA dictionary for the
> conventional Dictionary class we have, which entries are multiple tokens.
> This is used for the abbreviation dictionaries and for the Name Finder.
> Maybe we can use FSA, but we would have to adapt.
We should do the same for the lemmatizer, and if we one day change our
mind we could
still introduce a dictionary interface.
Jörn
Re: English lemmatizer using wordnet
Posted by William Colen <wi...@gmail.com>.
Yes, that would be a good step.
But actually I was always talking about lexical dictionary of the POS
Tagger, which default XML implementation relies on is called POSDictionary,
the interface is named TagDictionary. Because it is an interface, to
implement it using Morfologic FSA was easy. I also created a Featurizer
component for my thesis, but since I was in a hurry, I did not follow
OpenNLP structure for that.
I don't know if we can use the Morfologik FSA dictionary for the
conventional Dictionary class we have, which entries are multiple tokens.
This is used for the abbreviation dictionaries and for the Name Finder.
Maybe we can use FSA, but we would have to adapt.
On Wed, Apr 10, 2013 at 5:16 PM, Jörn Kottmann <ko...@gmail.com> wrote:
> The serializer we have currently uses a StringList as key for the
> dictionary
> and then encodes the stored information in the Entry object, we could move
> this up to the dictionary level, e.g.:
>
> interface Dictionary {
> Entry get(StringList key);
> }
>
> Would such an abstraction work for the Morfologik FSA dictionary?
>
> We have to see how we can make the interface efficient, there should no
> expensive
> object creation involved for a lookup.
>
> Jörn
>
>
> On 04/10/2013 05:24 PM, William Colen wrote:
>
>> On Wed, Apr 10, 2013 at 11:22 AM, Jörn Kottmann <ko...@gmail.com>
>> wrote:
>>
>>> Is the memory issue is caused by the fact the dictionaries (e.g.
>>> POSDictionary) are using
>>> the Java HashMap and String key/values?
>>>
>>> Yes. The dictionary I have has 800k entries. It is a huge hashmap.
>>
>> Did you implement your own POSDictionary for your thesis?
>> Yes, using Morfologik FSA.
>>
>> The current dictionary package has an API to read and serialize a
>>
>>> dictionary from and to the
>>> XML format. That could be changed to some binary based format which could
>>> be much faster.
>>> But as far as I understand is the main issue we have is the
>>> representation
>>> of the dictionary in memory
>>> and not the serialization of it.
>>>
>>
>> When instantiated, the dictionary XML is loaded to a hashtable. This
>> process takes a few seconds for a 800k entries dictionary, and depending
>> on
>> the requirements it might be an issue.
>>
>> I like the XML implementation, and looks like it works for most of the
>> OpenNLP users. But a binary option would be a plus for the ones that need
>> it.
>>
>> I could store Morfologik FSA dictionaries to the model using the custom
>> factory API, so it is quite transparent for the users, which can load the
>> model even from the CL. The only requirement is to add my Jar with
>> customizations to the classpath.
>>
>>
>
Re: English lemmatizer using wordnet
Posted by Jörn Kottmann <ko...@gmail.com>.
The serializer we have currently uses a StringList as key for the dictionary
and then encodes the stored information in the Entry object, we could move
this up to the dictionary level, e.g.:
interface Dictionary {
Entry get(StringList key);
}
Would such an abstraction work for the Morfologik FSA dictionary?
We have to see how we can make the interface efficient, there should no
expensive
object creation involved for a lookup.
Jörn
On 04/10/2013 05:24 PM, William Colen wrote:
> On Wed, Apr 10, 2013 at 11:22 AM, Jörn Kottmann <ko...@gmail.com> wrote:
>> Is the memory issue is caused by the fact the dictionaries (e.g.
>> POSDictionary) are using
>> the Java HashMap and String key/values?
>>
> Yes. The dictionary I have has 800k entries. It is a huge hashmap.
>
> Did you implement your own POSDictionary for your thesis?
> Yes, using Morfologik FSA.
>
> The current dictionary package has an API to read and serialize a
>> dictionary from and to the
>> XML format. That could be changed to some binary based format which could
>> be much faster.
>> But as far as I understand is the main issue we have is the representation
>> of the dictionary in memory
>> and not the serialization of it.
>
> When instantiated, the dictionary XML is loaded to a hashtable. This
> process takes a few seconds for a 800k entries dictionary, and depending on
> the requirements it might be an issue.
>
> I like the XML implementation, and looks like it works for most of the
> OpenNLP users. But a binary option would be a plus for the ones that need
> it.
>
> I could store Morfologik FSA dictionaries to the model using the custom
> factory API, so it is quite transparent for the users, which can load the
> model even from the CL. The only requirement is to add my Jar with
> customizations to the classpath.
>
Re: English lemmatizer using wordnet
Posted by William Colen <wi...@gmail.com>.
On Wed, Apr 10, 2013 at 11:22 AM, Jörn Kottmann <ko...@gmail.com> wrote:
>
> Is the memory issue is caused by the fact the dictionaries (e.g.
> POSDictionary) are using
> the Java HashMap and String key/values?
>
Yes. The dictionary I have has 800k entries. It is a huge hashmap.
Did you implement your own POSDictionary for your thesis?
>
Yes, using Morfologik FSA.
The current dictionary package has an API to read and serialize a
> dictionary from and to the
> XML format. That could be changed to some binary based format which could
> be much faster.
> But as far as I understand is the main issue we have is the representation
> of the dictionary in memory
> and not the serialization of it.
When instantiated, the dictionary XML is loaded to a hashtable. This
process takes a few seconds for a 800k entries dictionary, and depending on
the requirements it might be an issue.
I like the XML implementation, and looks like it works for most of the
OpenNLP users. But a binary option would be a plus for the ones that need
it.
I could store Morfologik FSA dictionaries to the model using the custom
factory API, so it is quite transparent for the users, which can load the
model even from the CL. The only requirement is to add my Jar with
customizations to the classpath.
Re: English lemmatizer using wordnet
Posted by Jörn Kottmann <ko...@gmail.com>.
On 04/10/2013 03:59 PM, William Colen wrote:
> Yes, good point. We discussed it already, and as far as I remember the
> conclusion was that it is a good idea. My issue with the XML implementation
> is its load time, and it requires a lot of memory for a big dictionary.
Let me have a closer look tonight.
Is the memory issue is caused by the fact the dictionaries (e.g.
POSDictionary) are using
the Java HashMap and String key/values?
Did you implement your own POSDictionary for your thesis?
The current dictionary package has an API to read and serialize a
dictionary from and to the
XML format. That could be changed to some binary based format which
could be much faster.
But as far as I understand is the main issue we have is the
representation of the dictionary in memory
and not the serialization of it.
Jörn
Re: English lemmatizer using wordnet
Posted by William Colen <wi...@gmail.com>.
Yes, good point. We discussed it already, and as far as I remember the
conclusion was that it is a good idea. My issue with the XML implementation
is its load time, and it requires a lot of memory for a big dictionary.
On Wed, Apr 10, 2013 at 10:54 AM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 04/10/2013 03:46 PM, William Colen wrote:
>
>> Here I store the dictionary as a text file and encoding the dictionary is
>> part of the build process, so it is easy to update the dictionary.
>> Maybe we should create an API that supports multiple implementations, a
>> default implementation can use JWNL, which is already available. We can
>> create other implementations in the sandbox as optional packages.
>>
>
> What about using our dictionary package for this? Couldn't we extend it
> somehow that different
> dictionary implementations are supported?
>
> Jörn
>
Re: English lemmatizer using wordnet
Posted by "Jim foo.bar" <ji...@gmail.com>.
On 10/04/13 14:54, Jörn Kottmann wrote:
> On 04/10/2013 03:46 PM, William Colen wrote:
>> Here I store the dictionary as a text file and encoding the
>> dictionary is
>> part of the build process, so it is easy to update the dictionary.
>> Maybe we should create an API that supports multiple implementations, a
>> default implementation can use JWNL, which is already available. We can
>> create other implementations in the sandbox as optional packages.
>
> What about using our dictionary package for this? Couldn't we extend
> it somehow that different
> dictionary implementations are supported?
>
> Jörn
Assuming that I understood Jorn correctly, I've already suggested this
6-7 months ago. Personally I don't see any reason for having he
DictionaryNameFinder accept a single Dictionary object. It should be
accepting any number of Dictionaries which should fall under the same
interface of course... That way different dictionary implementations
will be supported transparently by the same name-finder and with minimal
modifications. In fact, this is how my personal openNLP build works...
Jim
ps: apologies if I've completely misunderstood what is being discussed...
Re: English lemmatizer using wordnet
Posted by Jörn Kottmann <ko...@gmail.com>.
On 04/10/2013 03:46 PM, William Colen wrote:
> Here I store the dictionary as a text file and encoding the dictionary is
> part of the build process, so it is easy to update the dictionary.
> Maybe we should create an API that supports multiple implementations, a
> default implementation can use JWNL, which is already available. We can
> create other implementations in the sandbox as optional packages.
What about using our dictionary package for this? Couldn't we extend it
somehow that different
dictionary implementations are supported?
Jörn
Re: English lemmatizer using wordnet
Posted by William Colen <wi...@gmail.com>.
Yes. I see your point.
Here I store the dictionary as a text file and encoding the dictionary is
part of the build process, so it is easy to update the dictionary.
Maybe we should create an API that supports multiple implementations, a
default implementation can use JWNL, which is already available. We can
create other implementations in the sandbox as optional packages.
On Wed, Apr 10, 2013 at 10:14 AM, Rodrigo Agerri <ro...@ehu.es>wrote:
> Hello,
>
> I used morfologik and LanguageTool for grammar correction. It can be
> tricky to create and re-create the binary dictionaries, although it is
> true that once is created the speed is very good.
>
> In any case, that would also create a dependence on morfologik for
> creating and accessing the dictionaries.
>
> Cheers,
>
> Rodrigo
>
>
>
> On Wed, Apr 10, 2013 at 2:34 PM, William Colen <wi...@gmail.com>
> wrote:
> > Hi,
> >
> > +1 for a lemmatizer API
> >
> > For my Master's project I created a lemma dictionary, which keys were the
> > [token + POS tag] and the value one or more lemmas.
> >
> > To store and access the entries I used a very nice Java tool available
> > under BSD license that is part of the Morfologik tool (
> > http://sourceforge.net/projects/morfologik). This tool encodes the
> > dictionary in a finite-state automata, allowing a very efficient access
> and
> > a compact dictionary.
> > The tool also provide a efficient way of encoding and accessing lexical
> > dictionaries.
> >
> > The LanguageTools members wrote a tutorial on how to use Morfologik for
> > this: http://wiki.languagetool.org/developing-a-tagger-dictionary
> >
> >
> >
> > On Wed, Apr 10, 2013 at 9:02 AM, Rodrigo Agerri <rodrigo.agerri@ehu.es
> >wrote:
> >
> >> On Wed, Apr 10, 2013 at 1:00 PM, Jörn Kottmann <ko...@gmail.com>
> >> wrote:>
> >> >
> >> > +1, it would be nice to have control over the dictionary, maybe we can
> >> come
> >> > up with
> >> > a format to store it in. That will allow us to easily include it in
> our
> >> > models
> >> > as a resource for feature generation and eliminates the dependency on
> >> > external libraries.
> >>
> >> I do not know yet which dictionary format will be best, but I can try
> >> to come up with a proposal independent of WordNet or other third party
> >> resources, when I have it working, and then discuss it.
> >>
> >> >
> >> > +1
> >> >
> >> > We should define an interface which allows to use different
> >> implementations
> >> > like
> >> > we did for the other components.
> >>
> >> OK.
> >>
> >> Cheers,
> >>
> >> Rodrigo
> >>
>
Re: English lemmatizer using wordnet
Posted by Rodrigo Agerri <ro...@ehu.es>.
Hello,
I used morfologik and LanguageTool for grammar correction. It can be
tricky to create and re-create the binary dictionaries, although it is
true that once is created the speed is very good.
In any case, that would also create a dependence on morfologik for
creating and accessing the dictionaries.
Cheers,
Rodrigo
On Wed, Apr 10, 2013 at 2:34 PM, William Colen <wi...@gmail.com> wrote:
> Hi,
>
> +1 for a lemmatizer API
>
> For my Master's project I created a lemma dictionary, which keys were the
> [token + POS tag] and the value one or more lemmas.
>
> To store and access the entries I used a very nice Java tool available
> under BSD license that is part of the Morfologik tool (
> http://sourceforge.net/projects/morfologik). This tool encodes the
> dictionary in a finite-state automata, allowing a very efficient access and
> a compact dictionary.
> The tool also provide a efficient way of encoding and accessing lexical
> dictionaries.
>
> The LanguageTools members wrote a tutorial on how to use Morfologik for
> this: http://wiki.languagetool.org/developing-a-tagger-dictionary
>
>
>
> On Wed, Apr 10, 2013 at 9:02 AM, Rodrigo Agerri <ro...@ehu.es>wrote:
>
>> On Wed, Apr 10, 2013 at 1:00 PM, Jörn Kottmann <ko...@gmail.com>
>> wrote:>
>> >
>> > +1, it would be nice to have control over the dictionary, maybe we can
>> come
>> > up with
>> > a format to store it in. That will allow us to easily include it in our
>> > models
>> > as a resource for feature generation and eliminates the dependency on
>> > external libraries.
>>
>> I do not know yet which dictionary format will be best, but I can try
>> to come up with a proposal independent of WordNet or other third party
>> resources, when I have it working, and then discuss it.
>>
>> >
>> > +1
>> >
>> > We should define an interface which allows to use different
>> implementations
>> > like
>> > we did for the other components.
>>
>> OK.
>>
>> Cheers,
>>
>> Rodrigo
>>
Re: English lemmatizer using wordnet
Posted by William Colen <wi...@gmail.com>.
Hi,
+1 for a lemmatizer API
For my Master's project I created a lemma dictionary, which keys were the
[token + POS tag] and the value one or more lemmas.
To store and access the entries I used a very nice Java tool available
under BSD license that is part of the Morfologik tool (
http://sourceforge.net/projects/morfologik). This tool encodes the
dictionary in a finite-state automata, allowing a very efficient access and
a compact dictionary.
The tool also provide a efficient way of encoding and accessing lexical
dictionaries.
The LanguageTools members wrote a tutorial on how to use Morfologik for
this: http://wiki.languagetool.org/developing-a-tagger-dictionary
On Wed, Apr 10, 2013 at 9:02 AM, Rodrigo Agerri <ro...@ehu.es>wrote:
> On Wed, Apr 10, 2013 at 1:00 PM, Jörn Kottmann <ko...@gmail.com>
> wrote:>
> >
> > +1, it would be nice to have control over the dictionary, maybe we can
> come
> > up with
> > a format to store it in. That will allow us to easily include it in our
> > models
> > as a resource for feature generation and eliminates the dependency on
> > external libraries.
>
> I do not know yet which dictionary format will be best, but I can try
> to come up with a proposal independent of WordNet or other third party
> resources, when I have it working, and then discuss it.
>
> >
> > +1
> >
> > We should define an interface which allows to use different
> implementations
> > like
> > we did for the other components.
>
> OK.
>
> Cheers,
>
> Rodrigo
>
Re: English lemmatizer using wordnet
Posted by Jörn Kottmann <ko...@gmail.com>.
Hello,
thanks for the detailed view about the different implementations,
if possible it would probably make sense to have all three options?
Option 3 is especially nice for people concerned about throughput,
where option 2 seems to be really nice for people who want to try
things which work out of the box and are easy to get to work.
Option 1 would be nice to have for the coref component.
I suggest we define an interface for the Lemmatizer Dictionary and then
provide different implementations for it, similar as it can be done for
the POSTagger,
have a look at TagDictionary. The dictionary package contains util
classes which can
be used to serialize/deserialize to our xml format.
Jörn
On 04/15/2013 06:15 PM, Rodrigo Agerri wrote:
> Hello again,
>
> I have now working three different implementations for lemmatization
> using a dictionary for English and Spanish:
>
> 1. As mentioned in first email, using Princeton WordNet and JWNL.
> 2. Loading dictionary from plain text and perform HashMap .get on the
> dictionary to obtain the lemmas
> 3. Loading Morfologik dictionary and perform HashMap .get on a mini
> HashMap dictionary created from the Morfologik dictionary lookup.
>
> In the project there seem to be several Dictionary implementations. I
> can try to use any of those (in particular, the POSDictionary seems
> the most relevant for the task at hand), but in any case I think that
> for lemmatization the implementation can be very simple. We only need
> a plain text tab separated Word Postag Lemma dictionary and a
> Dictionary interface that defines a lemmatize(String word, String
> postag) method.
>
> Then each of the approaches above implement this interface to create 3
> different lemmatizers.
>
> 1. JWNLemmatizer constructs a JWNLDictionary (as in Coref OpenNLP
> package) and then uses JWNL API to obtain the lemmas.
> 2. A SimpleLemmatizer using a plain text tab separated dictionary in
> the form Word Postag Lemma that is loaded into a
> HashMap<List<String>,String>dictionary. If the dictionary is large,
> this is where the performance in terms of memory and speed seems to be
> most affected. The lemmatization performs .get operations on the
> HashMap to obtain the lemma.
> 3. A MorfologikLemmatizer using a binary dictionary created from the
> plain text tab separated dictionary of previous point. Morfologik
> DictionaryLookup is used to obtain the associated lemmas and postags
> for each word. From this a HashMap dictionary is created and .get
> obtains the lemma for each word,postag pair. Loading the Morfologik
> dictionary is very fast and it is memory cheap.
>
> The dictionaries used in 2. and 3. come from those distributed by
> LanguageTool. Using approach 2. is dependent free and 3. requires
> Morfologik stemming to create and access the dictionary. Furthermore,
> the LanguageTool dictionaries are more complete than WordNet and
> easier to maintain for lemmatization.
>
> I have measured the execution time and looked at memory resources
> while analyzing a standard article from Guardian for English and
> another one from a Spanish newspaper. You can look at the numbers
> below. Using morfologik dictionary is the fastest and memory cheap
> although in a good machine the differences in speed get quite smaller.
>
> What do you think?
>
> Cheers,
>
> Rodrigo
>
> English POS and lemmatization:
>
> - English dictionary 350K entries:
> + Plain text: 7.9MB
> + Morfologik: 1.2MB
>
> $ wc english.txt
>
> 33 684 4225
>
> Spanish POS and lemmatization:
>
> - Dictionary: 650K entries
> + Plain text: 19M
> + Morfologik: 600K
>
> $ wc spanish.txt
> 52 947 5857
>
> DELL Optiplex 790, Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 16GB RAM
> -------------------------------------------------------------------------------------------------------------
>
> ENGLISH
> --------------
>
> [JWNL and WordNet]
>
> real 0m1.652s
> user 0m2.063s
> sys 0m0.215s
>
> [plain text hashmap lookup]
>
> real 0m1.237s
> user 0m2.113s
> sys 0m0.122s
>
> RAM 250MB ~
>
> [morfologik stemming]
>
> real 0m1.134s
> user 0m1.869s
> sys 0m0.090s
>
> RAM 40MB ~
>
> SPANISH
> --------------
>
> [plain text hashmap lookup]
>
> real 0m2.764s
> user 0m9.654s
> sys 0m1.022s
>
> 400 MB RAM
>
> [morfologik stemming]
>
> real 0m1.209s
> user 0m2.043s
> sys 0m0.093s
>
> 40 MB RAM
>
> NETBOOK Acer Aspire One AMD Dual-Core C60 APU, 4GB RAM
> -----------------------------------------------------------------------------------------------
>
> ENGLISH
> --------------
>
> [using JWNL and WordNet]
>
> real 0m9.410s
> user 0m12.105s
> sys 0m1.004s
>
> [using plain text and hashmap lookup]
>
> real 0m10.273s
> user 0m13.857s
> sys 0m0.540s
>
> [using morfologik stemming]
>
> real 0m7.834s
> user 0m10.977s
> sys 0m0.556s
>
> SPANISH
> --------------
>
> [plain text hashmap lookup]
>
> real 0m17.233s
> user 0m23.597s
> sys 0m1.320s
>
> RAM 400MB ~
>
> [morfologik stemming]
>
> real 0m8.107s
> user 0m12.181s
> sys 0m0.408s
>
> RAM 40MB ~
>
> On Fri, Apr 12, 2013 at 4:13 PM, Rodrigo Agerri <ro...@ehu.es> wrote:
>> Sure,
>>
>> I am looking at it.
>>
>> Cheers,
>>
>> Rodrigo
>>
>> On Fri, Apr 12, 2013 at 3:36 PM, Aliaksandr Autayeu
>> <al...@autayeu.com> wrote:
>>>> I do not know yet which dictionary format will be best, but I can try
>>>> to come up with a proposal independent of WordNet or other third party
>>>> resources, when I have it working, and then discuss it.
>>>>
>>> Rodrigo, it would be nice if you write as soon as you have come up with
>>> data structures, before implementation. This will allow to take more
>>> languages into account.
Re: English lemmatizer using wordnet
Posted by Rodrigo Agerri <ag...@gmail.com>.
Hello again,
I have now working three different implementations for lemmatization
using a dictionary for English and Spanish:
1. As mentioned in first email, using Princeton WordNet and JWNL.
2. Loading dictionary from plain text and perform HashMap .get on the
dictionary to obtain the lemmas
3. Loading Morfologik dictionary and perform HashMap .get on a mini
HashMap dictionary created from the Morfologik dictionary lookup.
In the project there seem to be several Dictionary implementations. I
can try to use any of those (in particular, the POSDictionary seems
the most relevant for the task at hand), but in any case I think that
for lemmatization the implementation can be very simple. We only need
a plain text tab separated Word Postag Lemma dictionary and a
Dictionary interface that defines a lemmatize(String word, String
postag) method.
Then each of the approaches above implement this interface to create 3
different lemmatizers.
1. JWNLemmatizer constructs a JWNLDictionary (as in Coref OpenNLP
package) and then uses JWNL API to obtain the lemmas.
2. A SimpleLemmatizer using a plain text tab separated dictionary in
the form Word Postag Lemma that is loaded into a
HashMap<List<String>,String>dictionary. If the dictionary is large,
this is where the performance in terms of memory and speed seems to be
most affected. The lemmatization performs .get operations on the
HashMap to obtain the lemma.
3. A MorfologikLemmatizer using a binary dictionary created from the
plain text tab separated dictionary of previous point. Morfologik
DictionaryLookup is used to obtain the associated lemmas and postags
for each word. From this a HashMap dictionary is created and .get
obtains the lemma for each word,postag pair. Loading the Morfologik
dictionary is very fast and it is memory cheap.
The dictionaries used in 2. and 3. come from those distributed by
LanguageTool. Using approach 2. is dependent free and 3. requires
Morfologik stemming to create and access the dictionary. Furthermore,
the LanguageTool dictionaries are more complete than WordNet and
easier to maintain for lemmatization.
I have measured the execution time and looked at memory resources
while analyzing a standard article from Guardian for English and
another one from a Spanish newspaper. You can look at the numbers
below. Using morfologik dictionary is the fastest and memory cheap
although in a good machine the differences in speed get quite smaller.
What do you think?
Cheers,
Rodrigo
English POS and lemmatization:
- English dictionary 350K entries:
+ Plain text: 7.9MB
+ Morfologik: 1.2MB
$ wc english.txt
33 684 4225
Spanish POS and lemmatization:
- Dictionary: 650K entries
+ Plain text: 19M
+ Morfologik: 600K
$ wc spanish.txt
52 947 5857
DELL Optiplex 790, Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 16GB RAM
-------------------------------------------------------------------------------------------------------------
ENGLISH
--------------
[JWNL and WordNet]
real 0m1.652s
user 0m2.063s
sys 0m0.215s
[plain text hashmap lookup]
real 0m1.237s
user 0m2.113s
sys 0m0.122s
RAM 250MB ~
[morfologik stemming]
real 0m1.134s
user 0m1.869s
sys 0m0.090s
RAM 40MB ~
SPANISH
--------------
[plain text hashmap lookup]
real 0m2.764s
user 0m9.654s
sys 0m1.022s
400 MB RAM
[morfologik stemming]
real 0m1.209s
user 0m2.043s
sys 0m0.093s
40 MB RAM
NETBOOK Acer Aspire One AMD Dual-Core C60 APU, 4GB RAM
-----------------------------------------------------------------------------------------------
ENGLISH
--------------
[using JWNL and WordNet]
real 0m9.410s
user 0m12.105s
sys 0m1.004s
[using plain text and hashmap lookup]
real 0m10.273s
user 0m13.857s
sys 0m0.540s
[using morfologik stemming]
real 0m7.834s
user 0m10.977s
sys 0m0.556s
SPANISH
--------------
[plain text hashmap lookup]
real 0m17.233s
user 0m23.597s
sys 0m1.320s
RAM 400MB ~
[morfologik stemming]
real 0m8.107s
user 0m12.181s
sys 0m0.408s
RAM 40MB ~
On Fri, Apr 12, 2013 at 4:13 PM, Rodrigo Agerri <ro...@ehu.es> wrote:
> Sure,
>
> I am looking at it.
>
> Cheers,
>
> Rodrigo
>
> On Fri, Apr 12, 2013 at 3:36 PM, Aliaksandr Autayeu
> <al...@autayeu.com> wrote:
>>>
>>> I do not know yet which dictionary format will be best, but I can try
>>> to come up with a proposal independent of WordNet or other third party
>>> resources, when I have it working, and then discuss it.
>>>
>> Rodrigo, it would be nice if you write as soon as you have come up with
>> data structures, before implementation. This will allow to take more
>> languages into account.
Re: English lemmatizer using wordnet
Posted by Rodrigo Agerri <ro...@ehu.es>.
Sure,
I am looking at it.
Cheers,
Rodrigo
On Fri, Apr 12, 2013 at 3:36 PM, Aliaksandr Autayeu
<al...@autayeu.com> wrote:
>>
>> I do not know yet which dictionary format will be best, but I can try
>> to come up with a proposal independent of WordNet or other third party
>> resources, when I have it working, and then discuss it.
>>
> Rodrigo, it would be nice if you write as soon as you have come up with
> data structures, before implementation. This will allow to take more
> languages into account.
Re: English lemmatizer using wordnet
Posted by Aliaksandr Autayeu <al...@autayeu.com>.
>
> I do not know yet which dictionary format will be best, but I can try
> to come up with a proposal independent of WordNet or other third party
> resources, when I have it working, and then discuss it.
>
Rodrigo, it would be nice if you write as soon as you have come up with
data structures, before implementation. This will allow to take more
languages into account.
Re: English lemmatizer using wordnet
Posted by Rodrigo Agerri <ro...@ehu.es>.
On Wed, Apr 10, 2013 at 1:00 PM, Jörn Kottmann <ko...@gmail.com> wrote:>
>
> +1, it would be nice to have control over the dictionary, maybe we can come
> up with
> a format to store it in. That will allow us to easily include it in our
> models
> as a resource for feature generation and eliminates the dependency on
> external libraries.
I do not know yet which dictionary format will be best, but I can try
to come up with a proposal independent of WordNet or other third party
resources, when I have it working, and then discuss it.
>
> +1
>
> We should define an interface which allows to use different implementations
> like
> we did for the other components.
OK.
Cheers,
Rodrigo
Re: English lemmatizer using wordnet
Posted by Aliaksandr Autayeu <al...@autayeu.com>.
> Spanish pos + lemmatizer using this approach.
>>
>
> +1, it would be nice to have control over the dictionary, maybe we can
> come up with
> a format to store it in. That will allow us to easily include it in our
> models
> as a resource for feature generation and eliminates the dependency on
> external libraries.
That would be great! The format should then take into account
morphological features.
> Of course, another method would be to re-implement John Carroll and
>> colleagues' finite-state approach for English (and similar rule-based
>> approaches for other languages) which removes the dependence on a
>> dictionary. I will be exploring this further on.
>>
>
> +1
>
> We should define an interface which allows to use different
> implementations like
> we did for the other components.
+1. It seems that we have european languages represented here. Do we have
anybody from east? chinese? Would be nice to check them too.
Re: English lemmatizer using wordnet
Posted by Jörn Kottmann <ko...@gmail.com>.
On 04/10/2013 12:06 PM, Rodrigo Agerri wrote:
> The flexibility of the design would require to isolate the functions
> that access each dictionary for each language (including tagset
> mappings) from the lemmatization functionality itself. For example,
> JWNL only works for English original WordNet. I am working on a
> Spanish pos + lemmatizer using this approach.
+1, it would be nice to have control over the dictionary, maybe we can
come up with
a format to store it in. That will allow us to easily include it in our
models
as a resource for feature generation and eliminates the dependency on
external libraries.
> Of course, another method would be to re-implement John Carroll and
> colleagues' finite-state approach for English (and similar rule-based
> approaches for other languages) which removes the dependence on a
> dictionary. I will be exploring this further on.
+1
We should define an interface which allows to use different
implementations like
we did for the other components.
Jörn
Re: English lemmatizer using wordnet
Posted by Rodrigo Agerri <ro...@ehu.es>.
On Wed, Apr 10, 2013 at 11:43 AM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 04/10/2013 11:27 AM, Rodrigo Agerri wrote:
>>
>> I have put together a very simple English lemmatizer using JWNL as
>> part of a POS module based on opennlp API. The lemmatizer uses the
>> constructor of JWNLDictionary in opennlp coref package.
>>
>> If there is interest in explicitly providing an English lemmatizer in
>> the project I would not mind providing the code.
>
>
> Yes, it would be nice to have a lemmatizer component in OpenNLP, the design
> should be flexible enough so we can extend it with lemmatizers for other
> languages than English later.
>
> As far as I remember is the wordnet approach is a dictionary lookup with
> the token and its pos tag, right?
>
Yes, postag and token lookup.
The flexibility of the design would require to isolate the functions
that access each dictionary for each language (including tagset
mappings) from the lemmatization functionality itself. For example,
JWNL only works for English original WordNet. I am working on a
Spanish pos + lemmatizer using this approach.
Of course, another method would be to re-implement John Carroll and
colleagues' finite-state approach for English (and similar rule-based
approaches for other languages) which removes the dependence on a
dictionary. I will be exploring this further on.
Cheers,
Rodrigo
Re: English lemmatizer using wordnet
Posted by Jörn Kottmann <ko...@gmail.com>.
On 04/10/2013 11:27 AM, Rodrigo Agerri wrote:
> I have put together a very simple English lemmatizer using JWNL as
> part of a POS module based on opennlp API. The lemmatizer uses the
> constructor of JWNLDictionary in opennlp coref package.
>
> If there is interest in explicitly providing an English lemmatizer in
> the project I would not mind providing the code.
Yes, it would be nice to have a lemmatizer component in OpenNLP, the design
should be flexible enough so we can extend it with lemmatizers for other
languages than English later.
As far as I remember is the wordnet approach is a dictionary lookup with
the token and its pos tag, right?
Jörn