You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by "Jim - FooBar();" <ji...@gmail.com> on 2012/02/06 18:15:05 UTC

Problem with openNLP Name Finder API....

Hello openNLP users/developers,

I really need your help with openNLP NameFinder...Let me explain:

I am writing a drug-entity recogniser (NER for drugs) and i 'm using the 
openNLP API from Clojure. I don't have annotated text but i do an 
up-to-date dictionary of drugs (drugbank.xml). The dictionary includes 
all sorts of information so it needs a bit of preprocessing to extract 
the names and synonyms. Anyway my problem is twofold:

As i said i don't have annotated text, but i thought maybe i can make 
one. You see, i do have 383 pharmacology papers in raw text so i thought 
why not use the names from the dictionary and build regex patterns to 
replace all occurrences of that entry with the appropriate "<START:drug> 
/drug-name/ <END>" annotation tag. Now, you may be wondering at this 
point why on earth am i  not using all the names i extracted from 
drugbank.xml to build a proper openNLP dictionary to do lookup...Well, 
apart from not being what i'm trying to do here, i've already tried 
doing that but i got poor results  simply because words like "Folic 
acid" are being tokenized as 2 tokens rather than 1 and thus, the entry 
"Folic acid" in the dictionary matches no token! Even if it worked 
thoughi would still have to train a maxent model to recognise drugs that 
may not exist in the dictionary (brand new for instance). My initial 
approach was a bit fiddly but it sort of paid of at the end. I now have 
a small program that expects some text and, for each entry in the 
dictionary (6707 in total) it finds and replaces any occurrences of that 
entry in the text with the expected openNLP format for training. Here is 
where the 1st problem happens. I can tweak my regex pattern to add 
spaces to the entity tag or not, with the following results :

<START:drug> drug-name <END>  (with spaces inside)

causes problems for me because i get nested tags. To understand why 
think about the words "Folic acid" for example. Lets assume that folic 
acid is entry 3 in the dictionary and that entry 115 is "acid". First 
time round it will produce <START:drug> Folic acid <END> but when it 
processes "acid" it will match the word acid already tagged with "Folic 
acid". You can see where this is going. If you happen to have a complex 
compound name and after a while a slightly less complex compound name 
(maybe a word shorter or something), and then a smaller one, they can 
easily start to nest, especially when dealing with drug names. Now the 
easy and straightforward solution to that is to NOT add spaces in the 
tag like this :

<START:drug>Folic acid<END>   (this will NOT match "acid" in later parsing)

I honestly wasn't expecting that to make any difference to the training 
process but as it turns out it breaks it completely. Exceptions 
everywhere before it even starts!!! Could please someone explain what 
happens with those spaces around the entity name? How on earth can they 
make any difference? I can solve my problem by doing negative lookbehind 
assertion in my regex but that slows things down quite a bit! remember 
i'm dealing with 6707 entries, times 383 papers. Clojure's lazy attitude 
and loop/recur structure sure help a lot...

Ok now on to the 2nd problem...

Even when i manually sort all the nested tags and i  finally train a 
maxent model on the newly automatically annotated papers, i still get 
very poor results (poorer than the dictionary) and i think i can see why 
but that contradicts the openNLP documentation and all the examples i 
've seen so far.

On the openNLP tutorial it seems perfectly normal to have 2 words inside 
a tag like:

<START:name> Pierre Vinken <END>

but when the time comes to use the name-finder model you just trained 
everything has to be tokenized again. Therefore, "Pierre Vinken" becomes 
2 tokens and cannot be recognised. Sometimes just the one token may be 
recognised as an entity but without the rest of the name is not only 
meaningless but could be misleading. Again think about Folic acid... 
Neither "folic" nor "acid" are drugs...even if the name-finder 
recognises "folic", it's of no use!!! In the same way that "Vinken" is 
not necessarily a name...

Anyway sorry for the massive e-mail but i'm really struggling!


Please help me, i'm at a dead-end at the moment! i've tried literally 
everything...am i missing anything important?
Thanks in advance...keep up the good work!

JIm



Re: Problem with openNLP Name Finder API....

Posted by "Jim - FooBar();" <ji...@gmail.com>.
Also here is what the name finder found after having been trained on 
20000 lines  of text of the format i just sent you...The file attached 
contains the results after running the trained model on the same 200000 
lines (without the sgml tags of course)  - i do know that this is 
discouraged!

See attachment....

On 08/02/12 17:09, Joern Kottmann wrote:
> On Wed, Feb 8, 2012 at 5:56 PM, Jim - FooBar();<ji...@gmail.com>wrote:
>
>> aaa ok i see what you mean...but then again if it recognised it as a mere
>> token it would not throw "IncompatibleFormat" exceptions but rather skip it
>> as a token that is not of interest wouldn't it? I don't have any patches to
>> send you, i just think that not including spaces in the sgml tag is a more
>> wise approach...Unless of course you're extracting the sgml tags via
>> regex...The truth is i've not looked at the source but i would expect you
>> to use some sort of xml-ish means to extract the sgml tags. If your parser
>> is using regex then i'm sure you have your reasons for including the
>> spaces. But anyway, this is a very small problem for me cos i can indeed
>> sort it manually...My big problem still remains!!!
>>
> The code splits the input string by line and then by white space. Then the
> individual parts either
> match our start and end tags or not.
>
>
>
>> Anyway I'll stop bugging you...the fact that you tried to help means a lot
>> and certainly if i sort everything out i'll post what the problem was for
>> future users...
>>
>>
> We are also interested why it does not work for you, we usually use this
> kind of experience to
> improve OpenNLP.
>
> Would it be possible for you to show us a sample of your training data?
> Maybe one paper.
>
> Jörn
>


Re: Problem with openNLP Name Finder API....

Posted by "Jim - FooBar();" <ji...@gmail.com>.
> - Multiple sentences in a line 
oops!!! i sent a sample from the wrong training data!!! I tried so many 
combinations that i've ended up with several training data files...i do 
have one sentence per line in the correct file. In fact i've got a 
function that does it for me.

> -You data is not tokenized 

what exactly do you mean? the training data should be tokenized? I'm not 
sure i follow...
its after the training that i need to tokenize in order to do NER isn't it?

> - Adaptive data is not cleared 

Is that necessary since i'm treating the merged document as a single 
document? I mean i can do it but will that make any noticeable 
difference? I can easily append a newline when cat-ing the papers...

> Do you use our command line tools for training? 

No, i'm writing my own application that uses the API...

Jim


On 08/02/12 19:09, Jörn Kottmann wrote:
> I see the following issues:
> - Multiple sentences in a line
> - You data is not tokenized
> - Adaptive data is not cleared
>
> You can use our sentence detector to split
> your paragraphs. If you know your document
> boundaries you should write an empty line to
> that file to clear the adaptive data. If you cannot
> do that write an empty line after every sentence.
>
> Do you use our command line tools for training?
>
> Jörn
>
> On 02/08/2012 06:46 PM, Jim - FooBar(); wrote:
>>> Would it be possible for you to show us a sample of your training data?
>>> Maybe one paper.
>>
>> Absolutely here you go....a sample has been attached...Let me know if 
>> you want more but i can assure you that since the sgml tags are 
>> generated automatically (with regex replacement) they are all of the 
>> same format...
>>
>> Jim
>>
>> p.s: fire up your favourite editor press ctrl+f and search for 
>> "<START" just to see locate them easily!
>>
>>
>> On 08/02/12 17:09, Joern Kottmann wrote:
>>> On Wed, Feb 8, 2012 at 5:56 PM, Jim - 
>>> FooBar();<ji...@gmail.com>wrote:
>>>
>>>> aaa ok i see what you mean...but then again if it recognised it as 
>>>> a mere
>>>> token it would not throw "IncompatibleFormat" exceptions but rather 
>>>> skip it
>>>> as a token that is not of interest wouldn't it? I don't have any 
>>>> patches to
>>>> send you, i just think that not including spaces in the sgml tag is 
>>>> a more
>>>> wise approach...Unless of course you're extracting the sgml tags via
>>>> regex...The truth is i've not looked at the source but i would 
>>>> expect you
>>>> to use some sort of xml-ish means to extract the sgml tags. If your 
>>>> parser
>>>> is using regex then i'm sure you have your reasons for including the
>>>> spaces. But anyway, this is a very small problem for me cos i can 
>>>> indeed
>>>> sort it manually...My big problem still remains!!!
>>>>
>>> The code splits the input string by line and then by white space. 
>>> Then the
>>> individual parts either
>>> match our start and end tags or not.
>>>
>>>
>>>
>>>> Anyway I'll stop bugging you...the fact that you tried to help 
>>>> means a lot
>>>> and certainly if i sort everything out i'll post what the problem 
>>>> was for
>>>> future users...
>>>>
>>>>
>>> We are also interested why it does not work for you, we usually use 
>>> this
>>> kind of experience to
>>> improve OpenNLP.
>>>
>>> Would it be possible for you to show us a sample of your training data?
>>> Maybe one paper.
>>>
>>> Jörn
>>>
>>
>


Re: Problem with openNLP Name Finder API....

Posted by "Jim - FooBar();" <ji...@gmail.com>.
Ok so anyone who wants to reproduce my problem will need 2 files.

 1. A dummy training file (a single annotated pharmacology paper)
 2. The same paper but raw. In other words the file before being annotated.
 3. As an extra i'm including the model i just trained on this paper.

What i'm getting back from the trained model when i feed it with the 
tokenised sentences of the original raw text is:
/
("hydroxysaclofen") ("muscimol" "hydroxysaclofen") ("Hydroxysaclofen")/

This is really  not good!!! Only 3 sentences mentioned drugs?
No multi-word drugs even though they exist in training text.

_File description:_

  * PROPER-TRAIN-DATA.txt ---> as the name implies...one sentence per
    line, spaces in sgml tags, empty line at the end
  * my-2NER-short.bin  ----> the model i just trained using
    PROPER-TRAIN-DATA.txt
  * 48147.nex.txt  ----> the original raw paper.

It takes  less than 5 seconds to train from such a small file and even 
less to run the trained NER so try it out and let me know how get on...


Regards,
Dimitris


On 08/02/12 19:09, Jörn Kottmann wrote:
> I see the following issues:
> - Multiple sentences in a line
> - You data is not tokenized
> - Adaptive data is not cleared
>
> You can use our sentence detector to split
> your paragraphs. If you know your document
> boundaries you should write an empty line to
> that file to clear the adaptive data. If you cannot
> do that write an empty line after every sentence.
>
> Do you use our command line tools for training?
>
> Jörn
>
> On 02/08/2012 06:46 PM, Jim - FooBar(); wrote:
>>> Would it be possible for you to show us a sample of your training data?
>>> Maybe one paper.
>>
>> Absolutely here you go....a sample has been attached...Let me know if 
>> you want more but i can assure you that since the sgml tags are 
>> generated automatically (with regex replacement) they are all of the 
>> same format...
>>
>> Jim
>>
>> p.s: fire up your favourite editor press ctrl+f and search for 
>> "<START" just to see locate them easily!
>>
>>
>> On 08/02/12 17:09, Joern Kottmann wrote:
>>> On Wed, Feb 8, 2012 at 5:56 PM, Jim - 
>>> FooBar();<ji...@gmail.com>wrote:
>>>
>>>> aaa ok i see what you mean...but then again if it recognised it as 
>>>> a mere
>>>> token it would not throw "IncompatibleFormat" exceptions but rather 
>>>> skip it
>>>> as a token that is not of interest wouldn't it? I don't have any 
>>>> patches to
>>>> send you, i just think that not including spaces in the sgml tag is 
>>>> a more
>>>> wise approach...Unless of course you're extracting the sgml tags via
>>>> regex...The truth is i've not looked at the source but i would 
>>>> expect you
>>>> to use some sort of xml-ish means to extract the sgml tags. If your 
>>>> parser
>>>> is using regex then i'm sure you have your reasons for including the
>>>> spaces. But anyway, this is a very small problem for me cos i can 
>>>> indeed
>>>> sort it manually...My big problem still remains!!!
>>>>
>>> The code splits the input string by line and then by white space. 
>>> Then the
>>> individual parts either
>>> match our start and end tags or not.
>>>
>>>
>>>
>>>> Anyway I'll stop bugging you...the fact that you tried to help 
>>>> means a lot
>>>> and certainly if i sort everything out i'll post what the problem 
>>>> was for
>>>> future users...
>>>>
>>>>
>>> We are also interested why it does not work for you, we usually use 
>>> this
>>> kind of experience to
>>> improve OpenNLP.
>>>
>>> Would it be possible for you to show us a sample of your training data?
>>> Maybe one paper.
>>>
>>> Jörn
>>>
>>
>


Re: Problem with openNLP Name Finder API....

Posted by Jörn Kottmann <ko...@gmail.com>.
I see the following issues:
- Multiple sentences in a line
- You data is not tokenized
- Adaptive data is not cleared

You can use our sentence detector to split
your paragraphs. If you know your document
boundaries you should write an empty line to
that file to clear the adaptive data. If you cannot
do that write an empty line after every sentence.

Do you use our command line tools for training?

Jörn

On 02/08/2012 06:46 PM, Jim - FooBar(); wrote:
>> Would it be possible for you to show us a sample of your training data?
>> Maybe one paper.
>
> Absolutely here you go....a sample has been attached...Let me know if 
> you want more but i can assure you that since the sgml tags are 
> generated automatically (with regex replacement) they are all of the 
> same format...
>
> Jim
>
> p.s: fire up your favourite editor press ctrl+f and search for 
> "<START" just to see locate them easily!
>
>
> On 08/02/12 17:09, Joern Kottmann wrote:
>> On Wed, Feb 8, 2012 at 5:56 PM, Jim - 
>> FooBar();<ji...@gmail.com>wrote:
>>
>>> aaa ok i see what you mean...but then again if it recognised it as a 
>>> mere
>>> token it would not throw "IncompatibleFormat" exceptions but rather 
>>> skip it
>>> as a token that is not of interest wouldn't it? I don't have any 
>>> patches to
>>> send you, i just think that not including spaces in the sgml tag is 
>>> a more
>>> wise approach...Unless of course you're extracting the sgml tags via
>>> regex...The truth is i've not looked at the source but i would 
>>> expect you
>>> to use some sort of xml-ish means to extract the sgml tags. If your 
>>> parser
>>> is using regex then i'm sure you have your reasons for including the
>>> spaces. But anyway, this is a very small problem for me cos i can 
>>> indeed
>>> sort it manually...My big problem still remains!!!
>>>
>> The code splits the input string by line and then by white space. 
>> Then the
>> individual parts either
>> match our start and end tags or not.
>>
>>
>>
>>> Anyway I'll stop bugging you...the fact that you tried to help means 
>>> a lot
>>> and certainly if i sort everything out i'll post what the problem 
>>> was for
>>> future users...
>>>
>>>
>> We are also interested why it does not work for you, we usually use this
>> kind of experience to
>> improve OpenNLP.
>>
>> Would it be possible for you to show us a sample of your training data?
>> Maybe one paper.
>>
>> Jörn
>>
>


Re: Problem with openNLP Name Finder API....

Posted by "Jim - FooBar();" <ji...@gmail.com>.
> Would it be possible for you to show us a sample of your training data?
> Maybe one paper.

Absolutely here you go....a sample has been attached...Let me know if 
you want more but i can assure you that since the sgml tags are 
generated automatically (with regex replacement) they are all of the 
same format...

Jim

p.s: fire up your favourite editor press ctrl+f and search for "<START" 
just to see locate them easily!


On 08/02/12 17:09, Joern Kottmann wrote:
> On Wed, Feb 8, 2012 at 5:56 PM, Jim - FooBar();<ji...@gmail.com>wrote:
>
>> aaa ok i see what you mean...but then again if it recognised it as a mere
>> token it would not throw "IncompatibleFormat" exceptions but rather skip it
>> as a token that is not of interest wouldn't it? I don't have any patches to
>> send you, i just think that not including spaces in the sgml tag is a more
>> wise approach...Unless of course you're extracting the sgml tags via
>> regex...The truth is i've not looked at the source but i would expect you
>> to use some sort of xml-ish means to extract the sgml tags. If your parser
>> is using regex then i'm sure you have your reasons for including the
>> spaces. But anyway, this is a very small problem for me cos i can indeed
>> sort it manually...My big problem still remains!!!
>>
> The code splits the input string by line and then by white space. Then the
> individual parts either
> match our start and end tags or not.
>
>
>
>> Anyway I'll stop bugging you...the fact that you tried to help means a lot
>> and certainly if i sort everything out i'll post what the problem was for
>> future users...
>>
>>
> We are also interested why it does not work for you, we usually use this
> kind of experience to
> improve OpenNLP.
>
> Would it be possible for you to show us a sample of your training data?
> Maybe one paper.
>
> Jörn
>


Re: Problem with openNLP Name Finder API....

Posted by Joern Kottmann <ko...@gmail.com>.
On Wed, Feb 8, 2012 at 5:56 PM, Jim - FooBar(); <ji...@gmail.com>wrote:

> aaa ok i see what you mean...but then again if it recognised it as a mere
> token it would not throw "IncompatibleFormat" exceptions but rather skip it
> as a token that is not of interest wouldn't it? I don't have any patches to
> send you, i just think that not including spaces in the sgml tag is a more
> wise approach...Unless of course you're extracting the sgml tags via
> regex...The truth is i've not looked at the source but i would expect you
> to use some sort of xml-ish means to extract the sgml tags. If your parser
> is using regex then i'm sure you have your reasons for including the
> spaces. But anyway, this is a very small problem for me cos i can indeed
> sort it manually...My big problem still remains!!!
>

The code splits the input string by line and then by white space. Then the
individual parts either
match our start and end tags or not.



> Anyway I'll stop bugging you...the fact that you tried to help means a lot
> and certainly if i sort everything out i'll post what the problem was for
> future users...
>
>
We are also interested why it does not work for you, we usually use this
kind of experience to
improve OpenNLP.

Would it be possible for you to show us a sample of your training data?
Maybe one paper.

Jörn

Re: Problem with openNLP Name Finder API....

Posted by "Jim - FooBar();" <ji...@gmail.com>.
aaa ok i see what you mean...but then again if it recognised it as a 
mere token it would not throw "IncompatibleFormat" exceptions but rather 
skip it as a token that is not of interest wouldn't it? I don't have any 
patches to send you, i just think that not including spaces in the sgml 
tag is a more wise approach...Unless of course you're extracting the 
sgml tags via regex...The truth is i've not looked at the source but i 
would expect you to use some sort of xml-ish means to extract the sgml 
tags. If your parser is using regex then i'm sure you have your reasons 
for including the spaces. But anyway, this is a very small problem for 
me cos i can indeed sort it manually...My big problem still remains!!!

Anyway I'll stop bugging you...the fact that you tried to help means a 
lot and certainly if i sort everything out i'll post what the problem 
was for future users...

Cheers,
Jim


On 08/02/12 16:41, Joern Kottmann wrote:
> The parsing code for the format expects white space tokenized text. The
> <START>  and<END>  tags are handled different and are not
> a token in this sense, but when you directly attach it to a word like you
> did. acid<START>  then our parsing code just recognize it as a token
> and not the tag to mark entity boundaries.


Re: Problem with openNLP Name Finder API....

Posted by Joern Kottmann <ko...@gmail.com>.
On Wed, Feb 8, 2012 at 4:45 PM, Jim - FooBar(); <ji...@gmail.com>wrote:

> That is how we defined the training format. The<START>  and<END>  tag MUST
>> be white space separated,
>> otherwise it will be recognized as a token.
>>
> I think you meant it *WON'T* be recognised as a token...a mere typo or
> have i misunderstood?
> I mean its pretty obvious that spaces are necessary from all the
> exceptions thrown while counting the events! I'm just wondering why have
> you chosen to do that in the first place?No spaces makes a lot more sense
> to me regardless of the problem i am having with regex replacement...
>

The parsing code for the format expects white space tokenized text. The
<START> and <END> tags are handled different and are not
a token in this sense, but when you directly attach it to a word like you
did. acid<START> then our parsing code just recognize it as a token
and not the tag to mark entity boundaries.



>
>

>  Have a look at our documentation. The NER code you see there is correct.
>> If you have problems to detect multi-token names I suspect that something
>> with your training data is wrong.
>>
> I've spent the last 2 weeks reading the docs and i 've practically read
> all the external sources on stackoverflow and otehr sources. However
> everyone's demo is about the same thing shown in the docs which is the
> person name finder (which happens to include a multi-word token - Pierre
> Vinken)! As far as the training data is concerned i've systematically
> checked it!!! I had to because as i said in the beginning i had nested tags
> which had to be sorted manually...I spent a whole day doing that but at
> least i was thinking "Finally i am so close to training...!!!". On top of
> that if there was something wrong with my training data i would expect
> exceptions again but i 'm not getting any since i sorted out the nested
> tags !!!



You are welcome to send us patches for problems in our training data
parsing code, to my knowledge it just works as long as the
data is in the correct format. Format violations might he hard to find,
that is true, we already improved it a bit (I think that is already in
1.5.2).



>  The Name Finder takes a tokenized sentence at a time. After you are done
>> with a document
>> you should clear the adaptive data.
>>
> In order to avoid doing that i have merged all 383 papers into a single
> one with "cat *.txt -> merged.txt" and i'm treating it as a single
> document...
> Is that a problem? I don't see how it could be...
>
>
Yes, it costs you a little recall and precision because the previous map
feature does not work this way.

Empty lines are used in the training data to indicate document boundaries.
And you need to put each sentence into one line.
A line a i used to indicate a sentence boundary.

Anyway I still believe something with you training data is wrong. Would it
be possible to have a look at one of
these papers? Or a few sentences?

Do you pass a sentence at a time to the name finder?

Jörn

Re: Problem with openNLP Name Finder API....

Posted by "Jim - FooBar();" <ji...@gmail.com>.
> That is how we defined the training format. The<START>  and<END>  tag MUST
> be white space separated,
> otherwise it will be recognized as a token.
I think you meant it *WON'T* be recognised as a token...a mere typo or 
have i misunderstood?
I mean its pretty obvious that spaces are necessary from all the 
exceptions thrown while counting the events! I'm just wondering why have 
you chosen to do that in the first place?No spaces makes a lot more 
sense to me regardless of the problem i am having with regex replacement...

> Have a look at our documentation. The NER code you see there is correct.
> If you have problems to detect multi-token names I suspect that something
> with your training data is wrong.
I've spent the last 2 weeks reading the docs and i 've practically read 
all the external sources on stackoverflow and otehr sources. However 
everyone's demo is about the same thing shown in the docs which is the 
person name finder (which happens to include a multi-word token - Pierre 
Vinken)! As far as the training data is concerned i've systematically 
checked it!!! I had to because as i said in the beginning i had nested 
tags which had to be sorted manually...I spent a whole day doing that 
but at least i was thinking "Finally i am so close to training...!!!". 
On top of that if there was something wrong with my training data i 
would expect exceptions again but i 'm not getting any since i sorted 
out the nested tags !!!
> The Name Finder takes a tokenized sentence at a time. After you are done
> with a document
> you should clear the adaptive data.
In order to avoid doing that i have merged all 383 papers into a single 
one with "cat *.txt -> merged.txt" and i'm treating it as a single 
document...
Is that a problem? I don't see how it could be...

Regards,
Jim




On 08/02/12 15:29, Joern Kottmann wrote:
> On Mon, Feb 6, 2012 at 6:15 PM, Jim - FooBar();<ji...@gmail.com>wrote:
>
>> Now the easy and straightforward solution to that is to NOT add spaces in
>> the tag like this :
>>
>> <START:drug>Folic acid<END>    (this will NOT match "acid" in later parsing)
>>
>> I honestly wasn't expecting that to make any difference to the training
>> process but as it turns out it breaks it completely.
>
>
> That is how we defined the training format. The<START>  and<END>  tag MUST
> be white space separated,
> otherwise it will be recognized as a token.
>
> Jörn
>


Re: Problem with openNLP Name Finder API....

Posted by Joern Kottmann <ko...@gmail.com>.
On Mon, Feb 6, 2012 at 6:15 PM, Jim - FooBar(); <ji...@gmail.com>wrote:

> Now the easy and straightforward solution to that is to NOT add spaces in
> the tag like this :
>
> <START:drug>Folic acid<END>   (this will NOT match "acid" in later parsing)
>
> I honestly wasn't expecting that to make any difference to the training
> process but as it turns out it breaks it completely.



That is how we defined the training format. The <START> and <END> tag MUST
be white space separated,
otherwise it will be recognized as a token.

Jörn

Re: Problem with openNLP Name Finder API....

Posted by Joern Kottmann <ko...@gmail.com>.
On Thu, Feb 9, 2012 at 1:16 AM, william.colen@gmail.com <
william.colen@gmail.com> wrote:

> Yes, maybe we should make the module dependencies clear to understand,
> maybe with some diagrams.
> The Tokenizer Tools documentation includes a passage explaining how to work
> with raw text. Maybe it would be helpful to add something like that in the
> Name Finder Tools documentation, since it is the most popular module.
>
>
> http://incubator.apache.org/opennlp/documentation/1.5.2-incubating/manual/opennlp.html#tools.tokenizer.cmdline
>
>
+1 that is something everyone needs to know and I guess it also something
which often is not done entirely correct. It is easy to get it wrong because
the name finder will still work even when a user gives it a paragraph or a
whole
document. The results are just not good, but many don't know what "good"
looks
like anyway when they try it out the first time.

Jörn

Re: Problem with openNLP Name Finder API....

Posted by "william.colen@gmail.com" <wi...@gmail.com>.
Yes, maybe we should make the module dependencies clear to understand,
maybe with some diagrams.
The Tokenizer Tools documentation includes a passage explaining how to work
with raw text. Maybe it would be helpful to add something like that in the
Name Finder Tools documentation, since it is the most popular module.

http://incubator.apache.org/opennlp/documentation/1.5.2-incubating/manual/opennlp.html#tools.tokenizer.cmdline


On Wed, Feb 8, 2012 at 9:52 PM, James Kosin <ja...@gmail.com> wrote:

> On 2/8/2012 6:20 PM, Aliaksandr Autayeu wrote:
> > On Wed, Feb 8, 2012 at 3:02 PM, Jim - FooBar(); <jimpil1985@gmail.com
> >wrote:
> >
> >> Any chance you remember whether you tokenized the sentences *and
> >> pos-tagged the tokens* before feeding them to the maxent NER model? I' m
> >> asking because the docs say you *ONLY* need to tokenize sentences before
> >> sending
> > AFAIK, there was no POS tagging.
> >
> > Aliaksandr
> >
> Actually, the documentation should say that it needs to be run through
> the sentence detector and tokenizer before sending to the name finder.
>
> James
>

Re: Problem with openNLP Name Finder API....

Posted by James Kosin <ja...@gmail.com>.
On 2/8/2012 6:20 PM, Aliaksandr Autayeu wrote:
> On Wed, Feb 8, 2012 at 3:02 PM, Jim - FooBar(); <ji...@gmail.com>wrote:
>
>> Any chance you remember whether you tokenized the sentences *and
>> pos-tagged the tokens* before feeding them to the maxent NER model? I' m
>> asking because the docs say you *ONLY* need to tokenize sentences before
>> sending
> AFAIK, there was no POS tagging.
>
> Aliaksandr
>
Actually, the documentation should say that it needs to be run through
the sentence detector and tokenizer before sending to the name finder.

James

Re: Problem with openNLP Name Finder API....

Posted by Aliaksandr Autayeu <al...@autayeu.com>.
On Wed, Feb 8, 2012 at 3:02 PM, Jim - FooBar(); <ji...@gmail.com>wrote:

> Any chance you remember whether you tokenized the sentences *and
> pos-tagged the tokens* before feeding them to the maxent NER model? I' m
> asking because the docs say you *ONLY* need to tokenize sentences before
> sending

AFAIK, there was no POS tagging.

Aliaksandr

Re: Problem with openNLP Name Finder API....

Posted by Joern Kottmann <ko...@gmail.com>.
Have a look at our documentation. The NER code you see there is correct.
If you have problems to detect multi-token names I suspect that something
with your training data is wrong.

The Name Finder takes a tokenized sentence at a time. After you are done
with a document
you should clear the adaptive data. POS tags are not used by the Name
Finder and cannot be passed to it.

Jörn

On Wed, Feb 8, 2012 at 3:02 PM, Jim - FooBar(); <ji...@gmail.com>wrote:

> Any chance you remember whether you tokenized the sentences *and
> pos-tagged the tokens* before feeding them to the maxent NER model? I' m
> asking because the docs say you *ONLY* need to tokenize sentences before
> sending them over to the trained model. However, i just stumbled upon this
> website: http://tech.knime.org/named-**entity-recognizer-and-tag-**
> cloud-example<http://tech.knime.org/named-entity-recognizer-and-tag-cloud-example>
>
> which states:
>
> " After pos tagging, the names entities can be tagged. The "OpenNLP NE
> tagger" node uses an OpenNLP NER model to tag the data. It is suggested to
> apply the NE tagger nodes after the pos tagger, in order to keep the named
> entities consisting of multi-words. "
>
> According to this i must pos-tag the tokens and NOT SIMPLY tokenize them
> if i want to keep multi-word entities as one!!! Could this be the case? Can
> you remember the details from your case?
>
> Regards,
> Jim
>
>
>
>
> On 08/02/12 11:44, Aliaksandr Autayeu wrote:
>
>> Yes, we had multiword entities. Actually, the dataset was quite "dirty"
>> and
>> "funny" - there were names like "al`XXX" and "al XXX" and some other where
>> the separator was some funny unicode character. But I don't remember any
>> problems similar to those you have (I followed the thread). But that was
>> OpenNLP 1.4.0 or 1.4.3, somewhere in that range. I don't have exact
>> figures
>> now, but I've fished out a precision (for one class) from an old
>> email: 80.98%
>>
>> Aliaksandr
>>
>> On Wed, Feb 8, 2012 at 11:45 AM, Jim - FooBar();<jimpil1985@gmail.com**
>> >wrote:
>>
>>  Hi there Autayeu,
>>>
>>> Did you have any multi-word entities in your annotated corpus?
>>> If yes, how did the maxent NER model perform? Could it find them or was
>>> it
>>> just finding single-word entities?
>>> If you don't understand why i'm asking have a  look at the previous
>>> messages....
>>>
>>> I really appreciate the help...
>>>
>>> Regards,
>>> Jim
>>>
>>>
>>>
>>> On 08/02/12 10:39, Aliaksandr Autayeu wrote:
>>>
>>>  p.s: have you ever done any serious NER (not for demonstration purposes)
>>>>
>>>>> using openNLP?
>>>>>
>>>>>  I did experiments (more than a year ago, with 1.4.3) for standard
>>>> three
>>>> classes, got the state of the art for our private corpus, but then we
>>>> changed approach.
>>>>
>>>> Aliaksandr
>>>>
>>>>
>>>>
>

Re: Problem with openNLP Name Finder API....

Posted by "Jim - FooBar();" <ji...@gmail.com>.
Any chance you remember whether you tokenized the sentences *and 
pos-tagged the tokens* before feeding them to the maxent NER model? I' m 
asking because the docs say you *ONLY* need to tokenize sentences before 
sending them over to the trained model. However, i just stumbled upon 
this website: 
http://tech.knime.org/named-entity-recognizer-and-tag-cloud-example

which states:

" After pos tagging, the names entities can be tagged. The "OpenNLP NE 
tagger" node uses an OpenNLP NER model to tag the data. It is suggested 
to apply the NE tagger nodes after the pos tagger, in order to keep the 
named entities consisting of multi-words. "

According to this i must pos-tag the tokens and NOT SIMPLY tokenize them 
if i want to keep multi-word entities as one!!! Could this be the case? 
Can you remember the details from your case?

Regards,
Jim



On 08/02/12 11:44, Aliaksandr Autayeu wrote:
> Yes, we had multiword entities. Actually, the dataset was quite "dirty" and
> "funny" - there were names like "al`XXX" and "al XXX" and some other where
> the separator was some funny unicode character. But I don't remember any
> problems similar to those you have (I followed the thread). But that was
> OpenNLP 1.4.0 or 1.4.3, somewhere in that range. I don't have exact figures
> now, but I've fished out a precision (for one class) from an old
> email: 80.98%
>
> Aliaksandr
>
> On Wed, Feb 8, 2012 at 11:45 AM, Jim - FooBar();<ji...@gmail.com>wrote:
>
>> Hi there Autayeu,
>>
>> Did you have any multi-word entities in your annotated corpus?
>> If yes, how did the maxent NER model perform? Could it find them or was it
>> just finding single-word entities?
>> If you don't understand why i'm asking have a  look at the previous
>> messages....
>>
>> I really appreciate the help...
>>
>> Regards,
>> Jim
>>
>>
>>
>> On 08/02/12 10:39, Aliaksandr Autayeu wrote:
>>
>>> p.s: have you ever done any serious NER (not for demonstration purposes)
>>>> using openNLP?
>>>>
>>> I did experiments (more than a year ago, with 1.4.3) for standard three
>>> classes, got the state of the art for our private corpus, but then we
>>> changed approach.
>>>
>>> Aliaksandr
>>>
>>>


Re: Problem with openNLP Name Finder API....

Posted by Aliaksandr Autayeu <al...@autayeu.com>.
Yes, we had multiword entities. Actually, the dataset was quite "dirty" and
"funny" - there were names like "al`XXX" and "al XXX" and some other where
the separator was some funny unicode character. But I don't remember any
problems similar to those you have (I followed the thread). But that was
OpenNLP 1.4.0 or 1.4.3, somewhere in that range. I don't have exact figures
now, but I've fished out a precision (for one class) from an old
email: 80.98%

Aliaksandr

On Wed, Feb 8, 2012 at 11:45 AM, Jim - FooBar(); <ji...@gmail.com>wrote:

> Hi there Autayeu,
>
> Did you have any multi-word entities in your annotated corpus?
> If yes, how did the maxent NER model perform? Could it find them or was it
> just finding single-word entities?
> If you don't understand why i'm asking have a  look at the previous
> messages....
>
> I really appreciate the help...
>
> Regards,
> Jim
>
>
>
> On 08/02/12 10:39, Aliaksandr Autayeu wrote:
>
>> p.s: have you ever done any serious NER (not for demonstration purposes)
>>> using openNLP?
>>>
>> I did experiments (more than a year ago, with 1.4.3) for standard three
>> classes, got the state of the art for our private corpus, but then we
>> changed approach.
>>
>> Aliaksandr
>>
>>
>

Re: Problem with openNLP Name Finder API....

Posted by "Jim - FooBar();" <ji...@gmail.com>.
Hi there Autayeu,

Did you have any multi-word entities in your annotated corpus?
If yes, how did the maxent NER model perform? Could it find them or was 
it just finding single-word entities?
If you don't understand why i'm asking have a  look at the previous 
messages....

I really appreciate the help...

Regards,
Jim


On 08/02/12 10:39, Aliaksandr Autayeu wrote:
>> p.s: have you ever done any serious NER (not for demonstration purposes)
>> using openNLP?
> I did experiments (more than a year ago, with 1.4.3) for standard three
> classes, got the state of the art for our private corpus, but then we
> changed approach.
>
> Aliaksandr
>


Re: Problem with openNLP Name Finder API....

Posted by Aliaksandr Autayeu <al...@autayeu.com>.
>
> p.s: have you ever done any serious NER (not for demonstration purposes)
> using openNLP?

I did experiments (more than a year ago, with 1.4.3) for standard three
classes, got the state of the art for our private corpus, but then we
changed approach.

Aliaksandr

Re: Problem with openNLP Name Finder API....

Posted by "Jim - FooBar();" <ji...@gmail.com>.
Hi James,

Ok i'll have a look at the new code but to be honest i 'm not having any 
problems with the case-sensitivity flag...Also, are you referring to the 
flag of the Dictionary or the trained NER model or both?

Sorry to ask the same question again but how on earth is the dictionary 
supposed to recognise words like "Folic acid" when all that it sees are 
"Folic" & "acid" ? I can understand that a properly trained model has 
generated features in order to learn what tokens to recognise. This 
usually includes words before and after the token in question amongst 
other things.... But the dictionary which is purely an exhaustive, 
iterative search based solely on a list of words - only looks at the 
token in question so i don't think it is possible to find multi-word 
entities using the DictionaryNameFinder.

Of course in my case, i 've got the same problem with both approaches 
which comes down to the tokenization that happens before the dictionary 
or the maxent model take over. But at least i can understand how the 
maxent model can look for multi-word entities but i certainly don't 
understand how the Dictionary can...

One of my fellow researchers suggested that as far as the maxent model 
is concerned, i should look at the confidence probabilities of each 
token in each sentence and concatenate any tokens with low enough 
confidence probabilities in order to get my multi-word tokens. Of course 
that is assuming that both "Folic" and "acid" are found by the NER and 
that they have very similar confidence probabilities. But that is not 
hte case with me...I mean it finds "Folic" and "Domoic" but NOT "acid" 
which is what follows both folic and domoic...So again dead end!!!

Is there any chance you can point me to the person that wrote the 
official tutorial for openNLP or the person that actually did the 
training for the provided  person name-finder model? I really want to 
hear his/her opinion on the matter...My problem is that the tutorial 
shows how to recognize person names (which are 2-3 words most of the 
times) but i strongly suspect that this is impossible without doing 
something extra which the tutorial unfortunately does not show...

By the way, the words i'm choosing to mention are not random...The words 
folic acid and domoic acid appear at least 200 times in the training 
data! The words "acid" appears at least 1000 times as part of other 
drug-names but it is still not recognised by the name-finder!!! Not even 
as a single token like it does with folic and domoic...

Regards,
Jim

p.s: have you ever done any serious NER (not for demonstration purposes) 
using openNLP?




On 08/02/12 01:23, James Kosin wrote:
> On 2/7/2012 5:29 AM, Jim - FooBar(); wrote:
>> Hey James,
>>
>> First of all thanks for taking the time to reply to my massive e-mail.
>> I really appreciate it...
>>
>> Secondly i think you slightly misunderstood my problems..to be fair
>> it's quite complicated!
>>
>> Ok, so i'm using openNLP 1.5.2 so i don't think there is any newer
>> nameFinder code.
>>
> Hi again,
>
> Actually, SVN has a later version that fixes a problem with the name
> matching over long series.  A bug in the code was causing the case
> sensitivity flag to go away when matching other tokens in a series.
> This was due to an attempted optimization that was working before we
> fixed the case sensitivity issues which broke the optimization.
>
> James


Re: Problem with openNLP Name Finder API....

Posted by James Kosin <ja...@gmail.com>.
On 2/7/2012 5:29 AM, Jim - FooBar(); wrote:
> Hey James,
>
> First of all thanks for taking the time to reply to my massive e-mail.
> I really appreciate it...
>
> Secondly i think you slightly misunderstood my problems..to be fair
> it's quite complicated!
>
> Ok, so i'm using openNLP 1.5.2 so i don't think there is any newer
> nameFinder code.
>
Hi again,

Actually, SVN has a later version that fixes a problem with the name
matching over long series.  A bug in the code was causing the case
sensitivity flag to go away when matching other tokens in a series. 
This was due to an attempted optimization that was working before we
fixed the case sensitivity issues which broke the optimization.

James

Re: Problem with openNLP Name Finder API....

Posted by "Jim - FooBar();" <ji...@gmail.com>.
Hey James,

First of all thanks for taking the time to reply to my massive e-mail. I 
really appreciate it...

Secondly i think you slightly misunderstood my problems..to be fair it's 
quite complicated!

Ok, so i'm using openNLP 1.5.2 so i don't think there is any newer 
nameFinder code.

You wrote:
> I'm not sure if what you are describing is a result of Clojure or a
> problem with the dictionary.  Our dictionary should be matching and
> returning on the largest match of tokens.  But, it could be the method
> you are using to tokenize the sentences.  Or maybe the lack there of.
Well, the Dictionary lookup works fine for all single-word entities 
(drugs). The same goes for the trained maxent NameFinder model.
It is multi-word entities that none of the above is finding because both 
the Dictionary and the maxent model expect tokenized sentences.
How can a dictionary or the maxent model find "Folic acid" when all that 
they see is "Folic" and "acid" as separate tokens?This is my big problem!!!!

My second  smaller problem has to do with the actual annotation. 
Obviously there is something different between:
> <START:drug>  drug-name<END>   (with spaces inside)
and

> <START:drug>drug-name<END>   (no spaces inside)

because i can train just fine using the 1st format but not when i use 
the 2nd one. This implies that the spaces inside matter a lot!
What i'm really asking is "Is it really a good idea to have mandatory 
spaces in the tag?". This mainly concerns the developers/maintainers of 
openNLP and not the users. I merely stated a concrete example where 
including spaces made my life a lot harder due to nested tags. Other 
people have had this problem as well even though some of them were 
trying to train a multi-type entity recognizer. My NER will be 
single-type (drugs).

> Other than that, it really isn't good to just use machine parsed data to
> blindly train new models.  It doesn't really work that way just yet.
> Mostly because you have to have human intervention to help correct the
> errors in the detection to get training data that is valid.  Many of the
> data sets we use also have this limitation in that we can't train items
> like the POS tagger on some ConLL data because the POS tags for many
> have been generated by machine and have not been verified for accuracy.
> We do use many of these though for the namefinder because some have
> actually been hand worked to add the name information needed.
Ok i get that and i totaly agree with you...BUT, you would be surprised 
with the annotated text my little program produced. As a reference i 
shall mention that it managed almost half a million annotations in 18000 
lines of text!!! Of course it missed some, mainly very complex chemical 
names that are impossible to parse with regex but generally it did a 
very good job...Particularly for approved drugs which have shorter names 
(4 words max.). You could grep  the drugbank.xml dictionary yourself. It 
has 6707 entries at the moment. It is the most complete list of drugs  
around. I'm not exactly looking for gold standard accuracy but rather a 
model that more or less works.

Also as i mentioned in the previous email, the openNLP tutorial and wiki 
are misleading, at least for NER. Specifically the person NER tutorial 
shows that
"Pierre Vinken" should be annotated as <START:person> Pierre Vinken 
<END> for training and that the the model produced should recognize 
Pierre Vinken as a single entity regardless of having received it as 2 
independant tokens " Pierre " " Vinken ". However that is not the 
case!!! If it could find Pierre Vinken it should be able to find "Folic 
acid" or "Domoic acid" etc... shouldn't it???

Kind regards,
Jim

p.s:
Clojure is certainly not to blame...Clojure provides superb Java 
interoperability with no transition layer (really fast)! All i 'm doing 
is calling Java code from Clojure - how can that be a problem?





On 07/02/12 05:29, James Kosin wrote:
> Hi Jim,
>
> On 2/6/2012 12:15 PM, Jim - FooBar(); wrote:
>> Hello openNLP users/developers,
>>
>> I really need your help with openNLP NameFinder...Let me explain:
>>
>> I am writing a drug-entity recogniser (NER for drugs) and i 'm using
>> the openNLP API from Clojure. I don't have annotated text but i do an
>> up-to-date dictionary of drugs (drugbank.xml). The dictionary includes
>> all sorts of information so it needs a bit of preprocessing to extract
>> the names and synonyms. Anyway my problem is twofold:
>>
>> As i said i don't have annotated text, but i thought maybe i can make
>> one. You see, i do have 383 pharmacology papers in raw text so i
>> thought why not use the names from the dictionary and build regex
>> patterns to replace all occurrences of that entry with the appropriate
>> "<START:drug>  /drug-name/<END>" annotation tag. Now, you may be
>> wondering at this point why on earth am i  not using all the names i
>> extracted from drugbank.xml to build a proper openNLP dictionary to do
>> lookup...Well, apart from not being what i'm trying to do here, i've
>> already tried doing that but i got poor results  simply because words
>> like "Folic acid" are being tokenized as 2 tokens rather than 1 and
>> thus, the entry "Folic acid" in the dictionary matches no token! Even
>> if it worked thoughi would still have to train a maxent model to
>> recognise drugs that may not exist in the dictionary (brand new for
>> instance). My initial approach was a bit fiddly but it sort of paid of
>> at the end. I now have a small program that expects some text and, for
>> each entry in the dictionary (6707 in total) it finds and replaces any
>> occurrences of that entry in the text with the expected openNLP format
>> for training. Here is where the 1st problem happens. I can tweak my
>> regex pattern to add spaces to the entity tag or not, with the
>> following results :
>>
>> <START:drug>  drug-name<END>   (with spaces inside)
>>
>> causes problems for me because i get nested tags. To understand why
>> think about the words "Folic acid" for example. Lets assume that folic
>> acid is entry 3 in the dictionary and that entry 115 is "acid". First
>> time round it will produce<START:drug>  Folic acid<END>  but when it
>> processes "acid" it will match the word acid already tagged with
>> "Folic acid". You can see where this is going. If you happen to have a
>> complex compound name and after a while a slightly less complex
>> compound name (maybe a word shorter or something), and then a smaller
>> one, they can easily start to nest, especially when dealing with drug
>> names. Now the easy and straightforward solution to that is to NOT add
>> spaces in the tag like this :
>>
>> <START:drug>Folic acid<END>    (this will NOT match "acid" in later
>> parsing)
>>
>> I honestly wasn't expecting that to make any difference to the
>> training process but as it turns out it breaks it completely.
>> Exceptions everywhere before it even starts!!! Could please someone
>> explain what happens with those spaces around the entity name? How on
>> earth can they make any difference? I can solve my problem by doing
>> negative lookbehind assertion in my regex but that slows things down
>> quite a bit! remember i'm dealing with 6707 entries, times 383 papers.
>> Clojure's lazy attitude and loop/recur structure sure help a lot...
> I'm not sure if what you are describing is a result of Clojure or a
> problem with the dictionary.  Our dictionary should be matching and
> returning on the largest match of tokens.  But, it could be the method
> you are using to tokenize the sentences.  Or maybe the lack there of.
>
> What usually happens, is that we train or use a SentenceDetector to
> first take the document and return the sentences as separate lines.
> Then, a Tokenizer is used to parse the sentence into its individual
> words and tokens or punctuation.
>
> Example:
>      This is a good sentence to "parse" on the way out.
> When tokenized:
>      This is a good sentence to " parse " on the way out .
>
> The tokenizer splits everything into individual tokens to be parsed by
> later models.
>
>> Ok now on to the 2nd problem...
>>
>> Even when i manually sort all the nested tags and i  finally train a
>> maxent model on the newly automatically annotated papers, i still get
>> very poor results (poorer than the dictionary) and i think i can see
>> why but that contradicts the openNLP documentation and all the
>> examples i 've seen so far.
>>
>> On the openNLP tutorial it seems perfectly normal to have 2 words
>> inside a tag like:
>>
>> <START:name>  Pierre Vinken<END>
>>
>> but when the time comes to use the name-finder model you just trained
>> everything has to be tokenized again. Therefore, "Pierre Vinken"
>> becomes 2 tokens and cannot be recognised. Sometimes just the one
>> token may be recognised as an entity but without the rest of the name
>> is not only meaningless but could be misleading. Again think about
>> Folic acid... Neither "folic" nor "acid" are drugs...even if the
>> name-finder recognises "folic", it's of no use!!! In the same way that
>> "Vinken" is not necessarily a name...
>>
>> Anyway sorry for the massive e-mail but i'm really struggling!
>>
>>
>> Please help me, i'm at a dead-end at the moment! i've tried literally
>> everything...am i missing anything important?
> Have you tried the latest code for the namefinder?  We have fixed a few
> deficiencies in the code that may be causing some of your problems
> outside of the actual training.
>
> Other than that, it really isn't good to just use machine parsed data to
> blindly train new models.  It doesn't really work that way just yet.
> Mostly because you have to have human intervention to help correct the
> errors in the detection to get training data that is valid.  Many of the
> data sets we use also have this limitation in that we can't train items
> like the POS tagger on some ConLL data because the POS tags for many
> have been generated by machine and have not been verified for accuracy.
> We do use many of these though for the namefinder because some have
> actually been hand worked to add the name information needed.
>
>> Thanks in advance...keep up the good work!
>>
>> JIm
>>
>>
>>


Re: Problem with openNLP Name Finder API....

Posted by James Kosin <ja...@gmail.com>.
Hi Jim,

On 2/6/2012 12:15 PM, Jim - FooBar(); wrote:
> Hello openNLP users/developers,
>
> I really need your help with openNLP NameFinder...Let me explain:
>
> I am writing a drug-entity recogniser (NER for drugs) and i 'm using
> the openNLP API from Clojure. I don't have annotated text but i do an
> up-to-date dictionary of drugs (drugbank.xml). The dictionary includes
> all sorts of information so it needs a bit of preprocessing to extract
> the names and synonyms. Anyway my problem is twofold:
>
> As i said i don't have annotated text, but i thought maybe i can make
> one. You see, i do have 383 pharmacology papers in raw text so i
> thought why not use the names from the dictionary and build regex
> patterns to replace all occurrences of that entry with the appropriate
> "<START:drug> /drug-name/ <END>" annotation tag. Now, you may be
> wondering at this point why on earth am i  not using all the names i
> extracted from drugbank.xml to build a proper openNLP dictionary to do
> lookup...Well, apart from not being what i'm trying to do here, i've
> already tried doing that but i got poor results  simply because words
> like "Folic acid" are being tokenized as 2 tokens rather than 1 and
> thus, the entry "Folic acid" in the dictionary matches no token! Even
> if it worked thoughi would still have to train a maxent model to
> recognise drugs that may not exist in the dictionary (brand new for
> instance). My initial approach was a bit fiddly but it sort of paid of
> at the end. I now have a small program that expects some text and, for
> each entry in the dictionary (6707 in total) it finds and replaces any
> occurrences of that entry in the text with the expected openNLP format
> for training. Here is where the 1st problem happens. I can tweak my
> regex pattern to add spaces to the entity tag or not, with the
> following results :
>
> <START:drug> drug-name <END>  (with spaces inside)
>
> causes problems for me because i get nested tags. To understand why
> think about the words "Folic acid" for example. Lets assume that folic
> acid is entry 3 in the dictionary and that entry 115 is "acid". First
> time round it will produce <START:drug> Folic acid <END> but when it
> processes "acid" it will match the word acid already tagged with
> "Folic acid". You can see where this is going. If you happen to have a
> complex compound name and after a while a slightly less complex
> compound name (maybe a word shorter or something), and then a smaller
> one, they can easily start to nest, especially when dealing with drug
> names. Now the easy and straightforward solution to that is to NOT add
> spaces in the tag like this :
>
> <START:drug>Folic acid<END>   (this will NOT match "acid" in later
> parsing)
>
> I honestly wasn't expecting that to make any difference to the
> training process but as it turns out it breaks it completely.
> Exceptions everywhere before it even starts!!! Could please someone
> explain what happens with those spaces around the entity name? How on
> earth can they make any difference? I can solve my problem by doing
> negative lookbehind assertion in my regex but that slows things down
> quite a bit! remember i'm dealing with 6707 entries, times 383 papers.
> Clojure's lazy attitude and loop/recur structure sure help a lot...
I'm not sure if what you are describing is a result of Clojure or a
problem with the dictionary.  Our dictionary should be matching and
returning on the largest match of tokens.  But, it could be the method
you are using to tokenize the sentences.  Or maybe the lack there of.

What usually happens, is that we train or use a SentenceDetector to
first take the document and return the sentences as separate lines.
Then, a Tokenizer is used to parse the sentence into its individual
words and tokens or punctuation.

Example:
    This is a good sentence to "parse" on the way out.
When tokenized:
    This is a good sentence to " parse " on the way out .

The tokenizer splits everything into individual tokens to be parsed by
later models.

>
> Ok now on to the 2nd problem...
>
> Even when i manually sort all the nested tags and i  finally train a
> maxent model on the newly automatically annotated papers, i still get
> very poor results (poorer than the dictionary) and i think i can see
> why but that contradicts the openNLP documentation and all the
> examples i 've seen so far.
>
> On the openNLP tutorial it seems perfectly normal to have 2 words
> inside a tag like:
>
> <START:name> Pierre Vinken <END>
>
> but when the time comes to use the name-finder model you just trained
> everything has to be tokenized again. Therefore, "Pierre Vinken"
> becomes 2 tokens and cannot be recognised. Sometimes just the one
> token may be recognised as an entity but without the rest of the name
> is not only meaningless but could be misleading. Again think about
> Folic acid... Neither "folic" nor "acid" are drugs...even if the
> name-finder recognises "folic", it's of no use!!! In the same way that
> "Vinken" is not necessarily a name...
>
> Anyway sorry for the massive e-mail but i'm really struggling!
>
>
> Please help me, i'm at a dead-end at the moment! i've tried literally
> everything...am i missing anything important?
Have you tried the latest code for the namefinder?  We have fixed a few
deficiencies in the code that may be causing some of your problems
outside of the actual training.

Other than that, it really isn't good to just use machine parsed data to
blindly train new models.  It doesn't really work that way just yet. 
Mostly because you have to have human intervention to help correct the
errors in the detection to get training data that is valid.  Many of the
data sets we use also have this limitation in that we can't train items
like the POS tagger on some ConLL data because the POS tags for many
have been generated by machine and have not been verified for accuracy. 
We do use many of these though for the namefinder because some have
actually been hand worked to add the name information needed.

> Thanks in advance...keep up the good work!
>
> JIm
>
>
>