You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by "Jim - FooBar();" <ji...@gmail.com> on 2012/03/14 00:09:05 UTC
Dictionary name finder question
Could you please confirm that the dictionary name-finder can identify
multi-word entities after the recent case-sensitivity changes that were
made? All of a sudden my dictionary stopped recognising multi-word
entities even though they are encoded correctly as 2 in the .xml
dictionary file. I thought i had this sorted - my old problem had to do
with how i was creating the dictionary but now i 'm using the cmd-line
tool...
Jim
p.s: case sensitivity is indeed done correctly now!
Re: Dictionary name finder question
Posted by James Kosin <ja...@gmail.com>.
Jim,
I've confirmed and there are also tests in the test packages to catch
both case-sensitivity issues and name finder across multiple tokens (words).
James
On 3/13/2012 7:09 PM, Jim - FooBar(); wrote:
> Could you please confirm that the dictionary name-finder can identify
> multi-word entities after the recent case-sensitivity changes that
> were made? All of a sudden my dictionary stopped recognising
> multi-word entities even though they are encoded correctly as 2 in
> the .xml dictionary file. I thought i had this sorted - my old problem
> had to do with how i was creating the dictionary but now i 'm using
> the cmd-line tool...
>
> Jim
>
> p.s: case sensitivity is indeed done correctly now!
Re: Dictionary name finder question
Posted by Jörn Kottmann <ko...@gmail.com>.
On 03/14/2012 12:19 PM, Dimitrios wrote:
> On 14/03/12 11:13, Jörn Kottmann wrote:
>> In your case:
>> svn co https://svn.apache.org/repos/asf/opennlp/trunk/
>
> http works for me while https does not...just thought you should know
Not sure why is that, anyway https is only needed for commit access.
Jörn
Re: Dictionary name finder question
Posted by Dimitrios <ji...@gmail.com>.
On 14/03/12 11:13, Jörn Kottmann wrote:
> In your case:
> svn co https://svn.apache.org/repos/asf/opennlp/trunk/
http works for me while https does not...just thought you should know
Jim
Re: Dictionary name finder question
Posted by Jörn Kottmann <ko...@gmail.com>.
On 03/14/2012 12:07 PM, Dimitrios wrote:
> p.s: can i see the test sources you mentioned without checking out the
> repo? maybe online somewhere?
Instructions are here:
http://opennlp.apache.org/source-code.html
There is web interface you can use to browse the subversion contents.
To checkout you need subversion installed and then it is really just
one command you have to copy over to your console.
In your case:
svn co https://svn.apache.org/repos/asf/opennlp/trunk/
Jörn
Re: Dictionary name finder question
Posted by James Kosin <ja...@gmail.com>.
Jim,
What is the output of the tokenize model, look like?
James
On 3/14/2012 7:34 AM, Dimitrios wrote:
> On 14/03/12 11:23, Jörn Kottmann wrote:
>> Can you re-produce your issue with a dictionary which only contains a
>> single entry?
>
> Yes i can indeed re-produce the issue with the following dictionary:
> --------------------------------------------------------------------------------------------
>
> <?xml version="1.0" encoding="UTF-8"?>
> <dictionary case_sensitive="false">
> <entry>
> <token>Folic</token>
> <token>acid</token>
> </entry>
> <entry>
> <token>Baclofen</token>
> </entry>
> </dictionary>
> --------------------------------------------------------------------------------------------
>
>
> The small paragraph i'm using for testing is this:
>
> "Folic acid is one variable, but other factors remain.
> Studies suggest that substances active at the GABA receptor may
> produce NTDs.
> To test this hypothesis pregnant rats were exposed to either the GABA
> a agonist muscimol (1, 2 or 4 mg/kg), the GABA a antagonist
> bicuculline (.5, 1, or 2 mg/kg), the GABA b agonist baclofen (15, 30,
> 60 mg/kg), or the GABA b antagonist hydroxysaclofen (1, 3, or 5 mg/kg)
> during neural tube formation.
> Normal saline was used as a control and valproic acid (600 mg/kg) as a
> positive control."
>
>
> The dictionary finds "baclofen" but it does not find "Folic acid"! The
> workflow is as follows:
>
> 1. get-sentences
> 2. tokenize -sentences
> 3. call dictionary name finder ".find()" method with an array of srings
> (tokens of a single sentence)
>
> Jim
>
>
Re: Dictionary name finder question
Posted by Dimitrios <ji...@gmail.com>.
aaa ok sorry i did not mean to sound impatient...
Jim
On 14/03/12 14:28, Jörn Kottmann wrote:
> That simply means that nobody yet tested/debugged with
> your posted dictionary and sample.
>
> Jörn
>
> On 03/14/2012 03:24 PM, Dimitrios wrote:
>> No response in the last 2-3 hours...
>>
>> Should i assume that someone did confirm my issue using the tiny
>> dictionary and the test paragraph i posted before?
>>
>> Or should i assume that Jame's confirmation that everything works ok
>> still stands?
>>
>> Jim
>>
>>
>> On 14/03/12 11:39, Dimitrios wrote:
>>> Forgot to mention that after find() returns the Spans i'm doing
>>> Span.SpansToStrings(span-array token-array) to get the human
>>> readable array of strings....
>>>
>>> Jim
>>>
>>>
>>> On 14/03/12 11:34, Dimitrios wrote:
>>>> On 14/03/12 11:23, Jörn Kottmann wrote:
>>>>> Can you re-produce your issue with a dictionary which only
>>>>> contains a single entry?
>>>>
>>>> Yes i can indeed re-produce the issue with the following dictionary:
>>>> --------------------------------------------------------------------------------------------
>>>>
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <dictionary case_sensitive="false">
>>>> <entry>
>>>> <token>Folic</token>
>>>> <token>acid</token>
>>>> </entry>
>>>> <entry>
>>>> <token>Baclofen</token>
>>>> </entry>
>>>> </dictionary>
>>>> --------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> The small paragraph i'm using for testing is this:
>>>>
>>>> "Folic acid is one variable, but other factors remain.
>>>> Studies suggest that substances active at the GABA receptor may
>>>> produce NTDs.
>>>> To test this hypothesis pregnant rats were exposed to either the
>>>> GABA a agonist muscimol (1, 2 or 4 mg/kg), the GABA a antagonist
>>>> bicuculline (.5, 1, or 2 mg/kg), the GABA b agonist baclofen (15,
>>>> 30, 60 mg/kg), or the GABA b antagonist hydroxysaclofen (1, 3, or 5
>>>> mg/kg) during neural tube formation.
>>>> Normal saline was used as a control and valproic acid (600 mg/kg)
>>>> as a positive control."
>>>>
>>>>
>>>> The dictionary finds "baclofen" but it does not find "Folic acid"!
>>>> The workflow is as follows:
>>>>
>>>> 1. get-sentences
>>>> 2. tokenize -sentences
>>>> 3. call dictionary name finder ".find()" method with an array of
>>>> srings (tokens of a single sentence)
>>>>
>>>> Jim
>>>>
>>>
>>
>>
>
Re: Dictionary name finder question
Posted by James Kosin <ja...@gmail.com>.
Jorn,
We have a problem, but, need some guidance on how we want to address the
issue here.
The problem is that the hash code doesn't compute for the single token
unless we already have a single token in the dictionary with that match.
In simple English, the test dictionary has "venessa" and "venessa",
"williams" added as two entries. When the Dictionary name finder tries
to find... first "Venessa" we have a match for the first dictionary
entry... when it later tests with "Venessa", "Williams" we get a hit on
the second entry.
The problem is if we don't have "venessa" in the dictionary... we don't
get the first match; so, the DictionaryNameFinder overlooks the name
entirely.
So the problem is all in the hashcode.............
Question is how to fix, or address the issue without making this an 2^N
type problem.
James
On 3/14/2012 10:28 AM, Jörn Kottmann wrote:
> That simply means that nobody yet tested/debugged with
> your posted dictionary and sample.
>
> Jörn
>
> On 03/14/2012 03:24 PM, Dimitrios wrote:
>> No response in the last 2-3 hours...
>>
>> Should i assume that someone did confirm my issue using the tiny
>> dictionary and the test paragraph i posted before?
>>
>> Or should i assume that Jame's confirmation that everything works ok
>> still stands?
>>
>> Jim
>>
>>
>> On 14/03/12 11:39, Dimitrios wrote:
>>> Forgot to mention that after find() returns the Spans i'm doing
>>> Span.SpansToStrings(span-array token-array) to get the human
>>> readable array of strings....
>>>
>>> Jim
>>>
>>>
>>> On 14/03/12 11:34, Dimitrios wrote:
>>>> On 14/03/12 11:23, Jörn Kottmann wrote:
>>>>> Can you re-produce your issue with a dictionary which only
>>>>> contains a single entry?
>>>>
>>>> Yes i can indeed re-produce the issue with the following dictionary:
>>>> --------------------------------------------------------------------------------------------
>>>>
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <dictionary case_sensitive="false">
>>>> <entry>
>>>> <token>Folic</token>
>>>> <token>acid</token>
>>>> </entry>
>>>> <entry>
>>>> <token>Baclofen</token>
>>>> </entry>
>>>> </dictionary>
>>>> --------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> The small paragraph i'm using for testing is this:
>>>>
>>>> "Folic acid is one variable, but other factors remain.
>>>> Studies suggest that substances active at the GABA receptor may
>>>> produce NTDs.
>>>> To test this hypothesis pregnant rats were exposed to either the
>>>> GABA a agonist muscimol (1, 2 or 4 mg/kg), the GABA a antagonist
>>>> bicuculline (.5, 1, or 2 mg/kg), the GABA b agonist baclofen (15,
>>>> 30, 60 mg/kg), or the GABA b antagonist hydroxysaclofen (1, 3, or 5
>>>> mg/kg) during neural tube formation.
>>>> Normal saline was used as a control and valproic acid (600 mg/kg)
>>>> as a positive control."
>>>>
>>>>
>>>> The dictionary finds "baclofen" but it does not find "Folic acid"!
>>>> The workflow is as follows:
>>>>
>>>> 1. get-sentences
>>>> 2. tokenize -sentences
>>>> 3. call dictionary name finder ".find()" method with an array of
>>>> srings (tokens of a single sentence)
>>>>
>>>> Jim
>>>>
>>>
>>
>>
>
Re: Dictionary name finder question
Posted by Jörn Kottmann <ko...@gmail.com>.
That simply means that nobody yet tested/debugged with
your posted dictionary and sample.
Jörn
On 03/14/2012 03:24 PM, Dimitrios wrote:
> No response in the last 2-3 hours...
>
> Should i assume that someone did confirm my issue using the tiny
> dictionary and the test paragraph i posted before?
>
> Or should i assume that Jame's confirmation that everything works ok
> still stands?
>
> Jim
>
>
> On 14/03/12 11:39, Dimitrios wrote:
>> Forgot to mention that after find() returns the Spans i'm doing
>> Span.SpansToStrings(span-array token-array) to get the human readable
>> array of strings....
>>
>> Jim
>>
>>
>> On 14/03/12 11:34, Dimitrios wrote:
>>> On 14/03/12 11:23, Jörn Kottmann wrote:
>>>> Can you re-produce your issue with a dictionary which only contains
>>>> a single entry?
>>>
>>> Yes i can indeed re-produce the issue with the following dictionary:
>>> --------------------------------------------------------------------------------------------
>>>
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <dictionary case_sensitive="false">
>>> <entry>
>>> <token>Folic</token>
>>> <token>acid</token>
>>> </entry>
>>> <entry>
>>> <token>Baclofen</token>
>>> </entry>
>>> </dictionary>
>>> --------------------------------------------------------------------------------------------
>>>
>>>
>>> The small paragraph i'm using for testing is this:
>>>
>>> "Folic acid is one variable, but other factors remain.
>>> Studies suggest that substances active at the GABA receptor may
>>> produce NTDs.
>>> To test this hypothesis pregnant rats were exposed to either the
>>> GABA a agonist muscimol (1, 2 or 4 mg/kg), the GABA a antagonist
>>> bicuculline (.5, 1, or 2 mg/kg), the GABA b agonist baclofen (15,
>>> 30, 60 mg/kg), or the GABA b antagonist hydroxysaclofen (1, 3, or 5
>>> mg/kg) during neural tube formation.
>>> Normal saline was used as a control and valproic acid (600 mg/kg) as
>>> a positive control."
>>>
>>>
>>> The dictionary finds "baclofen" but it does not find "Folic acid"!
>>> The workflow is as follows:
>>>
>>> 1. get-sentences
>>> 2. tokenize -sentences
>>> 3. call dictionary name finder ".find()" method with an array of
>>> srings (tokens of a single sentence)
>>>
>>> Jim
>>>
>>
>
>
RE: Dictionary name finder question
Posted by Marcelo Lombardi <ma...@gmail.com>.
Thank you, Jörn! I will do that right now.
-----Mensaje original-----
De: Jörn Kottmann [mailto:kottmann@gmail.com]
Enviado el: viernes, 16 de marzo de 2012 20:00
Para: users@opennlp.apache.org
Asunto: Re: Dictionary name finder question
Send a mail to:
users-unsubscribe@opennlp.apache.org
Make sure you send it from your gmail account.
Jörn
On 03/16/2012 11:41 PM, Marcelo Lombardi wrote:
> Hello,
>
> I tried to unsubscribe 3 times with no success. I would appreciate it
> if somebody can get me unsubscribed for real. Clearly something does not
work.
> Any suggestions?
>
> Thanks.
>
> -----Mensaje original-----
> De: Jim - FooBar(); [mailto:jimpil1985@gmail.com] Enviado el: viernes,
> 16 de marzo de 2012 14:24
> Para: users@opennlp.apache.org
> Asunto: Re: Dictionary name finder question
>
> Hey Oliver,
>
> yes that is exactly the point...it does work for 1.5.2 (even though
> case-sensitivity is not to be trusted) but it doesn't work for the
> latest svn code which does case sensitivity properly...thanks for
confirming...
> James has already been notified...
>
> Jim
>
>
> On 16/03/12 14:58, Oliver Cromm wrote:
>> I just happened to test this functionality with the 1.5.2 version,
>> and I can confirm that "Barack Obama" was found, whether or not "Barack"
>> is also present in the dictionary.
>
>
Re: Dictionary name finder question
Posted by Jörn Kottmann <ko...@gmail.com>.
Send a mail to:
users-unsubscribe@opennlp.apache.org
Make sure you send it from your gmail account.
Jörn
On 03/16/2012 11:41 PM, Marcelo Lombardi wrote:
> Hello,
>
> I tried to unsubscribe 3 times with no success. I would appreciate it if
> somebody can get me unsubscribed for real. Clearly something does not work.
> Any suggestions?
>
> Thanks.
>
> -----Mensaje original-----
> De: Jim - FooBar(); [mailto:jimpil1985@gmail.com]
> Enviado el: viernes, 16 de marzo de 2012 14:24
> Para: users@opennlp.apache.org
> Asunto: Re: Dictionary name finder question
>
> Hey Oliver,
>
> yes that is exactly the point...it does work for 1.5.2 (even though
> case-sensitivity is not to be trusted) but it doesn't work for the latest
> svn code which does case sensitivity properly...thanks for confirming...
> James has already been notified...
>
> Jim
>
>
> On 16/03/12 14:58, Oliver Cromm wrote:
>> I just happened to test this functionality with the 1.5.2 version, and
>> I can confirm that "Barack Obama" was found, whether or not "Barack"
>> is also present in the dictionary.
>
>
RE: Dictionary name finder question
Posted by Marcelo Lombardi <ma...@gmail.com>.
Hello,
I tried to unsubscribe 3 times with no success. I would appreciate it if
somebody can get me unsubscribed for real. Clearly something does not work.
Any suggestions?
Thanks.
-----Mensaje original-----
De: Jim - FooBar(); [mailto:jimpil1985@gmail.com]
Enviado el: viernes, 16 de marzo de 2012 14:24
Para: users@opennlp.apache.org
Asunto: Re: Dictionary name finder question
Hey Oliver,
yes that is exactly the point...it does work for 1.5.2 (even though
case-sensitivity is not to be trusted) but it doesn't work for the latest
svn code which does case sensitivity properly...thanks for confirming...
James has already been notified...
Jim
On 16/03/12 14:58, Oliver Cromm wrote:
> I just happened to test this functionality with the 1.5.2 version, and
> I can confirm that "Barack Obama" was found, whether or not "Barack"
> is also present in the dictionary.
Re: Dictionary name finder question
Posted by "Jim - FooBar();" <ji...@gmail.com>.
Hey James,
On 17/03/12 02:49, James Kosin wrote:
> If you two could test the latest out, I'd appreciate.
> Especially any performance issues, if possible. I'm trying to be sure I
> haven't turned this into a N^2 type problem again. If so, I'll need to
> re-open the JIRA and fix the Index class to handle case sensitivity.
Well, the dictionary name finder is indeed a bit slower now, but that is
fine with me...Tested it with a relatively big test corpus and the the
Dictionary lookup took 2-3 seconds more than the maxent model. Now, even
though that is counter-intuitive (you expect the iterative search to be
extremely fast), i simply don't care at this point - it is not a problem
for me! The fact that is it finds multi-word tokens is the most
important fix with case-sensitivity coming second for me (i can always
uncapitalize my dictionary)...
> (b) I tried to fix the DictionaryNameFinder.... woops, I refactored
> incorrectly. Unfortunately, you two are but a few that use the
> DictionaryNameFinder.
Maybe we are just a few because the DictionaryNameFinder never quite
worked as advertised...i do expect more people to start using it
especially if it can be integrated with the maxent model (from the
evaluators point of view)...It is the easiest way to improve one's
results without cheating!
> Thanks for your patience, testing and posting to the list.
Don't mention it man... Thank YOU for addressing it! :-)
> I can see clearly now my mistakes.
That is what this is all about! Good stuff...
> The code currently in SVN trunk has sort-of a compromise until we get
> the Index working again properly with case sensitivity. It contains
> code that will keep trying longer token entries as long as the current
> length is less than the maximum held in the dictionary. This allows the
> DictionaryNameFinder's find() method to work; but, we have a small
> performance penalty due to the way the find() method isn't caring what
> words it adds to the token strings.
So, by 'performance penalty' you mean runtime performance as opposed to
accuracy performance yes?
This is exactly what i confirmed above...It takes roughly twice the time
for the Dictionary to do its job , than it takes the maxent
model...Nevertheless, what comes back are the correct, case-insensitive
named entities so all is good (at least for me!)...
> I'm going to look at possible solutions to getting the Index working
> again properly with the DictionaryNameFinder... but, it will take some time.
Excellent...is there any way for me to find out whenever you fix it? I
mean, will you post anything here or is there a JIRA i can start "watching"?
Also, if i manage to 'hack' the evaluator to take into account both the
maxent model and dictionary findings to improve the statistics, is that
something you would consider adding to openNLP? There is a JIRA for it
from last year, which i voted for and commented on...I'm not sure if
you've seen it...
Thanks again for the patch (regardless of the compromise)...:-)
Jim
Re: Dictionary name finder question
Posted by James Kosin <ja...@gmail.com>.
Oliver and Jim,
If you two could test the latest out, I'd appreciate.
Especially any performance issues, if possible. I'm trying to be sure I
haven't turned this into a N^2 type problem again. If so, I'll need to
re-open the JIRA and fix the Index class to handle case sensitivity.
The problem is two fold:
(a) We tried to fix case sensitivity in 1.5.2 for the Dictionary
class. This was fixed; however, as damage the DictionaryNameFinder also
was still broken.
(b) I tried to fix the DictionaryNameFinder.... woops, I refactored
incorrectly. Unfortunately, you two are but a few that use the
DictionaryNameFinder.
Thanks for your patience, testing and posting to the list.
The original code.. I understand a bit more now was looking in the Index
for words that were in the dictionary to help determine if we should be
trying to add more tokens. I misunderstood this in my refactoring to
take out the Index due to its lack of case sensitivity settings.
I can see clearly now my mistakes.
The code currently in SVN trunk has sort-of a compromise until we get
the Index working again properly with case sensitivity. It contains
code that will keep trying longer token entries as long as the current
length is less than the maximum held in the dictionary. This allows the
DictionaryNameFinder's find() method to work; but, we have a small
performance penalty due to the way the find() method isn't caring what
words it adds to the token strings.
I'm going to look at possible solutions to getting the Index working
again properly with the DictionaryNameFinder... but, it will take some time.
Thanks,
James
On 3/16/2012 1:24 PM, Jim - FooBar(); wrote:
> Hey Oliver,
>
> yes that is exactly the point...it does work for 1.5.2 (even though
> case-sensitivity is not to be trusted) but it doesn't work for the
> latest svn code which does case sensitivity properly...thanks for
> confirming... James has already been notified...
>
> Jim
>
>
> On 16/03/12 14:58, Oliver Cromm wrote:
>> I just happened to test this functionality with the 1.5.2 version,
>> and I can confirm that "Barack Obama" was found, whether or not
>> "Barack" is also present in the dictionary.
>
>
Re: Dictionary name finder question
Posted by "Jim - FooBar();" <ji...@gmail.com>.
Hey Oliver,
yes that is exactly the point...it does work for 1.5.2 (even though
case-sensitivity is not to be trusted) but it doesn't work for the
latest svn code which does case sensitivity properly...thanks for
confirming... James has already been notified...
Jim
On 16/03/12 14:58, Oliver Cromm wrote:
> I just happened to test this functionality with the 1.5.2 version,
> and I can confirm that "Barack Obama" was found, whether or not
> "Barack" is also present in the dictionary.
RE: Dictionary name finder question
Posted by Oliver Cromm <oc...@opentext.com>.
I just happened to test this functionality with the 1.5.2 version,
and I can confirm that "Barack Obama" was found, whether or not
"Barack" is also present in the dictionary.
As for case sensitivity, I'm not sure about the details - at first I
had case-insensitive matching which I didn't want, and it got better
when I manually changed the flag in the dictionary xml. However, I
cannot reproduce that with my small example, it seems to be case-
sensitive all the time.
____
Oliver Cromm, Computational Linguist
Open Text
-----Original Message-----
From: Jim - FooBar(); [mailto:jimpil1985@gmail.com]
Sent: Friday, 16 March, 2012 6:13
To: users@opennlp.apache.org
Subject: Re: Dictionary name finder question
Hi James,
Well, i'm not sure if this is good news but i just tested the dictionary
with version 1.5.2 and it does find multi-word entities just as i
remembered!Of course, case sensitivity does not quite work so it finds
"Folic acid" (exact match) but it doesn't find "folic acid" or
"baclofen" (both uncapitalized)...
Presumably, this is indeed good news as a working version exists yes? I
think we're getting somewhere aren't we?
Jim
On 16/03/12 02:48, James Kosin wrote:
> If I broke something, let me know so I can get original functionality
> even if it involves some minor changes to incorporate.
>
> I'll go ahead with a variable for the dictionary to store the longest
> token count just in case I go that route. The Index was a good idea;
> but, it doesn't carry over the case sensitivity to its members that are
> in the dictionary.
>
> James
>
> On 3/15/2012 8:06 PM, Jim - FooBar(); wrote:
>> aaaaa ok i see what you mean! that makes perfect sense - thanks for
>> being so thorough in explaining...
>>
>> I will check first thing tomorrow morning to see which one was the
>> version that was returning multi-word entities properly...In fact i
>> specifically remember getting back "both "folic acid" and "valproic
>> acid" in the little paragraph i posted....anyway i'll let you know how
>> i get on...
>>
>> Jim
>>
>>
>>
>> On 15/03/12 23:48, James Kosin wrote:
>>> Jim,
>>>
>>> The hashcode is used to lookup and compare items more quickly in Java.
>>> Basically, if the hashcode matches then the Java machine knows there is
>>> a strong possibility two entries are the same. The bad side is the
>>> hashcode for the dictionary entries is based on the entire set of tokens
>>> in the entry. This means Java won't try comparing two items if the
>>> hashcode isn't the same. It is an optimization commonly used.
>>>
>>> In 1.5.3 we fixed a few more issues with the Dictionary to properly
>>> handle the words and case sensitivity. I also made some changes to take
>>> out a small section in the DictionaryNameFinder's find() method that
>>> used an Index created to determine if we should look and add another
>>> word. I may have re-factored this wrong and need to come up with a
>>> better solution.
>>>
>>> We have several possibilities to fix and address this issue. However,
>>> some of them involve possibly making this an N^2 problem again for the
>>> code. I'm trying to avoid that and fix the problem correctly. Maybe I
>>> shouldn't have used hashcode so freely, but, it was how I found the
>>> problem. the hashcode for {"folic", "acid"} is different than that for
>>> {"folic"}... so, the Dictionary doesn't bother comparing the two. One
>>> possibility is to have the entry for {"folic", "acid"} and {"folic"} be
>>> the same... only drawback is we loose resolution in finding specific
>>> names.
>>> Another possible solution would be to keep a max_token_count for the
>>> Dictionary to represent the number of tokens that the
>>> DictionaryNameFinder would try to put together in the find() method...
>>> limiting the greediness to the longest token-list in the dictionary.
>>>
>>> Could you check with 1.5.2 to see if you can find multi-word with/or
>>> without the case sensitivity to verify. If so it limits it to the
>>> changes I made in the trunk.
>>>
>>> Thanks,
>>> James
>>>
>>> On 3/15/2012 5:39 AM, Jim - FooBar(); wrote:
>>>> So the problem is all in the hashcode.............
>>>>
>>>> Does that relate to the question i posted yesterday? I'm a bit
>>>> confused...How is the .hashCode() related with not finding multi-word
>>>> entities? and also, what happened between versions 1.5.2& 1.5.3
>>>> snapshot cos i do remember being able to find multi-word entities at
>>>> some point (i think with 1.5.2)...
>>>>
>>>> Jim
>>>>
>>>>
Re: Dictionary name finder question
Posted by "Jim - FooBar();" <ji...@gmail.com>.
Hi James,
Well, i'm not sure if this is good news but i just tested the dictionary
with version 1.5.2 and it does find multi-word entities just as i
remembered!Of course, case sensitivity does not quite work so it finds
"Folic acid" (exact match) but it doesn't find "folic acid" or
"baclofen" (both uncapitalized)...
Presumably, this is indeed good news as a working version exists yes? I
think we're getting somewhere aren't we?
Jim
On 16/03/12 02:48, James Kosin wrote:
> If I broke something, let me know so I can get original functionality
> even if it involves some minor changes to incorporate.
>
> I'll go ahead with a variable for the dictionary to store the longest
> token count just in case I go that route. The Index was a good idea;
> but, it doesn't carry over the case sensitivity to its members that are
> in the dictionary.
>
> James
>
> On 3/15/2012 8:06 PM, Jim - FooBar(); wrote:
>> aaaaa ok i see what you mean! that makes perfect sense - thanks for
>> being so thorough in explaining...
>>
>> I will check first thing tomorrow morning to see which one was the
>> version that was returning multi-word entities properly...In fact i
>> specifically remember getting back "both "folic acid" and "valproic
>> acid" in the little paragraph i posted....anyway i'll let you know how
>> i get on...
>>
>> Jim
>>
>>
>>
>> On 15/03/12 23:48, James Kosin wrote:
>>> Jim,
>>>
>>> The hashcode is used to lookup and compare items more quickly in Java.
>>> Basically, if the hashcode matches then the Java machine knows there is
>>> a strong possibility two entries are the same. The bad side is the
>>> hashcode for the dictionary entries is based on the entire set of tokens
>>> in the entry. This means Java won't try comparing two items if the
>>> hashcode isn't the same. It is an optimization commonly used.
>>>
>>> In 1.5.3 we fixed a few more issues with the Dictionary to properly
>>> handle the words and case sensitivity. I also made some changes to take
>>> out a small section in the DictionaryNameFinder's find() method that
>>> used an Index created to determine if we should look and add another
>>> word. I may have re-factored this wrong and need to come up with a
>>> better solution.
>>>
>>> We have several possibilities to fix and address this issue. However,
>>> some of them involve possibly making this an N^2 problem again for the
>>> code. I'm trying to avoid that and fix the problem correctly. Maybe I
>>> shouldn't have used hashcode so freely, but, it was how I found the
>>> problem. the hashcode for {"folic", "acid"} is different than that for
>>> {"folic"}... so, the Dictionary doesn't bother comparing the two. One
>>> possibility is to have the entry for {"folic", "acid"} and {"folic"} be
>>> the same... only drawback is we loose resolution in finding specific
>>> names.
>>> Another possible solution would be to keep a max_token_count for the
>>> Dictionary to represent the number of tokens that the
>>> DictionaryNameFinder would try to put together in the find() method...
>>> limiting the greediness to the longest token-list in the dictionary.
>>>
>>> Could you check with 1.5.2 to see if you can find multi-word with/or
>>> without the case sensitivity to verify. If so it limits it to the
>>> changes I made in the trunk.
>>>
>>> Thanks,
>>> James
>>>
>>> On 3/15/2012 5:39 AM, Jim - FooBar(); wrote:
>>>> So the problem is all in the hashcode.............
>>>>
>>>> Does that relate to the question i posted yesterday? I'm a bit
>>>> confused...How is the .hashCode() related with not finding multi-word
>>>> entities? and also, what happened between versions 1.5.2& 1.5.3
>>>> snapshot cos i do remember being able to find multi-word entities at
>>>> some point (i think with 1.5.2)...
>>>>
>>>> Jim
>>>>
>>>>
Re: Dictionary name finder question
Posted by James Kosin <ja...@gmail.com>.
If I broke something, let me know so I can get original functionality
even if it involves some minor changes to incorporate.
I'll go ahead with a variable for the dictionary to store the longest
token count just in case I go that route. The Index was a good idea;
but, it doesn't carry over the case sensitivity to its members that are
in the dictionary.
James
On 3/15/2012 8:06 PM, Jim - FooBar(); wrote:
> aaaaa ok i see what you mean! that makes perfect sense - thanks for
> being so thorough in explaining...
>
> I will check first thing tomorrow morning to see which one was the
> version that was returning multi-word entities properly...In fact i
> specifically remember getting back "both "folic acid" and "valproic
> acid" in the little paragraph i posted....anyway i'll let you know how
> i get on...
>
> Jim
>
>
>
> On 15/03/12 23:48, James Kosin wrote:
>> Jim,
>>
>> The hashcode is used to lookup and compare items more quickly in Java.
>> Basically, if the hashcode matches then the Java machine knows there is
>> a strong possibility two entries are the same. The bad side is the
>> hashcode for the dictionary entries is based on the entire set of tokens
>> in the entry. This means Java won't try comparing two items if the
>> hashcode isn't the same. It is an optimization commonly used.
>>
>> In 1.5.3 we fixed a few more issues with the Dictionary to properly
>> handle the words and case sensitivity. I also made some changes to take
>> out a small section in the DictionaryNameFinder's find() method that
>> used an Index created to determine if we should look and add another
>> word. I may have re-factored this wrong and need to come up with a
>> better solution.
>>
>> We have several possibilities to fix and address this issue. However,
>> some of them involve possibly making this an N^2 problem again for the
>> code. I'm trying to avoid that and fix the problem correctly. Maybe I
>> shouldn't have used hashcode so freely, but, it was how I found the
>> problem. the hashcode for {"folic", "acid"} is different than that for
>> {"folic"}... so, the Dictionary doesn't bother comparing the two. One
>> possibility is to have the entry for {"folic", "acid"} and {"folic"} be
>> the same... only drawback is we loose resolution in finding specific
>> names.
>> Another possible solution would be to keep a max_token_count for the
>> Dictionary to represent the number of tokens that the
>> DictionaryNameFinder would try to put together in the find() method...
>> limiting the greediness to the longest token-list in the dictionary.
>>
>> Could you check with 1.5.2 to see if you can find multi-word with/or
>> without the case sensitivity to verify. If so it limits it to the
>> changes I made in the trunk.
>>
>> Thanks,
>> James
>>
>> On 3/15/2012 5:39 AM, Jim - FooBar(); wrote:
>>> So the problem is all in the hashcode.............
>>>
>>> Does that relate to the question i posted yesterday? I'm a bit
>>> confused...How is the .hashCode() related with not finding multi-word
>>> entities? and also, what happened between versions 1.5.2& 1.5.3
>>> snapshot cos i do remember being able to find multi-word entities at
>>> some point (i think with 1.5.2)...
>>>
>>> Jim
>>>
>>>
>
Re: Dictionary name finder question
Posted by "Jim - FooBar();" <ji...@gmail.com>.
aaaaa ok i see what you mean! that makes perfect sense - thanks for
being so thorough in explaining...
I will check first thing tomorrow morning to see which one was the
version that was returning multi-word entities properly...In fact i
specifically remember getting back "both "folic acid" and "valproic
acid" in the little paragraph i posted....anyway i'll let you know how i
get on...
Jim
On 15/03/12 23:48, James Kosin wrote:
> Jim,
>
> The hashcode is used to lookup and compare items more quickly in Java.
> Basically, if the hashcode matches then the Java machine knows there is
> a strong possibility two entries are the same. The bad side is the
> hashcode for the dictionary entries is based on the entire set of tokens
> in the entry. This means Java won't try comparing two items if the
> hashcode isn't the same. It is an optimization commonly used.
>
> In 1.5.3 we fixed a few more issues with the Dictionary to properly
> handle the words and case sensitivity. I also made some changes to take
> out a small section in the DictionaryNameFinder's find() method that
> used an Index created to determine if we should look and add another
> word. I may have re-factored this wrong and need to come up with a
> better solution.
>
> We have several possibilities to fix and address this issue. However,
> some of them involve possibly making this an N^2 problem again for the
> code. I'm trying to avoid that and fix the problem correctly. Maybe I
> shouldn't have used hashcode so freely, but, it was how I found the
> problem. the hashcode for {"folic", "acid"} is different than that for
> {"folic"}... so, the Dictionary doesn't bother comparing the two. One
> possibility is to have the entry for {"folic", "acid"} and {"folic"} be
> the same... only drawback is we loose resolution in finding specific names.
> Another possible solution would be to keep a max_token_count for the
> Dictionary to represent the number of tokens that the
> DictionaryNameFinder would try to put together in the find() method...
> limiting the greediness to the longest token-list in the dictionary.
>
> Could you check with 1.5.2 to see if you can find multi-word with/or
> without the case sensitivity to verify. If so it limits it to the
> changes I made in the trunk.
>
> Thanks,
> James
>
> On 3/15/2012 5:39 AM, Jim - FooBar(); wrote:
>> So the problem is all in the hashcode.............
>>
>> Does that relate to the question i posted yesterday? I'm a bit
>> confused...How is the .hashCode() related with not finding multi-word
>> entities? and also, what happened between versions 1.5.2& 1.5.3
>> snapshot cos i do remember being able to find multi-word entities at
>> some point (i think with 1.5.2)...
>>
>> Jim
>>
>>
Re: Dictionary name finder question
Posted by James Kosin <ja...@gmail.com>.
Jim,
The hashcode is used to lookup and compare items more quickly in Java.
Basically, if the hashcode matches then the Java machine knows there is
a strong possibility two entries are the same. The bad side is the
hashcode for the dictionary entries is based on the entire set of tokens
in the entry. This means Java won't try comparing two items if the
hashcode isn't the same. It is an optimization commonly used.
In 1.5.3 we fixed a few more issues with the Dictionary to properly
handle the words and case sensitivity. I also made some changes to take
out a small section in the DictionaryNameFinder's find() method that
used an Index created to determine if we should look and add another
word. I may have re-factored this wrong and need to come up with a
better solution.
We have several possibilities to fix and address this issue. However,
some of them involve possibly making this an N^2 problem again for the
code. I'm trying to avoid that and fix the problem correctly. Maybe I
shouldn't have used hashcode so freely, but, it was how I found the
problem. the hashcode for {"folic", "acid"} is different than that for
{"folic"}... so, the Dictionary doesn't bother comparing the two. One
possibility is to have the entry for {"folic", "acid"} and {"folic"} be
the same... only drawback is we loose resolution in finding specific names.
Another possible solution would be to keep a max_token_count for the
Dictionary to represent the number of tokens that the
DictionaryNameFinder would try to put together in the find() method...
limiting the greediness to the longest token-list in the dictionary.
Could you check with 1.5.2 to see if you can find multi-word with/or
without the case sensitivity to verify. If so it limits it to the
changes I made in the trunk.
Thanks,
James
On 3/15/2012 5:39 AM, Jim - FooBar(); wrote:
> So the problem is all in the hashcode.............
>
> Does that relate to the question i posted yesterday? I'm a bit
> confused...How is the .hashCode() related with not finding multi-word
> entities? and also, what happened between versions 1.5.2 & 1.5.3
> snapshot cos i do remember being able to find multi-word entities at
> some point (i think with 1.5.2)...
>
> Jim
>
>
Re: Dictionary name finder question
Posted by "Jim - FooBar();" <ji...@gmail.com>.
On 15/03/12 01:21, James Kosin wrote:
> I'm on the east coast and went to bed at 1:00am yesterday.
> I can't work all night and during the day I do have a day job.
I'm really sorry James i did not mean to sound impatient at all...i was
under the impression Jonas asked for the single entry dictionary so he
could test it himself...i do appreciate your help!
> What is the output of the tokenize model, look like?
I'm not sure i understand this question...what output are you looking
for exactly? Is it the tokenized sentences from the small test paragraph
i posted? I can easily post these if you want...
> So the problem is all in the hashcode.............
Does that relate to the question i posted yesterday? I'm a bit
confused...How is the .hashCode() related with not finding multi-word
entities? and also, what happened between versions 1.5.2 & 1.5.3
snapshot cos i do remember being able to find multi-word entities at
some point (i think with 1.5.2)...
Jim
Re: Dictionary name finder question
Posted by James Kosin <ja...@gmail.com>.
Jim,
I'm on the east coast and went to bed at 1:00am yesterday.
I can't work all night and during the day I do have a day job.
James
On 3/14/2012 10:24 AM, Dimitrios wrote:
> No response in the last 2-3 hours...
>
> Should i assume that someone did confirm my issue using the tiny
> dictionary and the test paragraph i posted before?
>
> Or should i assume that Jame's confirmation that everything works ok
> still stands?
>
> Jim
>
>
> On 14/03/12 11:39, Dimitrios wrote:
>> Forgot to mention that after find() returns the Spans i'm doing
>> Span.SpansToStrings(span-array token-array) to get the human readable
>> array of strings....
>>
>> Jim
>>
>>
>> On 14/03/12 11:34, Dimitrios wrote:
>>> On 14/03/12 11:23, Jörn Kottmann wrote:
>>>> Can you re-produce your issue with a dictionary which only contains
>>>> a single entry?
>>>
>>> Yes i can indeed re-produce the issue with the following dictionary:
>>> --------------------------------------------------------------------------------------------
>>>
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <dictionary case_sensitive="false">
>>> <entry>
>>> <token>Folic</token>
>>> <token>acid</token>
>>> </entry>
>>> <entry>
>>> <token>Baclofen</token>
>>> </entry>
>>> </dictionary>
>>> --------------------------------------------------------------------------------------------
>>>
>>>
>>> The small paragraph i'm using for testing is this:
>>>
>>> "Folic acid is one variable, but other factors remain.
>>> Studies suggest that substances active at the GABA receptor may
>>> produce NTDs.
>>> To test this hypothesis pregnant rats were exposed to either the
>>> GABA a agonist muscimol (1, 2 or 4 mg/kg), the GABA a antagonist
>>> bicuculline (.5, 1, or 2 mg/kg), the GABA b agonist baclofen (15,
>>> 30, 60 mg/kg), or the GABA b antagonist hydroxysaclofen (1, 3, or 5
>>> mg/kg) during neural tube formation.
>>> Normal saline was used as a control and valproic acid (600 mg/kg) as
>>> a positive control."
>>>
>>>
>>> The dictionary finds "baclofen" but it does not find "Folic acid"!
>>> The workflow is as follows:
>>>
>>> 1. get-sentences
>>> 2. tokenize -sentences
>>> 3. call dictionary name finder ".find()" method with an array of
>>> srings (tokens of a single sentence)
>>>
>>> Jim
>>>
>>
>
>
Re: Dictionary name finder question
Posted by Dimitrios <ji...@gmail.com>.
No response in the last 2-3 hours...
Should i assume that someone did confirm my issue using the tiny
dictionary and the test paragraph i posted before?
Or should i assume that Jame's confirmation that everything works ok
still stands?
Jim
On 14/03/12 11:39, Dimitrios wrote:
> Forgot to mention that after find() returns the Spans i'm doing
> Span.SpansToStrings(span-array token-array) to get the human readable
> array of strings....
>
> Jim
>
>
> On 14/03/12 11:34, Dimitrios wrote:
>> On 14/03/12 11:23, Jörn Kottmann wrote:
>>> Can you re-produce your issue with a dictionary which only contains
>>> a single entry?
>>
>> Yes i can indeed re-produce the issue with the following dictionary:
>> --------------------------------------------------------------------------------------------
>> <?xml version="1.0" encoding="UTF-8"?>
>> <dictionary case_sensitive="false">
>> <entry>
>> <token>Folic</token>
>> <token>acid</token>
>> </entry>
>> <entry>
>> <token>Baclofen</token>
>> </entry>
>> </dictionary>
>> --------------------------------------------------------------------------------------------
>>
>> The small paragraph i'm using for testing is this:
>>
>> "Folic acid is one variable, but other factors remain.
>> Studies suggest that substances active at the GABA receptor may
>> produce NTDs.
>> To test this hypothesis pregnant rats were exposed to either the GABA
>> a agonist muscimol (1, 2 or 4 mg/kg), the GABA a antagonist
>> bicuculline (.5, 1, or 2 mg/kg), the GABA b agonist baclofen (15, 30,
>> 60 mg/kg), or the GABA b antagonist hydroxysaclofen (1, 3, or 5
>> mg/kg) during neural tube formation.
>> Normal saline was used as a control and valproic acid (600 mg/kg) as
>> a positive control."
>>
>>
>> The dictionary finds "baclofen" but it does not find "Folic acid"!
>> The workflow is as follows:
>>
>> 1. get-sentences
>> 2. tokenize -sentences
>> 3. call dictionary name finder ".find()" method with an array of
>> srings (tokens of a single sentence)
>>
>> Jim
>>
>
Re: Dictionary name finder question
Posted by Dimitrios <ji...@gmail.com>.
Forgot to mention that after find() returns the Spans i'm doing
Span.SpansToStrings(span-array token-array) to get the human readable
array of strings....
Jim
On 14/03/12 11:34, Dimitrios wrote:
> On 14/03/12 11:23, Jörn Kottmann wrote:
>> Can you re-produce your issue with a dictionary which only contains a
>> single entry?
>
> Yes i can indeed re-produce the issue with the following dictionary:
> --------------------------------------------------------------------------------------------
> <?xml version="1.0" encoding="UTF-8"?>
> <dictionary case_sensitive="false">
> <entry>
> <token>Folic</token>
> <token>acid</token>
> </entry>
> <entry>
> <token>Baclofen</token>
> </entry>
> </dictionary>
> --------------------------------------------------------------------------------------------
>
> The small paragraph i'm using for testing is this:
>
> "Folic acid is one variable, but other factors remain.
> Studies suggest that substances active at the GABA receptor may
> produce NTDs.
> To test this hypothesis pregnant rats were exposed to either the GABA
> a agonist muscimol (1, 2 or 4 mg/kg), the GABA a antagonist
> bicuculline (.5, 1, or 2 mg/kg), the GABA b agonist baclofen (15, 30,
> 60 mg/kg), or the GABA b antagonist hydroxysaclofen (1, 3, or 5 mg/kg)
> during neural tube formation.
> Normal saline was used as a control and valproic acid (600 mg/kg) as a
> positive control."
>
>
> The dictionary finds "baclofen" but it does not find "Folic acid"! The
> workflow is as follows:
>
> 1. get-sentences
> 2. tokenize -sentences
> 3. call dictionary name finder ".find()" method with an array of
> srings (tokens of a single sentence)
>
> Jim
>
Re: Dictionary name finder question
Posted by Dimitrios <ji...@gmail.com>.
On 14/03/12 11:23, Jörn Kottmann wrote:
> Can you re-produce your issue with a dictionary which only contains a
> single entry?
Yes i can indeed re-produce the issue with the following dictionary:
--------------------------------------------------------------------------------------------
<?xml version="1.0" encoding="UTF-8"?>
<dictionary case_sensitive="false">
<entry>
<token>Folic</token>
<token>acid</token>
</entry>
<entry>
<token>Baclofen</token>
</entry>
</dictionary>
--------------------------------------------------------------------------------------------
The small paragraph i'm using for testing is this:
"Folic acid is one variable, but other factors remain.
Studies suggest that substances active at the GABA receptor may produce
NTDs.
To test this hypothesis pregnant rats were exposed to either the GABA a
agonist muscimol (1, 2 or 4 mg/kg), the GABA a antagonist bicuculline
(.5, 1, or 2 mg/kg), the GABA b agonist baclofen (15, 30, 60 mg/kg), or
the GABA b antagonist hydroxysaclofen (1, 3, or 5 mg/kg) during neural
tube formation.
Normal saline was used as a control and valproic acid (600 mg/kg) as a
positive control."
The dictionary finds "baclofen" but it does not find "Folic acid"! The
workflow is as follows:
1. get-sentences
2. tokenize -sentences
3. call dictionary name finder ".find()" method with an array of srings
(tokens of a single sentence)
Jim
Re: Dictionary name finder question
Posted by Jörn Kottmann <ko...@gmail.com>.
On 03/14/2012 12:07 PM, Dimitrios wrote:
>
> Now, this is truly bizzare!!! I am 200% sure that a couple of weeks
> ago, after your response about using the command-line tool to produce
> my dictionary, all my dictionary problems disappeared !!! I
> specifically remember not being able to find multi-word entities with
> the old dictionary which encoded 2 tokens as one. But now i have
> produced a correct dictionary and i am indeed passing it tokenized
> sentences - but with no success!!! it only finds single word
> entities!!! I do understand this is probably not an openNLP issue
> simply because i do remember sorting it!!! In any case thanks for
> confirming...
>
> Debugged every single line of my code related with the dictionary and
> nothing! - this is TRULY BEYOND MY UNDERSTANDING!!! the funny thing is
> i only re-run the dictionary name finder in order to check if i could
> use it with the evaluator!!! Unbelievable coincidence don't you think?
Can you re-produce your issue with a dictionary which only contains a
single entry?
Jörn
Re: Dictionary name finder question
Posted by Dimitrios <ji...@gmail.com>.
Hi James,
Now, this is truly bizzare!!! I am 200% sure that a couple of weeks ago,
after your response about using the command-line tool to produce my
dictionary, all my dictionary problems disappeared !!! I specifically
remember not being able to find multi-word entities with the old
dictionary which encoded 2 tokens as one. But now i have produced a
correct dictionary and i am indeed passing it tokenized sentences - but
with no success!!! it only finds single word entities!!! I do understand
this is probably not an openNLP issue simply because i do remember
sorting it!!! In any case thanks for confirming...
Debugged every single line of my code related with the dictionary and
nothing! - this is TRULY BEYOND MY UNDERSTANDING!!! the funny thing is i
only re-run the dictionary name finder in order to check if i could use
it with the evaluator!!! Unbelievable coincidence don't you think?
Jim
p.s: can i see the test sources you mentioned without checking out the
repo? maybe online somewhere?
On 14/03/12 02:47, James Kosin wrote:
> Jim,
>
> Check to be sure you are still running the text through the tokenizer
> before using the Dictionary Name Finder? Just a thought.
>
> James
>
> On 3/13/2012 7:09 PM, Jim - FooBar(); wrote:
>> Could you please confirm that the dictionary name-finder can identify
>> multi-word entities after the recent case-sensitivity changes that
>> were made? All of a sudden my dictionary stopped recognising
>> multi-word entities even though they are encoded correctly as 2 in
>> the .xml dictionary file. I thought i had this sorted - my old problem
>> had to do with how i was creating the dictionary but now i 'm using
>> the cmd-line tool...
>>
>> Jim
>>
>> p.s: case sensitivity is indeed done correctly now!
Re: Dictionary name finder question
Posted by James Kosin <ja...@gmail.com>.
Jim,
Check to be sure you are still running the text through the tokenizer
before using the Dictionary Name Finder? Just a thought.
James
On 3/13/2012 7:09 PM, Jim - FooBar(); wrote:
> Could you please confirm that the dictionary name-finder can identify
> multi-word entities after the recent case-sensitivity changes that
> were made? All of a sudden my dictionary stopped recognising
> multi-word entities even though they are encoded correctly as 2 in
> the .xml dictionary file. I thought i had this sorted - my old problem
> had to do with how i was creating the dictionary but now i 'm using
> the cmd-line tool...
>
> Jim
>
> p.s: case sensitivity is indeed done correctly now!