You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@opennlp.apache.org by "James Kosin (Created) (JIRA)" <ji...@apache.org> on 2012/03/15 04:47:53 UTC
[jira] [Created] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
DictionaryNameFinder has HASHing issues
---------------------------------------
Key: OPENNLP-471
URL: https://issues.apache.org/jira/browse/OPENNLP-471
Project: OpenNLP
Issue Type: Bug
Components: Name Finder
Reporter: James Kosin
The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
Thanks to Jim for pushing this and to my debugging skills to find.
Two possiblilites come to mind:
1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "Joern Kottmann (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234202#comment-13234202 ]
Joern Kottmann commented on OPENNLP-471:
----------------------------------------
Thanks for the patch, we applied it. Please don't use tabs to indent, we always indent with two spaces.
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
> Attachments: DictionaryNameFinder.java_Hyosup_Shim.patch
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "James Kosin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231808#comment-13231808 ]
James Kosin commented on OPENNLP-471:
-------------------------------------
I've added code to expand the dictionary search as long as the buid token list is smaller than the dictionary max value.
This is done by adding 2 integer values to the Dictionary class to hold the longest and shortest token counts added to the dictionary. We currently use the max count; but, theoretically it can be easily expanded to use the min as well to prevent us from looking for shorter dictionary entries when we have none.
The added code to the find() shouldn't turn the search into an N^2 problem; but, should fix the issue of finding dictionary elements that are longer that have no shorter counterparts in the dictionary.
NOTE: I went with my last comments above as a solution; because it currently makes the most sence. If anyone has a better idea we can certainly entertain it in this JIRA.
Another possible option would be to bring back the Index.... but, we would have to make the StringList export the case sensitivity somehow to the Index so it can work correctly.
I can work on that as another option; since the Index allowed us to keep adding tokens as long as the tokens where in the Index. The problem was that case sensitivity issues need to be worked out there.
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Labels: dictionary, namefinder
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "James Kosin (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Kosin reopened OPENNLP-471:
---------------------------------
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "James Kosin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Kosin resolved OPENNLP-471.
---------------------------------
Resolution: Fixed
I believe this issue is fixed for now until we re-work the classes, to re-use the Index java class in our tools.
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
> Attachments: DictionaryNameFinder.java_Hyosup_Shim.patch
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "Hyosup Shim (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233585#comment-13233585 ]
Hyosup Shim commented on OPENNLP-471:
-------------------------------------
I made some changes (for names of variables) to make easier to read.
Please review my patch.
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "William Colen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232094#comment-13232094 ]
William Colen commented on OPENNLP-471:
---------------------------------------
James, is it working? I created a unit test for issue OPENNLP-477. The test creates a dictionary from a corpus and evaluate a DictionaryNameFinder created with this dictionary using the same corpus. The F-Measure result is not 100%, but should be, shouldn't it?
Thanks
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "William Colen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233147#comment-13233147 ]
William Colen commented on OPENNLP-471:
---------------------------------------
Again... sorry :D
I could reproduce the issue with DictionaryNameFinderTest
1) add an extra item to the dictionary:
StringList williams = new StringList(new String[]{"Williams"});
mDictionary.put(williams);
2) if you run the tests now some will fail...
To fix it you have to change the DictionaryNameFinder:
...
if (foundName != null) {
foundNames.add(foundName);
// skip the name we just found
startToken += foundName.length() - 1;
}
...
Now the tests will pass.
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "James Kosin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229866#comment-13229866 ]
James Kosin commented on OPENNLP-471:
-------------------------------------
Another possibility is to set an integer value that contains the longest size (in tokens) for the longest entry in the dictionary.
find() could use this to a) allow it to expand the search in the dictionary for more tokens and b) if only 1 token in all entries allows us to stop trying to find more tokens.
It may be the best compromize under the circumstances.
I'll let the group decide....
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Labels: dictionary, namefinder
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "James Kosin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Kosin closed OPENNLP-471.
-------------------------------
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
> Attachments: DictionaryNameFinder.java_Hyosup_Shim.patch
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "James Kosin (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Kosin resolved OPENNLP-471.
---------------------------------
Resolution: Fixed
Fix Version/s: tools-1.5.3
I'm setting this issue as resolved ... but, do comment here . We can re-open and address other issues.
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "William Colen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231960#comment-13231960 ]
William Colen commented on OPENNLP-471:
---------------------------------------
One way I see to make it faster would be to convert it to a state machine problem. We would need to create a new data structure from the dictionary, but it would let us know if we can keep trying more tokens or not. We would only advance tokens if we are in a known state.
Another alternative would be to add a method to Dictionary that can do partial match. Maybe returning an enumerator with "partial match", "complete match", "no match". Again we would be able to keep trying incrementing the token list only if we have a partial match.
But I don't really know this part of the code. Maybe none of my alternatives can be implemented :)
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "James Kosin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232973#comment-13232973 ]
James Kosin commented on OPENNLP-471:
-------------------------------------
I think maybe William got a little confused.
What should be happening is the Longest matching span is returned; however, the F1 score is based on successful and unsuccessful matches on what should be new data (test data) apart from the training data... which is probably why the low score. The recall is how well it matched the ones it did find, and 100% sounds right Since all the data in the dictionary should be found.
I'll have to verify this.
James
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "Joern Kottmann (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232516#comment-13232516 ]
Joern Kottmann commented on OPENNLP-471:
----------------------------------------
Isn't it now always searching for the longest matching span?
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "Hyosup Shim (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyosup Shim updated OPENNLP-471:
--------------------------------
Attachment: DictionaryNameFinder.java_Hyosup_Shim.patch
I made some changes (for names of variables) to make it easier to read.
Please review my patch.
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
> Attachments: DictionaryNameFinder.java_Hyosup_Shim.patch
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by James Kosin <ja...@gmail.com>.
Well then +1 on catching this if possible. I'm sure it isn't the first
time. But, this really should be handled when tokenized properly I guess.
I was trying to figure out the difference between using the actual
offsets and using the length. Both are equivalent... or should be in my
eyes.
On 3/19/2012 10:51 PM, william.colen@gmail.com wrote:
> Looks like we was checking it at the same time.
>
> Yes, because of this log I got confused. Actually this error is caused by
> an issue in the corpus. The is not correctly annotated:
>
> ... to visit us, <START> Luise <END> and <START> George Bauer <END>.
> The <END> was not catch because of the '.' and the corpus parser got
> confused.
>
>
> On Mon, Mar 19, 2012 at 11:41 PM, James Kosin <ja...@gmail.com> wrote:
>
>> William,
>>
>> I found a problem with the longest match... I wasn't jumping over the
>> match after adding it to the spans. This could cause George Bauer and
>> Bauer to return as two entries if Bauer was in the dictionary.
>>
>> I'm a little confused on an output I'm getting now:
>> -----
>> Running opennlp.tools.namefind.DictionaryNameFinderEvaluatorTest
>> Expected: {
>> Since then, our guests have to ring at Veilchenstra§e 11 if they want to
>> visit us, <START:default> Luise <END> and George Bauer <END>.}
>> Predicted: {
>> Since then, our guests have to ring at Veilchenstra§e 11 if they want to
>> visit us, <START:default> Luise <END> and <START:default> George <END>
>> Bauer <END>.}
>> False positives: {
>> [George]
>> } False negatives: {
>> []
>> }
>> ----
>> after adding a line assertTrue(fmeasure.getFmeasure() == 1); to the test
>> file...?
>>
>>
>>
>>
>> On 3/19/2012 10:11 PM, William Colen (Commented) (JIRA) wrote:
>>> [
>> https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233124#comment-13233124]
>>> William Colen commented on OPENNLP-471:
>>> ---------------------------------------
>>>
>>> Ops... sorry ! I am debugging the code and might have found the real
>> reason for the previous output. I will investigate it further.
>>>> DictionaryNameFinder has HASHing issues
>>>> ---------------------------------------
>>>>
>>>> Key: OPENNLP-471
>>>> URL: https://issues.apache.org/jira/browse/OPENNLP-471
>>>> Project: OpenNLP
>>>> Issue Type: Bug
>>>> Components: Name Finder
>>>> Reporter: James Kosin
>>>> Assignee: James Kosin
>>>> Labels: dictionary, namefinder
>>>> Fix For: tools-1.5.3
>>>>
>>>>
>>>> The DictionaryNameFinder has issues finding multi-token names when the
>> dictionary is searched a token at a time by the find() method. If, the
>> dictionary doesn't have a single (or shorter) token match available in the
>> dictionary.
>>>> Having a dictionary with {"folic", "acid"} without an entry for
>> {"folic"} will cause the find() method to totally skip the fact there is a
>> longer match possible.
>>>> Thanks to Jim for pushing this and to my debugging skills to find.
>>>> Two possiblilites come to mind:
>>>> 1) I don't really like, is we turn it into a larger problem by trying
>> longer matches when shorter ones don't match. Unfortunately, this turns
>> quickly into a race to see who can wait longer.
>>>> 2) A way of returning a possible match that may need exploring, or a
>> look-ahead type system to say we don't match "folic" but if you have "acid"
>> after "folic" we have a match for that in the dictionary.
>>>> 3) Leave it as is and modify the dictionary to add shorter terms to
>> the dictionary... maybe marking as not-a-valid entry so we can know we need
>> a longer match.
>>> --
>>> This message is automatically generated by JIRA.
>>> If you think it was sent incorrectly, please contact your JIRA
>> administrators:
>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>>> For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>>>
>>
Re: [jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by James Kosin <ja...@gmail.com>.
It is catching both names correctly now in the sentence.
I've got it all fixed now and in SVN.
Could you review.
Thanks
James
On 3/19/2012 11:25 PM, william.colen@gmail.com wrote:
> I am not sure wich is the best. We just should make sure that it catches
> two following names correctly:
>
> <START> Luise <END> <START> George Bauer <END>
>
> On Tue, Mar 20, 2012 at 12:07 AM, James Kosin <ja...@gmail.com> wrote:
>
>> Thanks....
>>
>> We have some collateral damage for the NameSampleDataStreamTest that is
>> failing....
>> I'm trying to fix all this locally as well. yes...
>> Although your idea looks better than using getEnd() method I was using
>> and checked in... I may change it.
>>
>> James
>>
>> On 3/19/2012 10:51 PM, william.colen@gmail.com wrote:
>>> Looks like we was checking it at the same time.
>>>
>>> Yes, because of this log I got confused. Actually this error is caused by
>>> an issue in the corpus. The is not correctly annotated:
>>>
>>> ... to visit us, <START> Luise <END> and <START> George Bauer <END>.
>>> The <END> was not catch because of the '.' and the corpus parser got
>>> confused.
>>>
>>>
>>> On Mon, Mar 19, 2012 at 11:41 PM, James Kosin <ja...@gmail.com>
>> wrote:
>>>> William,
>>>>
>>>> I found a problem with the longest match... I wasn't jumping over the
>>>> match after adding it to the spans. This could cause George Bauer and
>>>> Bauer to return as two entries if Bauer was in the dictionary.
>>>>
>>>> I'm a little confused on an output I'm getting now:
>>>> -----
>>>> Running opennlp.tools.namefind.DictionaryNameFinderEvaluatorTest
>>>> Expected: {
>>>> Since then, our guests have to ring at Veilchenstra呈 11 if they want to
>>>> visit us, <START:default> Luise <END> and George Bauer <END>.}
>>>> Predicted: {
>>>> Since then, our guests have to ring at Veilchenstra呈 11 if they want to
>>>> visit us, <START:default> Luise <END> and <START:default> George <END>
>>>> Bauer <END>.}
>>>> False positives: {
>>>> [George]
>>>> } False negatives: {
>>>> []
>>>> }
>>>> ----
>>>> after adding a line assertTrue(fmeasure.getFmeasure() == 1); to the test
>>>> file...?
>>>>
>>>>
>>>>
>>>>
>>>> On 3/19/2012 10:11 PM, William Colen (Commented) (JIRA) wrote:
>>>>> [
>> https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233124#comment-13233124
>> ]
>>>>> William Colen commented on OPENNLP-471:
>>>>> ---------------------------------------
>>>>>
>>>>> Ops... sorry ! I am debugging the code and might have found the real
>>>> reason for the previous output. I will investigate it further.
>>>>>> DictionaryNameFinder has HASHing issues
>>>>>> ---------------------------------------
>>>>>>
>>>>>> Key: OPENNLP-471
>>>>>> URL:
>> https://issues.apache.org/jira/browse/OPENNLP-471
>>>>>> Project: OpenNLP
>>>>>> Issue Type: Bug
>>>>>> Components: Name Finder
>>>>>> Reporter: James Kosin
>>>>>> Assignee: James Kosin
>>>>>> Labels: dictionary, namefinder
>>>>>> Fix For: tools-1.5.3
>>>>>>
>>>>>>
>>>>>> The DictionaryNameFinder has issues finding multi-token names when the
>>>> dictionary is searched a token at a time by the find() method. If, the
>>>> dictionary doesn't have a single (or shorter) token match available in
>> the
>>>> dictionary.
>>>>>> Having a dictionary with {"folic", "acid"} without an entry for
>>>> {"folic"} will cause the find() method to totally skip the fact there
>> is a
>>>> longer match possible.
>>>>>> Thanks to Jim for pushing this and to my debugging skills to find.
>>>>>> Two possiblilites come to mind:
>>>>>> 1) I don't really like, is we turn it into a larger problem by trying
>>>> longer matches when shorter ones don't match. Unfortunately, this turns
>>>> quickly into a race to see who can wait longer.
>>>>>> 2) A way of returning a possible match that may need exploring, or a
>>>> look-ahead type system to say we don't match "folic" but if you have
>> "acid"
>>>> after "folic" we have a match for that in the dictionary.
>>>>>> 3) Leave it as is and modify the dictionary to add shorter terms to
>>>> the dictionary... maybe marking as not-a-valid entry so we can know we
>> need
>>>> a longer match.
>>>>> --
>>>>> This message is automatically generated by JIRA.
>>>>> If you think it was sent incorrectly, please contact your JIRA
>>>> administrators:
>>>>
>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>>>>> For more information on JIRA, see:
>>>> http://www.atlassian.com/software/jira
>>
Re: [jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing issues
Posted by "william.colen@gmail.com" <wi...@gmail.com>.
I am not sure wich is the best. We just should make sure that it catches
two following names correctly:
<START> Luise <END> <START> George Bauer <END>
On Tue, Mar 20, 2012 at 12:07 AM, James Kosin <ja...@gmail.com> wrote:
> Thanks....
>
> We have some collateral damage for the NameSampleDataStreamTest that is
> failing....
> I'm trying to fix all this locally as well. yes...
> Although your idea looks better than using getEnd() method I was using
> and checked in... I may change it.
>
> James
>
> On 3/19/2012 10:51 PM, william.colen@gmail.com wrote:
> > Looks like we was checking it at the same time.
> >
> > Yes, because of this log I got confused. Actually this error is caused by
> > an issue in the corpus. The is not correctly annotated:
> >
> > ... to visit us, <START> Luise <END> and <START> George Bauer <END>.
> > The <END> was not catch because of the '.' and the corpus parser got
> > confused.
> >
> >
> > On Mon, Mar 19, 2012 at 11:41 PM, James Kosin <ja...@gmail.com>
> wrote:
> >
> >> William,
> >>
> >> I found a problem with the longest match... I wasn't jumping over the
> >> match after adding it to the spans. This could cause George Bauer and
> >> Bauer to return as two entries if Bauer was in the dictionary.
> >>
> >> I'm a little confused on an output I'm getting now:
> >> -----
> >> Running opennlp.tools.namefind.DictionaryNameFinderEvaluatorTest
> >> Expected: {
> >> Since then, our guests have to ring at Veilchenstra呈 11 if they want to
> >> visit us, <START:default> Luise <END> and George Bauer <END>.}
> >> Predicted: {
> >> Since then, our guests have to ring at Veilchenstra呈 11 if they want to
> >> visit us, <START:default> Luise <END> and <START:default> George <END>
> >> Bauer <END>.}
> >> False positives: {
> >> [George]
> >> } False negatives: {
> >> []
> >> }
> >> ----
> >> after adding a line assertTrue(fmeasure.getFmeasure() == 1); to the test
> >> file...?
> >>
> >>
> >>
> >>
> >> On 3/19/2012 10:11 PM, William Colen (Commented) (JIRA) wrote:
> >>> [
> >>
> https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233124#comment-13233124
> ]
> >>> William Colen commented on OPENNLP-471:
> >>> ---------------------------------------
> >>>
> >>> Ops... sorry ! I am debugging the code and might have found the real
> >> reason for the previous output. I will investigate it further.
> >>>> DictionaryNameFinder has HASHing issues
> >>>> ---------------------------------------
> >>>>
> >>>> Key: OPENNLP-471
> >>>> URL:
> https://issues.apache.org/jira/browse/OPENNLP-471
> >>>> Project: OpenNLP
> >>>> Issue Type: Bug
> >>>> Components: Name Finder
> >>>> Reporter: James Kosin
> >>>> Assignee: James Kosin
> >>>> Labels: dictionary, namefinder
> >>>> Fix For: tools-1.5.3
> >>>>
> >>>>
> >>>> The DictionaryNameFinder has issues finding multi-token names when the
> >> dictionary is searched a token at a time by the find() method. If, the
> >> dictionary doesn't have a single (or shorter) token match available in
> the
> >> dictionary.
> >>>> Having a dictionary with {"folic", "acid"} without an entry for
> >> {"folic"} will cause the find() method to totally skip the fact there
> is a
> >> longer match possible.
> >>>> Thanks to Jim for pushing this and to my debugging skills to find.
> >>>> Two possiblilites come to mind:
> >>>> 1) I don't really like, is we turn it into a larger problem by trying
> >> longer matches when shorter ones don't match. Unfortunately, this turns
> >> quickly into a race to see who can wait longer.
> >>>> 2) A way of returning a possible match that may need exploring, or a
> >> look-ahead type system to say we don't match "folic" but if you have
> "acid"
> >> after "folic" we have a match for that in the dictionary.
> >>>> 3) Leave it as is and modify the dictionary to add shorter terms to
> >> the dictionary... maybe marking as not-a-valid entry so we can know we
> need
> >> a longer match.
> >>> --
> >>> This message is automatically generated by JIRA.
> >>> If you think it was sent incorrectly, please contact your JIRA
> >> administrators:
> >>
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> >>> For more information on JIRA, see:
> >> http://www.atlassian.com/software/jira
> >>>
> >>
>
>
Re: [jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by James Kosin <ja...@gmail.com>.
Thanks....
We have some collateral damage for the NameSampleDataStreamTest that is
failing....
I'm trying to fix all this locally as well. yes...
Although your idea looks better than using getEnd() method I was using
and checked in... I may change it.
James
On 3/19/2012 10:51 PM, william.colen@gmail.com wrote:
> Looks like we was checking it at the same time.
>
> Yes, because of this log I got confused. Actually this error is caused by
> an issue in the corpus. The is not correctly annotated:
>
> ... to visit us, <START> Luise <END> and <START> George Bauer <END>.
> The <END> was not catch because of the '.' and the corpus parser got
> confused.
>
>
> On Mon, Mar 19, 2012 at 11:41 PM, James Kosin <ja...@gmail.com> wrote:
>
>> William,
>>
>> I found a problem with the longest match... I wasn't jumping over the
>> match after adding it to the spans. This could cause George Bauer and
>> Bauer to return as two entries if Bauer was in the dictionary.
>>
>> I'm a little confused on an output I'm getting now:
>> -----
>> Running opennlp.tools.namefind.DictionaryNameFinderEvaluatorTest
>> Expected: {
>> Since then, our guests have to ring at Veilchenstra§e 11 if they want to
>> visit us, <START:default> Luise <END> and George Bauer <END>.}
>> Predicted: {
>> Since then, our guests have to ring at Veilchenstra§e 11 if they want to
>> visit us, <START:default> Luise <END> and <START:default> George <END>
>> Bauer <END>.}
>> False positives: {
>> [George]
>> } False negatives: {
>> []
>> }
>> ----
>> after adding a line assertTrue(fmeasure.getFmeasure() == 1); to the test
>> file...?
>>
>>
>>
>>
>> On 3/19/2012 10:11 PM, William Colen (Commented) (JIRA) wrote:
>>> [
>> https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233124#comment-13233124]
>>> William Colen commented on OPENNLP-471:
>>> ---------------------------------------
>>>
>>> Ops... sorry ! I am debugging the code and might have found the real
>> reason for the previous output. I will investigate it further.
>>>> DictionaryNameFinder has HASHing issues
>>>> ---------------------------------------
>>>>
>>>> Key: OPENNLP-471
>>>> URL: https://issues.apache.org/jira/browse/OPENNLP-471
>>>> Project: OpenNLP
>>>> Issue Type: Bug
>>>> Components: Name Finder
>>>> Reporter: James Kosin
>>>> Assignee: James Kosin
>>>> Labels: dictionary, namefinder
>>>> Fix For: tools-1.5.3
>>>>
>>>>
>>>> The DictionaryNameFinder has issues finding multi-token names when the
>> dictionary is searched a token at a time by the find() method. If, the
>> dictionary doesn't have a single (or shorter) token match available in the
>> dictionary.
>>>> Having a dictionary with {"folic", "acid"} without an entry for
>> {"folic"} will cause the find() method to totally skip the fact there is a
>> longer match possible.
>>>> Thanks to Jim for pushing this and to my debugging skills to find.
>>>> Two possiblilites come to mind:
>>>> 1) I don't really like, is we turn it into a larger problem by trying
>> longer matches when shorter ones don't match. Unfortunately, this turns
>> quickly into a race to see who can wait longer.
>>>> 2) A way of returning a possible match that may need exploring, or a
>> look-ahead type system to say we don't match "folic" but if you have "acid"
>> after "folic" we have a match for that in the dictionary.
>>>> 3) Leave it as is and modify the dictionary to add shorter terms to
>> the dictionary... maybe marking as not-a-valid entry so we can know we need
>> a longer match.
>>> --
>>> This message is automatically generated by JIRA.
>>> If you think it was sent incorrectly, please contact your JIRA
>> administrators:
>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>>> For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>>>
>>
Re: [jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing issues
Posted by "william.colen@gmail.com" <wi...@gmail.com>.
Looks like we was checking it at the same time.
Yes, because of this log I got confused. Actually this error is caused by
an issue in the corpus. The is not correctly annotated:
... to visit us, <START> Luise <END> and <START> George Bauer <END>.
The <END> was not catch because of the '.' and the corpus parser got
confused.
On Mon, Mar 19, 2012 at 11:41 PM, James Kosin <ja...@gmail.com> wrote:
> William,
>
> I found a problem with the longest match... I wasn't jumping over the
> match after adding it to the spans. This could cause George Bauer and
> Bauer to return as two entries if Bauer was in the dictionary.
>
> I'm a little confused on an output I'm getting now:
> -----
> Running opennlp.tools.namefind.DictionaryNameFinderEvaluatorTest
> Expected: {
> Since then, our guests have to ring at Veilchenstra§e 11 if they want to
> visit us, <START:default> Luise <END> and George Bauer <END>.}
> Predicted: {
> Since then, our guests have to ring at Veilchenstra§e 11 if they want to
> visit us, <START:default> Luise <END> and <START:default> George <END>
> Bauer <END>.}
> False positives: {
> [George]
> } False negatives: {
> []
> }
> ----
> after adding a line assertTrue(fmeasure.getFmeasure() == 1); to the test
> file...?
>
>
>
>
> On 3/19/2012 10:11 PM, William Colen (Commented) (JIRA) wrote:
> > [
> https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233124#comment-13233124]
> >
> > William Colen commented on OPENNLP-471:
> > ---------------------------------------
> >
> > Ops... sorry ! I am debugging the code and might have found the real
> reason for the previous output. I will investigate it further.
> >
> >> DictionaryNameFinder has HASHing issues
> >> ---------------------------------------
> >>
> >> Key: OPENNLP-471
> >> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> >> Project: OpenNLP
> >> Issue Type: Bug
> >> Components: Name Finder
> >> Reporter: James Kosin
> >> Assignee: James Kosin
> >> Labels: dictionary, namefinder
> >> Fix For: tools-1.5.3
> >>
> >>
> >> The DictionaryNameFinder has issues finding multi-token names when the
> dictionary is searched a token at a time by the find() method. If, the
> dictionary doesn't have a single (or shorter) token match available in the
> dictionary.
> >> Having a dictionary with {"folic", "acid"} without an entry for
> {"folic"} will cause the find() method to totally skip the fact there is a
> longer match possible.
> >> Thanks to Jim for pushing this and to my debugging skills to find.
> >> Two possiblilites come to mind:
> >> 1) I don't really like, is we turn it into a larger problem by trying
> longer matches when shorter ones don't match. Unfortunately, this turns
> quickly into a race to see who can wait longer.
> >> 2) A way of returning a possible match that may need exploring, or a
> look-ahead type system to say we don't match "folic" but if you have "acid"
> after "folic" we have a match for that in the dictionary.
> >> 3) Leave it as is and modify the dictionary to add shorter terms to
> the dictionary... maybe marking as not-a-valid entry so we can know we need
> a longer match.
> > --
> > This message is automatically generated by JIRA.
> > If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
> >
>
>
Re: [jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by James Kosin <ja...@gmail.com>.
William,
I found a problem with the longest match... I wasn't jumping over the
match after adding it to the spans. This could cause George Bauer and
Bauer to return as two entries if Bauer was in the dictionary.
I'm a little confused on an output I'm getting now:
-----
Running opennlp.tools.namefind.DictionaryNameFinderEvaluatorTest
Expected: {
Since then, our guests have to ring at Veilchenstra§e 11 if they want to
visit us, <START:default> Luise <END> and George Bauer <END>.}
Predicted: {
Since then, our guests have to ring at Veilchenstra§e 11 if they want to
visit us, <START:default> Luise <END> and <START:default> George <END>
Bauer <END>.}
False positives: {
[George]
} False negatives: {
[]
}
----
after adding a line assertTrue(fmeasure.getFmeasure() == 1); to the test
file...?
On 3/19/2012 10:11 PM, William Colen (Commented) (JIRA) wrote:
> [ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233124#comment-13233124 ]
>
> William Colen commented on OPENNLP-471:
> ---------------------------------------
>
> Ops... sorry ! I am debugging the code and might have found the real reason for the previous output. I will investigate it further.
>
>> DictionaryNameFinder has HASHing issues
>> ---------------------------------------
>>
>> Key: OPENNLP-471
>> URL: https://issues.apache.org/jira/browse/OPENNLP-471
>> Project: OpenNLP
>> Issue Type: Bug
>> Components: Name Finder
>> Reporter: James Kosin
>> Assignee: James Kosin
>> Labels: dictionary, namefinder
>> Fix For: tools-1.5.3
>>
>>
>> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
>> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
>> Thanks to Jim for pushing this and to my debugging skills to find.
>> Two possiblilites come to mind:
>> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
>> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
>> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "William Colen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233124#comment-13233124 ]
William Colen commented on OPENNLP-471:
---------------------------------------
Ops... sorry ! I am debugging the code and might have found the real reason for the previous output. I will investigate it further.
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "James Kosin (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Kosin reassigned OPENNLP-471:
-----------------------------------
Assignee: James Kosin
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "James Kosin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233172#comment-13233172 ]
James Kosin commented on OPENNLP-471:
-------------------------------------
William and I found several issues.
His issue above was found at the same time I found it also...
His method of using foundName.length() is more readable; so, I changed my original commit to use the same.
He also found a tokenizing error in one of the test sentences for the name finder that was produceing the bad results.
It also turns out the test set for this also contains the same names that are in the dictonary; so, F measure also returns 1.
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "William Colen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233115#comment-13233115 ]
William Colen commented on OPENNLP-471:
---------------------------------------
Hi James, this is what I mean with neasted span:
Since then, our guests have to ring at Veilchenstrae 11 if they want to visit us, <START:default> Luise <END> and <START:default> George <END> Bauer <END>.
The dictionary contains both [George] and [George, Bauer]
This causes a bad precision, because according to the reference corpus <START:default> George <END> is a false positive, because it expects the longest match.
William
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "William Colen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232331#comment-13232331 ]
William Colen commented on OPENNLP-471:
---------------------------------------
I added some logs to DictionaryNameFinderEvaluatorTest. If you run it you will see that DictionaryNameFinder is creating nested spans. That is why the test reports a low F_1 , but the recall is 100%.
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OPENNLP-471) DictionaryNameFinder has HASHing
issues
Posted by "Hyosup Shim (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyosup Shim updated OPENNLP-471:
--------------------------------
Comment: was deleted
(was: I made some changes (for names of variables) to make easier to read.
Please review my patch.)
> DictionaryNameFinder has HASHing issues
> ---------------------------------------
>
> Key: OPENNLP-471
> URL: https://issues.apache.org/jira/browse/OPENNLP-471
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Reporter: James Kosin
> Assignee: James Kosin
> Labels: dictionary, namefinder
> Fix For: tools-1.5.3
>
>
> The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.
> Having a dictionary with {"folic", "acid"} without an entry for {"folic"} will cause the find() method to totally skip the fact there is a longer match possible.
> Thanks to Jim for pushing this and to my debugging skills to find.
> Two possiblilites come to mind:
> 1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.
> 2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.
> 3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira