You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opennlp.apache.org by "Jörn Kottmann (JIRA)" <ji...@apache.org> on 2010/12/14 11:44:20 UTC
[jira] Created: (OPENNLP-15) Add support for the CoNLL 03 data
format
Add support for the CoNLL 03 data format
----------------------------------------
Key: OPENNLP-15
URL: https://issues.apache.org/jira/browse/OPENNLP-15
Project: OpenNLP
Issue Type: Improvement
Components: Name Finder
Affects Versions: tools-1.5.0-sourceforge
Reporter: Jörn Kottmann
Fix For: tools-1.5.1-incubating
Adding support to convert CoNLL 03 Reurters Support to NameFinder.
Work on this issue began over at sourceforge:
http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "Jörn Kottmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jörn Kottmann closed OPENNLP-15.
--------------------------------
Resolution: Fixed
Can be closed now, implementation has been tested by me and James.
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: Jörn Kottmann
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "Jörn Kottmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jörn Kottmann reassigned OPENNLP-15:
------------------------------------
Assignee: James Kosin
The code should have a unit test with small samples for the English and German data. Hopefully this test will catch bugs which might be introduced while maintaining the code in the future.
James, can you add such a test ? I will provide the German test samples.
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: James Kosin
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "Jörn Kottmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972055#action_12972055 ]
Jörn Kottmann commented on OPENNLP-15:
--------------------------------------
The handling of the file encoding must be improved. Currently the UTF-8 encoding is hardcoded for both generated input files, german and english.
The input files are generated by conll03 scripts. The input encoding might be defined by the
conll03 scripts (must be checked). In the case that is true the encoding could be hardcoded,
otherwise it should be configurable.
If the encoding is hardcoded, the cmd line interface must be improved to not accept an encoding parameter
which is not used.
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "James Kosin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Kosin reassigned OPENNLP-15:
----------------------------------
Assignee: Jörn Kottmann (was: James Kosin)
You have the Gernan data; so, I'll pass the rest of the verification test to you.
-James
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: Jörn Kottmann
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "James Kosin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973999#action_12973999 ]
James Kosin commented on OPENNLP-15:
------------------------------------
Jorn,
Yes, I can provide some test data. I'll try and cover all the bases; since we have several possible outcomes here.
I'll have to check the copyright to determine how much may be too much. They do allow for samples; but, I haven't looked into how much yet.
I may end up having to write our own samples.
James
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: James Kosin
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "Jörn Kottmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971222#action_12971222 ]
Jörn Kottmann commented on OPENNLP-15:
--------------------------------------
Current status is:
"I'm adding the logic for the format for the German data. I'll leave the
testing of this for someone who has the corpus for this to validate the
model." (James Kosin)
James does not have the German data, so he has no chance of testing the
code. I bought the data and will now test his code.
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "James Kosin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977064#action_12977064 ]
James Kosin commented on OPENNLP-15:
------------------------------------
Jorn,
I think with your latest modifications, we can close this issue.
James
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: Jörn Kottmann
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "James Kosin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977536#action_12977536 ]
James Kosin commented on OPENNLP-15:
------------------------------------
No problems. Thanks for testing...
I've also done some testing myself... which is how I caught the -DOCSTART- bug with the parser.
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: Jörn Kottmann
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "James Kosin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974045#action_12974045 ]
James Kosin commented on OPENNLP-15:
------------------------------------
Jorn,
I already found a BUG, in the format converter. The english data appears to always have a blank line after -DOCSTART- tag causing the AdaptiveData to not get cleared properly on the start of a new document.
I'm also fixing this bug with this issue and submitting a very small sample from the ConLL03 data set.
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: James Kosin
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "James Kosin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974456#action_12974456 ]
James Kosin commented on OPENNLP-15:
------------------------------------
Jorn,
I noticed one of the comments you put into Conll03 Parser.
while ((line = lineStream.read()) != null && !StringUtil.isEmpty(line)) {
if (LANGUAGE.EN.equals(lang) && line.startsWith(Conll02NameSampleStream.DOCSTART)) {
isClearAdaptiveData = true;
// english data has a blank line after DOCSTART tag
lineStream.read(); // TODO: Why isn't that caught by isEmpty ?!
continue;
}
The TODO here is explained simply.
1) The while statement is while ! empty. The continue in the if() statement causes a branch to the test. The next line after DOCSTART is blank... isClearAdaptiveData is set to true.
The problem is the statements below here check the blank line condition and performs a secondary read() and return that value.
else if (line != null) {
// Just filter out empty events, if two lines in a row are empty
return read();
}
This secondary read() call clears the isClearAdaptiveData to false again and returns the sentence correctly. Unfortunately, as a side effect the flag is inadvertantly cleared unintentionally I believe with this second call.
To fix this, would mean we would have to refactor the entire function.
-James
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: Jörn Kottmann
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (OPENNLP-15) Add support for the CoNLL 03 data
format
Posted by "Jörn Kottmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENNLP-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977208#action_12977208 ]
Jörn Kottmann commented on OPENNLP-15:
--------------------------------------
I am still busy doing some testing, I will close the issue when that is done.
> Add support for the CoNLL 03 data format
> ----------------------------------------
>
> Key: OPENNLP-15
> URL: https://issues.apache.org/jira/browse/OPENNLP-15
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: Jörn Kottmann
> Fix For: tools-1.5.1-incubating
>
>
> Adding support to convert CoNLL 03 Reurters Support to NameFinder.
> Work on this issue began over at sourceforge:
> http://sourceforge.net/tracker/?func=detail&aid=3081785&group_id=3368&atid=353368
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.