You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by Rohana Rajapakse <Ro...@gossinteractive.com> on 2011/03/02 13:47:05 UTC
RE: Tokenizer issue - Quotation marks
My NameFinder training model (created from CONLL + Reuters) has <START> and <END> markups for person names. It doesn't have <SPLIT> markups. I am trying the testTokenizer() test with TokenizerTestUtil.createMaxentTokenModel() to create a model using my training data file. I had to remove <START> and <END> tags and add few <SPLIT> tags to get the test to work (to get "Number of Outcomes" to match). It learns a model now, but not perfect. I need to add <SPLIT> markups for all single and double quotes etc.
By the way, where is the " TokenizerConverter" that you had mentioned. My download (from sourceforge) doesn't have it. Also, where is the converter to produce name
Finder that you have created to convert CONLL03. Am I missing some code in my download.
Also, please point me to your "docbook". Would like to know more about the detokenizer. I can't find a "release candidate" in the download site.
Thanks
Rohana Rajapakse
Senior Software Developer
GOSS Interactive
t: +44 (0)844 880 3637
f: +44 (0)844 880 3638
e: rohana.rajapakse@gossinteractive.com
w: www.gossinteractive.com
-----Original Message-----
From: Rohana Rajapakse [mailto:Rohana.Rajapakse@gossinteractive.com]
Sent: 24 February 2011 11:10
To: opennlp-users@incubator.apache.org
Subject: RE: Tokenizer issue - Quotation marks
Thanks a lot.
I did create a name finder training data file using CONNEL sometimes ago. Will have a look at how I did it. I may be able to convert this training file to produce a tokenizer training file.
Thanks a lot. Will let you know how I get on with this. Would contribute anything that might be useful to others.
Rohana
-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com]
Sent: 24 February 2011 10:59
To: opennlp-users@incubator.apache.org
Subject: Re: Tokenizer issue - Quotation marks
On 2/23/11 2:33 PM, Rohana Rajapakse wrote:
> Yes I do have Reuters corpus with me. Also used Browns corpus (subset of Reuters?) some times ago. So, I should have that as well. Would like to know the steps to create a training set.
>
For CONLL03 (uses Reuters data) we created a converter to produce name
finder
training material. The process how to do that is described in our
docbook (just build
it or download the release candidate).
After you produced the name finder training file you can use the
converter for
the tokenizer and a detokenizer file to produce training data for the
tokenizer.
There is a sample detokenizer file which maybe must be extended a little
for english,
maybe we should create a new folder within the tools project to collect
all the
non-statistical model files. I guess a good detokenizer for the reuters
corpus could be
useful for others too.
That can be done with a command like this
bin/opennlp TokenizerConverter namefinder -encoding UTF-8 -data
en-ner-reuters.train -detokenizer lating-detokenizer.xml > en-tok.train
Hope that helps to get you started, contribution about this to the
documentation is very welcome.
Jörn
Mail about the RC:
http://mail-archives.apache.org/mod_mbox/incubator-opennlp-dev/201102.mbox/%3C4D659A52.1090806@gmail.com%3E
GOSS community User Group for clients. Sign-up here: www.gossinteractive.com/usergroup
Have you registered for our e-Newsletter? www.gossinteractive.com/newsletter
Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908
This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.
Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.
RE: Tokenizer issue - Quotation marks
Posted by Rohana Rajapakse <Ro...@gossinteractive.com>.
Thanks Jörn.
I can see in the code that depending on the operation a token will be merged to the left or right token. But I can't see where (in the code) it adds a <SPLIT> token. Can you please point me to the right place in the code?
Thanks
Rohana
-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com]
Sent: 04 March 2011 15:05
To: opennlp-users@incubator.apache.org
Subject: Re: Tokenizer issue - Quotation marks
On 3/4/11 3:46 PM, Rohana Rajapakse wrote:
> That works great. It was not clear to me where/how the detokenizer rules are used. I thought it's for combining a given token to the one before or after it.
Exactly, the converter detokenizes your input tokens with the
detokenizer, now it knows which tokens will be merged
together and is able to add the <SPLIT> tag between the token instead of
just cat the tokens together.
> I can see in the training file that it has added<SPLIT> tags before "'s".
> Can I include spaces in the rule (e.g. "'s " note the trailing space). This will make it explicit.
Not sure I understand you correctly here. I mean we already have
tokenized data, the tokens do not contain
any information about the spaces between them.
Lets say we have these two strings:
1: "A sample"
2: "A sample"
Now we use a white space tokenizer to tokenize them and
the result would be like this:
1: "A", "sample"
2: "A", "sample"
In the token representation we do not have any white spaces anymore.
Jörn
GOSS community User Group for clients. Sign-up here: www.gossinteractive.com/usergroup
Have you registered for our e-Newsletter? www.gossinteractive.com/newsletter
Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908
This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.
Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.
Re: Tokenizer issue - Quotation marks
Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/4/11 3:46 PM, Rohana Rajapakse wrote:
> That works great. It was not clear to me where/how the detokenizer rules are used. I thought it's for combining a given token to the one before or after it.
Exactly, the converter detokenizes your input tokens with the
detokenizer, now it knows which tokens will be merged
together and is able to add the <SPLIT> tag between the token instead of
just cat the tokens together.
> I can see in the training file that it has added<SPLIT> tags before "'s".
> Can I include spaces in the rule (e.g. "'s " note the trailing space). This will make it explicit.
Not sure I understand you correctly here. I mean we already have
tokenized data, the tokens do not contain
any information about the spaces between them.
Lets say we have these two strings:
1: "A sample"
2: "A sample"
Now we use a white space tokenizer to tokenize them and
the result would be like this:
1: "A", "sample"
2: "A", "sample"
In the token representation we do not have any white spaces anymore.
Jörn
RE: Tokenizer issue - Quotation marks
Posted by Rohana Rajapakse <Ro...@gossinteractive.com>.
That works great. It was not clear to me where/how the detokenizer rules are used. I thought it's for combining a given token to the one before or after it. I can see in the training file that it has added <SPLIT> tags before "'s".
Can I include spaces in the rule (e.g. "'s " note the trailing space). This will make it explicit.
Thanks
Rohana
-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com]
Sent: 04 March 2011 12:34
To: opennlp-users@incubator.apache.org
Subject: Re: Tokenizer issue - Quotation marks
On 3/4/11 1:06 PM, Rohana Rajapakse wrote:
> Hi Jorn,
>
>
>
> I have modified the toString() method in TokenSample.java as given below. This is to add a<SPLIT> token before the token 's .
>
> This helped me to train a tokenizer model that splits for eg. "it's" into two tokens "it" and "'s" at the same time detokenizer rule (same as the rule for double quote) splitting single quotes from expressions that are enclosed in between a pair of single quotes.
>
> This does not handle other cases of single quotes (e.g. don't, can't etc and names like O'Conner).
>
Had a look at the change. The tokenization information must be provided
to TokenSample, this class
then just encapsulates that knowledge. So it is not the responsibility
of it to figure out how
things should be tokenized or not.
In your case I think you can just add "'s" to your detokenizer
dictionary like this:
<entry operation="MOVE_LEFT">
<token>'s</token>
</entry>
Doesn't that fix your issue?
Jörn
GOSS community User Group for clients. Sign-up here: www.gossinteractive.com/usergroup
Have you registered for our e-Newsletter? www.gossinteractive.com/newsletter
Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908
This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.
Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.
Re: Tokenizer issue - Quotation marks
Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/4/11 1:06 PM, Rohana Rajapakse wrote:
> Hi Jorn,
>
>
>
> I have modified the toString() method in TokenSample.java as given below. This is to add a<SPLIT> token before the token 's .
>
> This helped me to train a tokenizer model that splits for eg. "it's" into two tokens "it" and "'s" at the same time detokenizer rule (same as the rule for double quote) splitting single quotes from expressions that are enclosed in between a pair of single quotes.
>
> This does not handle other cases of single quotes (e.g. don't, can't etc and names like O'Conner).
>
Had a look at the change. The tokenization information must be provided
to TokenSample, this class
then just encapsulates that knowledge. So it is not the responsibility
of it to figure out how
things should be tokenized or not.
In your case I think you can just add "'s" to your detokenizer
dictionary like this:
<entry operation="MOVE_LEFT">
<token>'s</token>
</entry>
Doesn't that fix your issue?
Jörn
RE: Tokenizer issue - Quotation marks
Posted by Rohana Rajapakse <Ro...@gossinteractive.com>.
Hi Jorn,
I have modified the toString() method in TokenSample.java as given below. This is to add a <SPLIT> token before the token 's .
This helped me to train a tokenizer model that splits for eg. "it's" into two tokens "it" and "'s" at the same time detokenizer rule (same as the rule for double quote) splitting single quotes from expressions that are enclosed in between a pair of single quotes.
This does not handle other cases of single quotes (e.g. don't, can't etc and names like O'Conner).
I am not sure if this change of code affects other functionalities of opennlp (where else the TokenSample class used?) and if it was the right place to do it.
Please let me know what you think!
Regards
Rohana
public String toString() {
StringBuilder sentence = new StringBuilder();
int lastEndIndex = -1;
for (Span token : tokenSpans) {
if (lastEndIndex != -1) {
// If there are no chars between last token
// and this token insert the separator chars
// otherwise insert a space
String separator = "";
if (lastEndIndex == token.getStart())
separator = separatorChars;
else {
separator = " ";
//New condition for adding <SPLIT> before 's into the training file when creating/converting conll03 to produce tokenizer training
//data using "TokenizerConverter"
if (token.getCoveredText(text).equals("'s")) {
separator = separatorChars;
}
}
sentence.append(separator);
}
sentence.append(token.getCoveredText(text));
lastEndIndex = token.getEnd();
}
return sentence.toString();
}
-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com]
Sent: 03 March 2011 16:01
To: opennlp-users@incubator.apache.org
Subject: Re: Tokenizer issue - Quotation marks
On 3/3/11 4:33 PM, Rohana Rajapakse wrote:
> Thanks. I have got the training files created (conll03 + Reuters) and models trained. Used the latin-detokenizer that came with the download. The trained model solves the double quotation problem (e.g. "mistakes" now results in three tokens: ", mistakes and ").
>
> I have tried adding the same detokenizer rules for single quote. However, it seems to conflict with the different usage of the single quote (e.g. possession as Tom's, It's etc.) This means we will have such cases separately. I will try adding<SPLIT> tags for those cases (e.g. Tom<SPLIT>'s , it<SPLIT>'s etc.). Don't know which gets the priority, rules in the detokenizer or<SPLIT> tags...
Yes you need to add all the tokens which should be attached to the
previous one, like "'s", "'t", etc.
It would be nice to have such a file as part of the project.
Jörn
GOSS community User Group for clients. Sign-up here: www.gossinteractive.com/usergroup
Have you registered for our e-Newsletter? www.gossinteractive.com/newsletter
Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908
This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.
Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.
Re: Tokenizer issue - Quotation marks
Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/3/11 4:33 PM, Rohana Rajapakse wrote:
> Thanks. I have got the training files created (conll03 + Reuters) and models trained. Used the latin-detokenizer that came with the download. The trained model solves the double quotation problem (e.g. "mistakes" now results in three tokens: ", mistakes and ").
>
> I have tried adding the same detokenizer rules for single quote. However, it seems to conflict with the different usage of the single quote (e.g. possession as Tom's, It's etc.) This means we will have such cases separately. I will try adding<SPLIT> tags for those cases (e.g. Tom<SPLIT>'s , it<SPLIT>'s etc.). Don't know which gets the priority, rules in the detokenizer or<SPLIT> tags...
Yes you need to add all the tokens which should be attached to the
previous one, like "'s", "'t", etc.
It would be nice to have such a file as part of the project.
Jörn
RE: Tokenizer issue - Quotation marks
Posted by Rohana Rajapakse <Ro...@gossinteractive.com>.
Thanks. I have got the training files created (conll03 + Reuters) and models trained. Used the latin-detokenizer that came with the download. The trained model solves the double quotation problem (e.g. "mistakes" now results in three tokens: ", mistakes and ").
I have tried adding the same detokenizer rules for single quote. However, it seems to conflict with the different usage of the single quote (e.g. possession as Tom's, It's etc.) This means we will have such cases separately. I will try adding <SPLIT> tags for those cases (e.g. Tom <SPLIT>'s , it<SPLIT>'s etc.). Don't know which gets the priority, rules in the detokenizer or <SPLIT> tags...
Rohana
-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com]
Sent: 02 March 2011 13:08
To: opennlp-users@incubator.apache.org
Subject: Re: Tokenizer issue - Quotation marks
On 3/2/11 1:47 PM, Rohana Rajapakse wrote:
> My NameFinder training model (created from CONLL + Reuters) has<START> and<END> markups for person names. It doesn't have<SPLIT> markups. I am trying the testTokenizer() test with TokenizerTestUtil.createMaxentTokenModel() to create a model using my training data file. I had to remove<START> and<END> tags and add few<SPLIT> tags to get the test to work (to get "Number of Outcomes" to match). It learns a model now, but not perfect. I need to add<SPLIT> markups for all single and double quotes etc.
>
> By the way, where is the " TokenizerConverter" that you had mentioned. My download (from sourceforge) doesn't have it. Also, where is the converter to produce name
> Finder that you have created to convert CONLL03. Am I missing some code in my download.
>
> Also, please point me to your "docbook". Would like to know more about the detokenizer. I can't find a "release candidate" in the download site.
>
The release candidate can be found here:
http://people.apache.org/~joern/releases/opennlp-1.5.1-incubating/rc1/
Just use your name finder training file with the TokenizerConverter.
Pieces of the work
is in 1.5.0 and all the things you are missing are in 1.5.1. The docbook
is also included
in the 1.5.1 distribution.
I suggest that you just re-try with the rc1.
Jörn
GOSS community User Group for clients. Sign-up here: www.gossinteractive.com/usergroup
Have you registered for our e-Newsletter? www.gossinteractive.com/newsletter
Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908
This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.
Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.
Re: Tokenizer issue - Quotation marks
Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/2/11 1:47 PM, Rohana Rajapakse wrote:
> My NameFinder training model (created from CONLL + Reuters) has<START> and<END> markups for person names. It doesn't have<SPLIT> markups. I am trying the testTokenizer() test with TokenizerTestUtil.createMaxentTokenModel() to create a model using my training data file. I had to remove<START> and<END> tags and add few<SPLIT> tags to get the test to work (to get "Number of Outcomes" to match). It learns a model now, but not perfect. I need to add<SPLIT> markups for all single and double quotes etc.
>
> By the way, where is the " TokenizerConverter" that you had mentioned. My download (from sourceforge) doesn't have it. Also, where is the converter to produce name
> Finder that you have created to convert CONLL03. Am I missing some code in my download.
>
> Also, please point me to your "docbook". Would like to know more about the detokenizer. I can't find a "release candidate" in the download site.
>
The release candidate can be found here:
http://people.apache.org/~joern/releases/opennlp-1.5.1-incubating/rc1/
Just use your name finder training file with the TokenizerConverter.
Pieces of the work
is in 1.5.0 and all the things you are missing are in 1.5.1. The docbook
is also included
in the 1.5.1 distribution.
I suggest that you just re-try with the rc1.
Jörn