You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by Rohana Rajapakse <Ro...@gossinteractive.com> on 2011/03/02 13:47:05 UTC

RE: Tokenizer issue - Quotation marks

My NameFinder training model (created from CONLL + Reuters) has <START> and <END> markups for person names. It doesn't have <SPLIT> markups. I am trying the testTokenizer() test with TokenizerTestUtil.createMaxentTokenModel() to create a model using my training data file. I had to remove <START> and <END> tags and add few <SPLIT> tags to get the test to work (to get "Number of Outcomes" to match). It learns a model now, but not perfect. I need to add <SPLIT> markups for all single and double quotes etc. 

By the way, where is the " TokenizerConverter" that you had mentioned. My download (from sourceforge) doesn't have it. Also, where is the converter to produce name 
Finder that you have created to convert CONLL03. Am I missing some code in my download.

Also, please point me to your "docbook". Would like to know more about the detokenizer. I can't find a "release candidate" in the download site. 

Thanks


Rohana Rajapakse
Senior Software Developer
GOSS Interactive
 
t:  +44 (0)844 880 3637
f:  +44 (0)844 880 3638
e: rohana.rajapakse@gossinteractive.com
w: www.gossinteractive.com 

-----Original Message-----
From: Rohana Rajapakse [mailto:Rohana.Rajapakse@gossinteractive.com] 
Sent: 24 February 2011 11:10
To: opennlp-users@incubator.apache.org
Subject: RE: Tokenizer issue - Quotation marks

Thanks a lot.

I did create a name finder training data file using CONNEL sometimes ago. Will have a look at how I did it. I may be able to convert this training file to produce a tokenizer training file.

Thanks a lot. Will let you know how I get on with this. Would contribute anything that might be useful to others.

Rohana


-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com] 
Sent: 24 February 2011 10:59
To: opennlp-users@incubator.apache.org
Subject: Re: Tokenizer issue - Quotation marks

On 2/23/11 2:33 PM, Rohana Rajapakse wrote:
> Yes I do have Reuters corpus with me. Also used Browns corpus (subset of Reuters?) some times ago. So, I should have that as well. Would like to know the steps to create a training set.
>
For CONLL03 (uses Reuters data) we created a converter to produce name 
finder
training material. The process how to do that is described in our 
docbook (just build
it or download the release candidate).

After you produced the name finder training file you can use the 
converter for
the tokenizer and a detokenizer file to produce training data for the 
tokenizer.
There is a sample detokenizer file which maybe must be extended a little 
for english,
maybe we should create a new folder within the tools project to collect 
all the
non-statistical model files. I guess a good detokenizer for the reuters 
corpus could be
useful for others too.

That can be done with a command like this
bin/opennlp TokenizerConverter namefinder -encoding UTF-8 -data 
en-ner-reuters.train -detokenizer lating-detokenizer.xml > en-tok.train

Hope that helps to get you started, contribution about this to the 
documentation is very welcome.

Jörn

Mail about the RC:
http://mail-archives.apache.org/mod_mbox/incubator-opennlp-dev/201102.mbox/%3C4D659A52.1090806@gmail.com%3E


GOSS community User Group for clients. Sign-up here: www.gossinteractive.com/usergroup 

Have you registered for our e-Newsletter? www.gossinteractive.com/newsletter 

Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908 

This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.

Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.




RE: Tokenizer issue - Quotation marks

Posted by Rohana Rajapakse <Ro...@gossinteractive.com>.
Thanks Jörn. 

I can see in the code that depending on the operation a token will be merged to the left or right token. But I can't see where (in the code) it adds a <SPLIT> token. Can you please point me to the right place in the code?

Thanks

Rohana

-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com] 
Sent: 04 March 2011 15:05
To: opennlp-users@incubator.apache.org
Subject: Re: Tokenizer issue - Quotation marks

On 3/4/11 3:46 PM, Rohana Rajapakse wrote:
> That works great. It was not clear to me where/how the detokenizer rules are used. I thought it's for combining a given token to the one before or after it.

Exactly, the converter detokenizes your input tokens with the 
detokenizer, now it knows which tokens will be merged
together and is able to add the <SPLIT> tag between the token instead of 
just cat the tokens together.
> I can see in the training file that it has added<SPLIT>  tags before "'s".
> Can I include spaces in the rule (e.g. "'s " note the trailing space). This will make it explicit.
Not sure I understand you correctly here. I mean we already have 
tokenized data, the tokens do not contain
any information about the spaces between them.

Lets say we have these two strings:

1: "A    sample"
2: "A sample"

Now we use a white space tokenizer to tokenize them and
the result would be like this:
1: "A", "sample"
2: "A", "sample"

In the token representation we do not have any white spaces anymore.

Jörn


GOSS community User Group for clients. Sign-up here: www.gossinteractive.com/usergroup 

Have you registered for our e-Newsletter? www.gossinteractive.com/newsletter 

Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908 

This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.

Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.




Re: Tokenizer issue - Quotation marks

Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/4/11 3:46 PM, Rohana Rajapakse wrote:
> That works great. It was not clear to me where/how the detokenizer rules are used. I thought it's for combining a given token to the one before or after it.

Exactly, the converter detokenizes your input tokens with the 
detokenizer, now it knows which tokens will be merged
together and is able to add the <SPLIT> tag between the token instead of 
just cat the tokens together.
> I can see in the training file that it has added<SPLIT>  tags before "'s".
> Can I include spaces in the rule (e.g. "'s " note the trailing space). This will make it explicit.
Not sure I understand you correctly here. I mean we already have 
tokenized data, the tokens do not contain
any information about the spaces between them.

Lets say we have these two strings:

1: "A    sample"
2: "A sample"

Now we use a white space tokenizer to tokenize them and
the result would be like this:
1: "A", "sample"
2: "A", "sample"

In the token representation we do not have any white spaces anymore.

Jörn

RE: Tokenizer issue - Quotation marks

Posted by Rohana Rajapakse <Ro...@gossinteractive.com>.
That works great. It was not clear to me where/how the detokenizer rules are used. I thought it's for combining a given token to the one before or after it. I can see in the training file that it has added <SPLIT> tags before "'s".
Can I include spaces in the rule (e.g. "'s " note the trailing space). This will make it explicit.


Thanks

Rohana


-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com] 
Sent: 04 March 2011 12:34
To: opennlp-users@incubator.apache.org
Subject: Re: Tokenizer issue - Quotation marks

On 3/4/11 1:06 PM, Rohana Rajapakse wrote:
> Hi Jorn,
>
>
>
> I have modified the toString() method in TokenSample.java as given below. This is to add a<SPLIT>  token before the token 's .
>
> This helped me to train a tokenizer model that splits for eg. "it's" into two tokens "it" and "'s" at the same time detokenizer rule (same as the rule for double quote) splitting single quotes from expressions that are enclosed in between a pair of single quotes.
>
> This does not handle other cases of single quotes (e.g. don't, can't etc and names like O'Conner).
>
Had a look at the change. The tokenization information must be provided 
to TokenSample, this class
then just encapsulates that knowledge. So it is not the responsibility 
of it to figure out how
things should be tokenized or not.

In your case I think you can just add "'s" to your detokenizer 
dictionary like this:
<entry operation="MOVE_LEFT">
<token>'s</token>
</entry>

Doesn't that fix your issue?

Jörn



GOSS community User Group for clients. Sign-up here: www.gossinteractive.com/usergroup 

Have you registered for our e-Newsletter? www.gossinteractive.com/newsletter 

Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908 

This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.

Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.




Re: Tokenizer issue - Quotation marks

Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/4/11 1:06 PM, Rohana Rajapakse wrote:
> Hi Jorn,
>
>
>
> I have modified the toString() method in TokenSample.java as given below. This is to add a<SPLIT>  token before the token 's .
>
> This helped me to train a tokenizer model that splits for eg. "it's" into two tokens "it" and "'s" at the same time detokenizer rule (same as the rule for double quote) splitting single quotes from expressions that are enclosed in between a pair of single quotes.
>
> This does not handle other cases of single quotes (e.g. don't, can't etc and names like O'Conner).
>
Had a look at the change. The tokenization information must be provided 
to TokenSample, this class
then just encapsulates that knowledge. So it is not the responsibility 
of it to figure out how
things should be tokenized or not.

In your case I think you can just add "'s" to your detokenizer 
dictionary like this:
<entry operation="MOVE_LEFT">
<token>'s</token>
</entry>

Doesn't that fix your issue?

Jörn


RE: Tokenizer issue - Quotation marks

Posted by Rohana Rajapakse <Ro...@gossinteractive.com>.
Hi Jorn,

 

I have modified the toString() method in TokenSample.java as given below. This is to add a <SPLIT> token before the token 's .

This helped me to train a tokenizer model that splits for eg. "it's" into two tokens "it" and "'s" at the same time detokenizer rule (same as the rule for double quote) splitting single quotes from expressions that are enclosed in between a pair of single quotes.

This does not handle other cases of single quotes (e.g. don't, can't etc and names like O'Conner).

 

I am not sure if this change of code affects other functionalities of opennlp (where else the TokenSample class used?) and if it was the right place to do it. 

 

Please let me know what you think!

 

Regards

 

Rohana

 

 

  public String toString() {

    

    StringBuilder sentence = new StringBuilder();

    

    int lastEndIndex = -1;

    for (Span token : tokenSpans) {

      

      if (lastEndIndex != -1) {

 

        // If there are no chars between last token

        // and this token insert the separator chars

        // otherwise insert a space

        

        String separator = "";

        if (lastEndIndex == token.getStart())

          separator = separatorChars; 

         else {

             separator = " ";

             //New condition for adding <SPLIT> before 's into the training file when creating/converting conll03 to produce tokenizer training 

             //data using "TokenizerConverter"

             if (token.getCoveredText(text).equals("'s")) { 

                   separator = separatorChars; 

             }

         }

        sentence.append(separator);

      }

      

      sentence.append(token.getCoveredText(text));

      

      lastEndIndex = token.getEnd();

    }

    

    return sentence.toString();

  }

 

-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com] 
Sent: 03 March 2011 16:01
To: opennlp-users@incubator.apache.org
Subject: Re: Tokenizer issue - Quotation marks

 

On 3/3/11 4:33 PM, Rohana Rajapakse wrote:

> Thanks. I have got the training files created (conll03 + Reuters) and models trained. Used the latin-detokenizer that came with the download. The trained model solves the double quotation problem (e.g. "mistakes" now results in three tokens: ", mistakes and ").

> 

> I have tried adding the same detokenizer rules for single quote. However, it seems to conflict with the different usage of the single quote (e.g. possession as Tom's, It's etc.) This means we will have such cases separately. I will try adding<SPLIT>  tags for those cases (e.g. Tom<SPLIT>'s , it<SPLIT>'s  etc.). Don't know which gets the priority, rules in the detokenizer or<SPLIT>  tags...

 

Yes you need to add all the tokens which should be attached to the 

previous one, like "'s", "'t", etc.

It would be nice to have such a file as part of the project.

 

Jörn

 



GOSS community User Group for clients. Sign-up here: www.gossinteractive.com/usergroup 

Have you registered for our e-Newsletter? www.gossinteractive.com/newsletter 

Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908 

This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.

Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.




Re: Tokenizer issue - Quotation marks

Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/3/11 4:33 PM, Rohana Rajapakse wrote:
> Thanks. I have got the training files created (conll03 + Reuters) and models trained. Used the latin-detokenizer that came with the download. The trained model solves the double quotation problem (e.g. "mistakes" now results in three tokens: ", mistakes and ").
>
> I have tried adding the same detokenizer rules for single quote. However, it seems to conflict with the different usage of the single quote (e.g. possession as Tom's, It's etc.) This means we will have such cases separately. I will try adding<SPLIT>  tags for those cases (e.g. Tom<SPLIT>'s , it<SPLIT>'s  etc.). Don't know which gets the priority, rules in the detokenizer or<SPLIT>  tags...

Yes you need to add all the tokens which should be attached to the 
previous one, like "'s", "'t", etc.
It would be nice to have such a file as part of the project.

Jörn


RE: Tokenizer issue - Quotation marks

Posted by Rohana Rajapakse <Ro...@gossinteractive.com>.
Thanks. I have got the training files created (conll03 + Reuters) and models trained. Used the latin-detokenizer that came with the download. The trained model solves the double quotation problem (e.g. "mistakes" now results in three tokens: ", mistakes and ").

I have tried adding the same detokenizer rules for single quote. However, it seems to conflict with the different usage of the single quote (e.g. possession as Tom's, It's etc.) This means we will have such cases separately. I will try adding <SPLIT> tags for those cases (e.g. Tom <SPLIT>'s , it<SPLIT>'s  etc.). Don't know which gets the priority, rules in the detokenizer or <SPLIT> tags...

Rohana

-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com] 
Sent: 02 March 2011 13:08
To: opennlp-users@incubator.apache.org
Subject: Re: Tokenizer issue - Quotation marks

On 3/2/11 1:47 PM, Rohana Rajapakse wrote:
> My NameFinder training model (created from CONLL + Reuters) has<START>  and<END>  markups for person names. It doesn't have<SPLIT>  markups. I am trying the testTokenizer() test with TokenizerTestUtil.createMaxentTokenModel() to create a model using my training data file. I had to remove<START>  and<END>  tags and add few<SPLIT>  tags to get the test to work (to get "Number of Outcomes" to match). It learns a model now, but not perfect. I need to add<SPLIT>  markups for all single and double quotes etc.
>
> By the way, where is the " TokenizerConverter" that you had mentioned. My download (from sourceforge) doesn't have it. Also, where is the converter to produce name
> Finder that you have created to convert CONLL03. Am I missing some code in my download.
>
> Also, please point me to your "docbook". Would like to know more about the detokenizer. I can't find a "release candidate" in the download site.
>
The release candidate can be found here:
http://people.apache.org/~joern/releases/opennlp-1.5.1-incubating/rc1/

Just use your name finder training file with the TokenizerConverter. 
Pieces of the work
is in 1.5.0 and all the things you are missing are in 1.5.1. The docbook 
is also included
in the 1.5.1 distribution.

I suggest that you just re-try with the rc1.

Jörn


GOSS community User Group for clients. Sign-up here: www.gossinteractive.com/usergroup 

Have you registered for our e-Newsletter? www.gossinteractive.com/newsletter 

Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908 

This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.

Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.




Re: Tokenizer issue - Quotation marks

Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/2/11 1:47 PM, Rohana Rajapakse wrote:
> My NameFinder training model (created from CONLL + Reuters) has<START>  and<END>  markups for person names. It doesn't have<SPLIT>  markups. I am trying the testTokenizer() test with TokenizerTestUtil.createMaxentTokenModel() to create a model using my training data file. I had to remove<START>  and<END>  tags and add few<SPLIT>  tags to get the test to work (to get "Number of Outcomes" to match). It learns a model now, but not perfect. I need to add<SPLIT>  markups for all single and double quotes etc.
>
> By the way, where is the " TokenizerConverter" that you had mentioned. My download (from sourceforge) doesn't have it. Also, where is the converter to produce name
> Finder that you have created to convert CONLL03. Am I missing some code in my download.
>
> Also, please point me to your "docbook". Would like to know more about the detokenizer. I can't find a "release candidate" in the download site.
>
The release candidate can be found here:
http://people.apache.org/~joern/releases/opennlp-1.5.1-incubating/rc1/

Just use your name finder training file with the TokenizerConverter. 
Pieces of the work
is in 1.5.0 and all the things you are missing are in 1.5.1. The docbook 
is also included
in the 1.5.1 distribution.

I suggest that you just re-try with the rc1.

Jörn