Re: Good example of multiple tokenizers for a single field

[Jonathan Rochkind <ro...@jhu.edu>]

On 11/29/2010 5:43 PM, Robert Muir wrote:
> On Mon, Nov 29, 2010 at 5:41 PM, Jonathan Rochkind<ro...@jhu.edu>  wrote:
>> * As a tokenizer, I use the WhitespaceTokenizer.
>>
>> * Then I apply a custom filter that looks for CJK chars, and re-tokenizes
>> any CJK chars into one-token-per-char. This custom filter was written by
>> someone other than me; it is open source; but I'm not sure if it's actually
>> in a public repo, or how well documented it is.  I can put you in touch with
>> the author to try and ask. There may also be a more standard filter other
>> than the custom one I'm using that does the same thing?
>>
> You are describing what standardtokenizer does.


Wait, standardtokenizer already handles CJK and will put each CJK char 
into it's own token?  Really? I had no idea!  Is that documented 
anywhere, or you just have to look at the source to see it?

I had assumed that standardtokenizer didn't have any special handling of 
bytes known to be UTF-8 CJK, because that wasn't mentioned in the 
documentation -- but it does?   That would be convenient and not require 
my custom code.

Jonathan

Re: Good example of multiple tokenizers for a single field

[Robert Muir <rc...@gmail.com>]

On Wed, Dec 1, 2010 at 12:25 PM, Jacob Elder <je...@locamoda.com> wrote:
>
> What does this mean to those of us on Solr 1.4 and Lucene 2.9.3? Does the
> current stable StandardTokenizer handle CJK?
>

yes

Re: Good example of multiple tokenizers for a single field

[Jacob Elder <je...@locamoda.com>]

On Wed, Dec 1, 2010 at 11:01 AM, Robert Muir <rc...@gmail.com> wrote:

> (Jonathan, I apologize for emailing you twice, i meant to hit reply-all)
>
> On Wed, Dec 1, 2010 at 10:49 AM, Jonathan Rochkind <ro...@jhu.edu>
> wrote:
> >
> > Wait, standardtokenizer already handles CJK and will put each CJK char
> into
> > it's own token?  Really? I had no idea!  Is that documented anywhere, or
> you
> > just have to look at the source to see it?
> >
>
> Yes, you are right, the documentation should have been more explicit:
> in previous releases it doesn't say anything about how it tokenizes
> CJK in the documentation. But it does do them this way, and tagged
> them as "CJ" token type.
>
> I think the documentation issue is "fixed" in branch_3x and trunk:
>
>  * As of Lucene version 3.1, this class implements the Word Break rules
> from the
>  * Unicode Text Segmentation algorithm, as specified in
>  * <a href="http://unicode.org/reports/tr29/">Unicode Standard Annex
> #29</a>.
> (from
> http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
> )
>
> So you can read the UAX#29 report and then you know how it tokenizes text
> You can also just use this demo app to see how the new one works:
> http://unicode.org/cldr/utility/breaks.jsp (choose "Word")
>

What does this mean to those of us on Solr 1.4 and Lucene 2.9.3? Does the
current stable StandardTokenizer handle CJK?

-- 
Jacob Elder
@jelder
(646) 535-3379

Re: Good example of multiple tokenizers for a single field

[Robert Muir <rc...@gmail.com>]

(Jonathan, I apologize for emailing you twice, i meant to hit reply-all)

On Wed, Dec 1, 2010 at 10:49 AM, Jonathan Rochkind <ro...@jhu.edu> wrote:
>
> Wait, standardtokenizer already handles CJK and will put each CJK char into
> it's own token?  Really? I had no idea!  Is that documented anywhere, or you
> just have to look at the source to see it?
>

Yes, you are right, the documentation should have been more explicit:
in previous releases it doesn't say anything about how it tokenizes
CJK in the documentation. But it does do them this way, and tagged
them as "CJ" token type.

I think the documentation issue is "fixed" in branch_3x and trunk:

 * As of Lucene version 3.1, this class implements the Word Break rules from the
 * Unicode Text Segmentation algorithm, as specified in
 * <a href="http://unicode.org/reports/tr29/">Unicode Standard Annex #29</a>.
(from http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java)

So you can read the UAX#29 report and then you know how it tokenizes text
You can also just use this demo app to see how the new one works:
http://unicode.org/cldr/utility/breaks.jsp (choose "Word")