Solr, ICUTokenizer with Latin-break-only-on-whitespace

[Jonathan Rochkind <ro...@jhu.edu>]

(to solr-user, CC'ing author I'm responding to)

I found the solr-user listserv contribution at:

https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201305.mbox/%3C51965E70.6070409@elyograg.org%3E

Which explain a way you can supply custom rulefiles to ICUTokenizer, in 
this case to tell it to only break on whitespace for Latin character 
substrings.

I am trying to use the technique explained there in Solr 4.3, but either 
it's not working, or it's not doing what I'd expect.

I want, for instance, "C++ Language" to be tokenized into "C++", 
"Language".  But the ICUTokenizer, even with the 
rulefiles="Latn:Latin-break-only-on-whitespace.rbbi", with the rbbi file 
from the Solr 4.3 source [1].

But the ICUTokenizer, even with the that rulefile, is still stripping 
the punctuation, and tokenizing that into "C", "Language".

Can anyone give me any guidance or hints? I don't entirely understand 
the semantics of the rbbi file to try debugging there. Is something not 
working, or does the rbbi file just not express the semantics I want?

Thanks for any tips.



[1] 
http://svn.apache.org/viewvc/lucene/dev/tags/lucene_solr_4_3_0/lucene/analysis/icu/src/test/org/apache/lucene/analysis/icu/segmentation/Latin-break-only-on-whitespace.rbbi?revision=1479557&view=markup

Re: Solr, ICUTokenizer with Latin-break-only-on-whitespace

[Jonathan Rochkind <ro...@jhu.edu>]

Thank you... I started out writing an email with screenshots proving 
that it wasn't working for me in 4.3.0... and of course, having to 
confirm every single detail in order to say I confirmed it... I realized 
it was a mistake on my part, not testing what I thought I was testing.

Does indeed appear to be working now. Thanks! And thanks for this feature.


On 6/20/2013 3:40 PM, Shawn Heisey wrote:
> On 6/20/2013 1:26 PM, Jonathan Rochkind wrote:
>> I want, for instance, "C++ Language" to be tokenized into "C++",
>> "Language".  But the ICUTokenizer, even with the
>> rulefiles="Latn:Latin-break-only-on-whitespace.rbbi", with the rbbi
>> file from the Solr 4.3 source [1].
>>
>> But the ICUTokenizer, even with the that rulefile, is still stripping
>> the punctuation, and tokenizing that into "C", "Language".
>
> This screenshot is using branch_4x downloaded and compiled a couple of
> hours ago, with the rbbi file you mentioned copied to the conf directory:
>
> https://dl.dropboxusercontent.com/u/97770508/icutokenizer-whitespace-only.png
>
>
> It shows that the ++ is maintained by the ICU tokenizer. It also
> illustrates a UI bug that I will have to show to steffkes where the ++
> is lost from the input field after analysis.
>
> Thanks,
> Shawn
>

Re: Solr, ICUTokenizer with Latin-break-only-on-whitespace

[Shawn Heisey <so...@elyograg.org>]

On 6/20/2013 1:26 PM, Jonathan Rochkind wrote:
> I want, for instance, "C++ Language" to be tokenized into "C++", 
> "Language".  But the ICUTokenizer, even with the 
> rulefiles="Latn:Latin-break-only-on-whitespace.rbbi", with the rbbi 
> file from the Solr 4.3 source [1].
>
> But the ICUTokenizer, even with the that rulefile, is still stripping 
> the punctuation, and tokenizing that into "C", "Language".

This screenshot is using branch_4x downloaded and compiled a couple of 
hours ago, with the rbbi file you mentioned copied to the conf directory:

https://dl.dropboxusercontent.com/u/97770508/icutokenizer-whitespace-only.png

It shows that the ++ is maintained by the ICU tokenizer. It also 
illustrates a UI bug that I will have to show to steffkes where the ++ 
is lost from the input field after analysis.

Thanks,
Shawn