You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by Arne Claassen <ar...@mindtouch.com> on 2010/06/14 23:11:59 UTC

Are custom tokenizers not invoked when a wildcard is detected

I created a tokenizer that replaces '-', whitespace and "_" all with  
'_'.

Now if i query on that field with "foo-", the parser turns it into  
"foo_", but if i parse "foo-*" it stays "foo-*" and my tokenizer is  
never invoked. Am I not implementing something in my tokenizer, is  
this intentionial or is it a bug?

thanks,
Arne Claassen

MindTouch
San Diego, CA
http://twitter.com/sdether


Re: Are custom tokenizers not invoked when a wildcard is detected

Posted by digy digy <di...@gmail.com>.
Please see the issue  https://issues.apache.org/jira/browse/LUCENENET-51

> Hoss Man commented on LUCENENET-51:

> this is known behavior in the Java library due to the fact that there is no
> "clean" way to let an analyzer process a prefix (or fuzzy or wildcard) query
> ... it is covered in the FAQ...

> http://wiki.apache.org/lucene-java/LuceneFAQ#head-133cf44dd3dff3680c96c1316a663e881eeac35a
> "Are Wildcard, Prefix, and Fuzzy queries case sensitive?"


DIGY


On Tue, Jun 15, 2010 at 12:11 AM, Arne Claassen <ar...@mindtouch.com> wrote:

> I created a tokenizer that replaces '-', whitespace and "_" all with '_'.
>
> Now if i query on that field with "foo-", the parser turns it into "foo_",
> but if i parse "foo-*" it stays "foo-*" and my tokenizer is never invoked.
> Am I not implementing something in my tokenizer, is this intentionial or is
> it a bug?
>
> thanks,
> Arne Claassen
>
> MindTouch
> San Diego, CA
> http://twitter.com/sdether
>
>