You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by j philoon <jp...@penbaymedia.com> on 2009/11/24 16:15:31 UTC

PatternTokenizer question

I have defined a comma-delimited pattern tokenizer as follows:
    <fieldType name="text_comma" class="solr.TextField"
positionIncrementGap="100">
      <analyzer>
        <tokenizer class="solr.PatternTokenizerFactory" pattern=","/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
    </fieldType>

<field name="commafld" type="text_comma" indexed="true" stored="true"/>

This appears to work fine when adding documents, since if I add a field
commafld as "word1,WORD2,word 3" I see terms in the index as expected:
"word1", "word2", and "word 3".

When I query, I am expecting that the same tokenization would take place, so
a query that has 'commafld:(word 3)' would match term "word 3".  However, I
find I have to submit the query as 'commafld:("word 3")'.  That is, it seems
as if whitespace tokenization is taking place, not the comma-delimited
tokenization.

Am I misunderstanding what should be happening or making some basic mistake? 
Thanks. 
-- 
View this message in context: http://old.nabble.com/PatternTokenizer-question-tp26497675p26497675.html
Sent from the Solr - User mailing list archive at Nabble.com.


Re: PatternTokenizer question

Posted by Walter Underwood <wu...@wunderwood.org>.
I think you are looking for a "free text" query parser. Essentially, it would use the analyzer chain, but then turn the token stream into a query tree. Very useful if you are searching for a band named "+/-".

wunder

On Nov 25, 2009, at 12:00 PM, Chris Hostetter wrote:

> 
> : I think the answer to my question is contained in the wiki when discussing
> : the SynonymFilter, "The Lucene QueryParser tokenizes on white space before
> : giving any text to the Analyzer".  This would indeed explain what I am
> : getting.  Next question - can I avoid that behavior?
> 
> it's the nature of the lucene query parser -- whitespace is a "meta 
> character" that provides instructions to the parse, just like '+', '\', 
> '"', etc...
> 
> you could always use a quoted string (so the parser treats all of your 
> input as one phrase) or try the "field" QParser (which is essentailly the 
> same thing as using a quoted phrase but doesn't require the quotes, or 
> respect any of the other escape characters)
> 
> 
> 
> -Hoss
> 


Re: PatternTokenizer question

Posted by Chris Hostetter <ho...@fucit.org>.
: I think the answer to my question is contained in the wiki when discussing
: the SynonymFilter, "The Lucene QueryParser tokenizes on white space before
: giving any text to the Analyzer".  This would indeed explain what I am
: getting.  Next question - can I avoid that behavior?

it's the nature of the lucene query parser -- whitespace is a "meta 
character" that provides instructions to the parse, just like '+', '\', 
'"', etc...

you could always use a quoted string (so the parser treats all of your 
input as one phrase) or try the "field" QParser (which is essentailly the 
same thing as using a quoted phrase but doesn't require the quotes, or 
respect any of the other escape characters)



-Hoss


Re: PatternTokenizer question

Posted by j philoon <jp...@penbaymedia.com>.
I think the answer to my question is contained in the wiki when discussing
the SynonymFilter, "The Lucene QueryParser tokenizes on white space before
giving any text to the Analyzer".  This would indeed explain what I am
getting.  Next question - can I avoid that behavior?


j philoon wrote:
> 
> I have defined a comma-delimited pattern tokenizer as follows:
>     <fieldType name="text_comma" class="solr.TextField"
> positionIncrementGap="100">
>       <analyzer>
>         <tokenizer class="solr.PatternTokenizerFactory" pattern=","/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>       </analyzer>
>     </fieldType>
> 
> <field name="commafld" type="text_comma" indexed="true" stored="true"/>
> 
> This appears to work fine when adding documents, since if I add a field
> commafld as "word1,WORD2,word 3" I see terms in the index as expected:
> "word1", "word2", and "word 3".
> 
> When I query, I am expecting that the same tokenization would take place,
> so a query that has 'commafld:(word 3)' would match term "word 3". 
> However, I find I have to submit the query as 'commafld:("word 3")'.  That
> is, it seems as if whitespace tokenization is taking place, not the
> comma-delimited tokenization.
> 
> Am I misunderstanding what should be happening or making some basic
> mistake?  Thanks. 
> 

-- 
View this message in context: http://old.nabble.com/PatternTokenizer-question-tp26497675p26503324.html
Sent from the Solr - User mailing list archive at Nabble.com.