You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Peter Keane <pk...@mail.utexas.edu> on 2009/08/03 21:56:27 UTC
Solr Search probem w/ phrase searches, text type, w/ escaped
characters
Greetings--
I am having a problem with a search that has stumped me. I have a field
(called "source") with the following value:
Miguel : three dimensions : [Exhibition]
when I perform this search:
source:"Miguel \: three dimensions \: \[Exhibition\]"
it fails to match.
Yet when I change the value to:
Miguel : three dimensions : [Exhibitio
the following search matches:
source:"Miguel \: three dimensions \: \[Exhibitio"
This is a text field type w/ schema.xml:
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
generateNumberParts="1" catenateWords="1" catenateNumbers="1"
catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
generateNumberParts="1" catenateWords="0" catenateNumbers="0"
catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
I've tried lots of variations with combinations of characters that require
escaping and have not found a pattern as to why it sometimes works and
sometimes not. (Note the quotations marks -- these are phrase searches). I
had other problems that kep me from using a string tye for these fields.
--peter keane
Re: Solr Search probem w/ phrase searches, text type, w/ escaped
characters
Posted by Peter Keane <pk...@mail.utexas.edu>.
Thanks!
Any idea why
Miguel : three dimensions : [Exhibitio
parse to: miguel, three,dimensions, exhibitio
BUT
Miguel : three dimensions : [Exhibition]
parses to miguel, three, dimensions, null_1, exhibition
seems quite strange...
--peter
On Mon, Aug 3, 2009 at 4:02 PM, Andrzej Bialecki <ab...@getopt.org> wrote:
> Peter Keane wrote:
>
>> I've used Luke to figure out what is going on, and I see in the fields
>> that
>> fail to match, a "null_1". Could someone tell me what that is? I see
>> some
>> null_100s there as well, which see to separate field values. Clearly the
>> null_1s are causing the search to fail.
>>
>
> You used the "Reconstruct" function to obtain the field values for unstored
> fields, right? null_NNN is Luke's way of telling you that the tokens that
> should be on these positions are absent, because they were removed by
> analyzer during indexing, and there is no stored value of this field from
> which you could recover the original text. In other words, they are holes in
> the token stream, of length NNN.
>
> Such holes may be also produced by artificially increasing the token
> positions, hence the null_100 that serves to separate multiple field values
> so that e.g. phrase queries don't match unrelated text.
>
> Phrase queries that you can construct using QueryParser can't match two
> tokens separated by a hole, unless you set a slop value > 0.
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
Re: Solr Search probem w/ phrase searches, text type, w/ escaped
characters
Posted by Andrzej Bialecki <ab...@getopt.org>.
Peter Keane wrote:
> I've used Luke to figure out what is going on, and I see in the fields that
> fail to match, a "null_1". Could someone tell me what that is? I see some
> null_100s there as well, which see to separate field values. Clearly the
> null_1s are causing the search to fail.
You used the "Reconstruct" function to obtain the field values for
unstored fields, right? null_NNN is Luke's way of telling you that the
tokens that should be on these positions are absent, because they were
removed by analyzer during indexing, and there is no stored value of
this field from which you could recover the original text. In other
words, they are holes in the token stream, of length NNN.
Such holes may be also produced by artificially increasing the token
positions, hence the null_100 that serves to separate multiple field
values so that e.g. phrase queries don't match unrelated text.
Phrase queries that you can construct using QueryParser can't match two
tokens separated by a hole, unless you set a slop value > 0.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Solr Search probem w/ phrase searches, text type, w/ escaped
characters
Posted by Peter Keane <pk...@mail.utexas.edu>.
I've used Luke to figure out what is going on, and I see in the fields that
fail to match, a "null_1". Could someone tell me what that is? I see some
null_100s there as well, which see to separate field values. Clearly the
null_1s are causing the search to fail.
--peter keane
On Mon, Aug 3, 2009 at 2:56 PM, Peter Keane <pk...@mail.utexas.edu> wrote:
> Greetings--
>
> I am having a problem with a search that has stumped me. I have a field
> (called "source") with the following value:
>
> Miguel : three dimensions : [Exhibition]
>
> when I perform this search:
>
> source:"Miguel \: three dimensions \: \[Exhibition\]"
>
> it fails to match.
>
> Yet when I change the value to:
>
> Miguel : three dimensions : [Exhibitio
>
> the following search matches:
>
> source:"Miguel \: three dimensions \: \[Exhibitio"
>
> This is a text field type w/ schema.xml:
>
> <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
>
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
> generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> catenateAll="0" splitOnCaseChange="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
>
> <analyzer type="query">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
> generateNumberParts="1" catenateWords="0" catenateNumbers="0"
> catenateAll="0" splitOnCaseChange="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
>
> </fieldType>
>
> I've tried lots of variations with combinations of characters that require
> escaping and have not found a pattern as to why it sometimes works and
> sometimes not. (Note the quotations marks -- these are phrase searches). I
> had other problems that kep me from using a string tye for these fields.
>
> --peter keane
>