You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "T. Kuro Kurosaka" <ku...@basistech.com> on 2012/10/12 01:35:58 UTC
Any filter to map mutiple tokens into one ?
I am looking for a way to fold a particular sequence of tokens into one
token.
Concretely, I'd like to detect a three-token sequence of "*", ":" and
"*", and replace it with a token of the text "*:*".
I tried SynonymFIlter but it seems it can only deal with a single input
token. "* : * => *:*" seems to be interpreted
as one input token of 5 characters "*", space, ":", space and "*".
I'm using Solr 3.5.
Background:
My tokenizer separate the three character sequence "*:*" into 3 tokens
of one character each.
The edismax parser, when given the query "*:*", i.e. find every doc,
seems to pass the entire string "*:*" to the query analyzer (I suspect
a bug.),
and feed the tokenized result to DisjunctionMaxQuery object,
according to this debug output:
<lst name="debug">
<str name="rawquerystring">*:*</str>
<str name="querystring">*:*</str>
<str name="parsedquery">+MatchAllDocsQuery(*:*)
DisjunctionMaxQuery((body:"* : *"~100^0.5 | title:"* :
*"~100^1.2)~0.01)</str>
<str name="parsedquery_toString">+*:* (body:"* : *"~100^0.5 | title:"* :
*"~100^1.2)~0.01</str>
Notice that there is a space between * and : in
DisjunctionMaxQuery((body:"* : *" ....)
Probably because of this, the hit score is as low as 0.109, while it is
1.000 if an analyzer that doesn't break "*:*" is used.
So I'd like to stitch together "*", ":", "*" into "*:*" again to make
DisjunctionMaxQuery happy.
Thanks.
T. "Kuro" Kurosaka
Re: Any filter to map mutiple tokens into one ?
Posted by Konrad Lötzsch <ko...@antibodies-online.com>.
You can build shingles and then use the synonym filter. in this case you
will have to think about all these token that you don't need after the
shingle filter.
Am 12.10.2012 01:35, schrieb T. Kuro Kurosaka:
> I am looking for a way to fold a particular sequence of tokens into
> one token.
> Concretely, I'd like to detect a three-token sequence of "*", ":" and
> "*", and replace it with a token of the text "*:*".
> I tried SynonymFIlter but it seems it can only deal with a single
> input token. "* : * => *:*" seems to be interpreted
> as one input token of 5 characters "*", space, ":", space and "*".
>
> I'm using Solr 3.5.
>
> Background:
> My tokenizer separate the three character sequence "*:*" into 3 tokens
> of one character each.
> The edismax parser, when given the query "*:*", i.e. find every doc,
> seems to pass the entire string "*:*" to the query analyzer (I suspect
> a bug.),
> and feed the tokenized result to DisjunctionMaxQuery object,
> according to this debug output:
>
> <lst name="debug">
> <str name="rawquerystring">*:*</str>
> <str name="querystring">*:*</str>
> <str name="parsedquery">+MatchAllDocsQuery(*:*)
> DisjunctionMaxQuery((body:"* : *"~100^0.5 | title:"* :
> *"~100^1.2)~0.01)</str>
> <str name="parsedquery_toString">+*:* (body:"* : *"~100^0.5 | title:"*
> : *"~100^1.2)~0.01</str>
>
> Notice that there is a space between * and : in
> DisjunctionMaxQuery((body:"* : *" ....)
>
> Probably because of this, the hit score is as low as 0.109, while it
> is 1.000 if an analyzer that doesn't break "*:*" is used.
> So I'd like to stitch together "*", ":", "*" into "*:*" again to make
> DisjunctionMaxQuery happy.
>
>
> Thanks.
>
>
> T. "Kuro" Kurosaka
>
>
Re: Any filter to map mutiple tokens into one ?
Posted by "T. Kuro Kurosaka" <ku...@basistech.com>.
Jack,
Now I have a direct evidence. I ran Solr under Eclipse and caught that
NGramTokenizer was called from the edismax query parser with "*:*" as
input text.
I have filed a bug with jira:
https://issues.apache.org/jira/browse/SOLR-3962
On 10/15/12 11:32 AM, T. Kuro Kurosaka wrote:
> On 10/15/12 10:35 AM, Jack Krupansky wrote:
>> And you're absolutely certain you see "*:*" being passed to your
>> analyzer in the final release of Solr 4.0???
> I don't have a direct evidence. This is the only theory I have that
> explains why changing FieldType causes the sub-optimal scores.
> If you know of a way to tell if a tokenizer is really invoked, let me
> know.
Re: Any filter to map mutiple tokens into one ?
Posted by "T. Kuro Kurosaka" <ku...@basistech.com>.
On 10/15/12 10:35 AM, Jack Krupansky wrote:
> And you're absolutely certain you see "*:*" being passed to your
> analyzer in the final release of Solr 4.0???
I don't have a direct evidence. This is the only theory I have that
explains why changing FieldType causes the sub-optimal scores.
If you know of a way to tell if a tokenizer is really invoked, let me know.
>
> -- Jack Krupansky
>
> -----Original Message----- From: T. Kuro Kurosaka
> Sent: Monday, October 15, 2012 1:28 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Any filter to map mutiple tokens into one ?
>
> On 10/14/12 12:19 PM, Jack Krupansky wrote:
>> There's a miscommunication here somewhere. Is Solr 4.0 still passing
>> "*:*" to the analyzer? Show us the parsed query for "*:*", as well as
>> the debugQuery "explain" for the score.
> I'm not quite sure what you mean by the parsed query for "*:*".
> This fake analyzer using NGramTokenizer divides "*:*" into three tokens
> "*", ":", and "*", on purpose to simulate our Tokenizer's behavior.
>
> An excerpt of he XML results from the query is pasted in the bottom of
> this message.
>>
>> I mean, "*:*" (MatchAllDocsQuery) has a "constant score", so there
>> isn't any way for it to be "suboptimal".
> That's exactly the point I'd like to raise.
> No matter what analyzers are assigned to fields, the hit score for "*:*"
> must remain 1.0, but it's not happening when an analyzer that divides
> "*:*" are in use.
>
>
> Here's an excerpt of the query response. Notice this element, which
> should not be there, in my opinion:
> DisjunctionMaxQuery((name:"* : *"^0.5))
> There is a space between * and :, and another space between : and *.
>
> <response>
> <lstname="responseHeader">
> <intname="status">0</int>
> <intname="QTime">33</int>
> <lstname="params">
> <strname="indent">on</str>
> <strname="wt"/>
> <strname="version">2.2</str>
> <strname="rows">10</str>
> <strname="defType">edismax</str>
> <strname="pf">name^0.5</str>
> <strname="fl">*,score</str>
> <strname="debugQuery">on</str>
> <strname="start">0</str>
> <strname="q">*:*</str>
> <strname="qt"/>
> <strname="fq"/>
> </lst>
> </lst>
> <resultname="response"numFound="32"start="0"maxScore="0.14764866">
> <doc>
> <strname="id">GB18030TEST</str>
> <strname="name">Test with some GB18030 encoded characters</str>
> <arrname="features">
> <str>No accents here</str>
> <str>这是一个功能</str>
> <str>This is a feature (translated)</str>
> <str>这份文件是很有光泽</str>
> <str>This document is very shiny (translated)</str>
> </arr>
> <floatname="price">0.0</float>
> <strname="price_c">0,USD</str>
> <boolname="inStock">true</bool>
> <longname="_version_">1415830106215022592</long>
> <floatname="score">0.14764866</float>
> </doc>
> ...
> </result>
> <lstname="debug">
> <strname="rawquerystring">*:*</str>
> <strname="querystring">*:*</str>
> <strname="parsedquery">
> (+MatchAllDocsQuery(*:*) DisjunctionMaxQuery((name:"* :
> *"^0.5)))/no_coord
> </str>
> <strname="parsedquery_toString">+*:* (name:"* : *"^0.5)</str>
> <lstname="explain">
> <strname="GB18030TEST">
> 0.14764866 = (MATCH) sum of: 0.14764866 = (MATCH) MatchAllDocsQuery,
> product of: 0.14764866 = queryNorm
> </str>
> </lst>
> <strname="QParser">ExtendedDismaxQParser</str>
> <nullname="altquerystring"/>
> <nullname="boostfuncs"/>
> ...
>
> </lst>
> </lst>
> </lst>
> </response>
Re: Any filter to map mutiple tokens into one ?
Posted by Jack Krupansky <ja...@basetechnology.com>.
And you're absolutely certain you see "*:*" being passed to your analyzer in
the final release of Solr 4.0???
-- Jack Krupansky
-----Original Message-----
From: T. Kuro Kurosaka
Sent: Monday, October 15, 2012 1:28 PM
To: solr-user@lucene.apache.org
Subject: Re: Any filter to map mutiple tokens into one ?
On 10/14/12 12:19 PM, Jack Krupansky wrote:
> There's a miscommunication here somewhere. Is Solr 4.0 still passing "*:*"
> to the analyzer? Show us the parsed query for "*:*", as well as the
> debugQuery "explain" for the score.
I'm not quite sure what you mean by the parsed query for "*:*".
This fake analyzer using NGramTokenizer divides "*:*" into three tokens
"*", ":", and "*", on purpose to simulate our Tokenizer's behavior.
An excerpt of he XML results from the query is pasted in the bottom of
this message.
>
> I mean, "*:*" (MatchAllDocsQuery) has a "constant score", so there isn't
> any way for it to be "suboptimal".
That's exactly the point I'd like to raise.
No matter what analyzers are assigned to fields, the hit score for "*:*"
must remain 1.0, but it's not happening when an analyzer that divides
"*:*" are in use.
Here's an excerpt of the query response. Notice this element, which
should not be there, in my opinion:
DisjunctionMaxQuery((name:"* : *"^0.5))
There is a space between * and :, and another space between : and *.
<response>
<lstname="responseHeader">
<intname="status">0</int>
<intname="QTime">33</int>
<lstname="params">
<strname="indent">on</str>
<strname="wt"/>
<strname="version">2.2</str>
<strname="rows">10</str>
<strname="defType">edismax</str>
<strname="pf">name^0.5</str>
<strname="fl">*,score</str>
<strname="debugQuery">on</str>
<strname="start">0</str>
<strname="q">*:*</str>
<strname="qt"/>
<strname="fq"/>
</lst>
</lst>
<resultname="response"numFound="32"start="0"maxScore="0.14764866">
<doc>
<strname="id">GB18030TEST</str>
<strname="name">Test with some GB18030 encoded characters</str>
<arrname="features">
<str>No accents here</str>
<str>这是一个功能</str>
<str>This is a feature (translated)</str>
<str>这份文件是很有光泽</str>
<str>This document is very shiny (translated)</str>
</arr>
<floatname="price">0.0</float>
<strname="price_c">0,USD</str>
<boolname="inStock">true</bool>
<longname="_version_">1415830106215022592</long>
<floatname="score">0.14764866</float>
</doc>
...
</result>
<lstname="debug">
<strname="rawquerystring">*:*</str>
<strname="querystring">*:*</str>
<strname="parsedquery">
(+MatchAllDocsQuery(*:*) DisjunctionMaxQuery((name:"* : *"^0.5)))/no_coord
</str>
<strname="parsedquery_toString">+*:* (name:"* : *"^0.5)</str>
<lstname="explain">
<strname="GB18030TEST">
0.14764866 = (MATCH) sum of: 0.14764866 = (MATCH) MatchAllDocsQuery,
product of: 0.14764866 = queryNorm
</str>
</lst>
<strname="QParser">ExtendedDismaxQParser</str>
<nullname="altquerystring"/>
<nullname="boostfuncs"/>
...
</lst>
</lst>
</lst>
</response>
Re: Any filter to map mutiple tokens into one ?
Posted by "T. Kuro Kurosaka" <ku...@basistech.com>.
On 10/14/12 12:19 PM, Jack Krupansky wrote:
> There's a miscommunication here somewhere. Is Solr 4.0 still passing
> "*:*" to the analyzer? Show us the parsed query for "*:*", as well as
> the debugQuery "explain" for the score.
I'm not quite sure what you mean by the parsed query for "*:*".
This fake analyzer using NGramTokenizer divides "*:*" into three tokens
"*", ":", and "*", on purpose to simulate our Tokenizer's behavior.
An excerpt of he XML results from the query is pasted in the bottom of
this message.
>
> I mean, "*:*" (MatchAllDocsQuery) has a "constant score", so there
> isn't any way for it to be "suboptimal".
That's exactly the point I'd like to raise.
No matter what analyzers are assigned to fields, the hit score for "*:*"
must remain 1.0, but it's not happening when an analyzer that divides
"*:*" are in use.
Here's an excerpt of the query response. Notice this element, which
should not be there, in my opinion:
DisjunctionMaxQuery((name:"* : *"^0.5))
There is a space between * and :, and another space between : and *.
<response>
<lstname="responseHeader">
<intname="status">0</int>
<intname="QTime">33</int>
<lstname="params">
<strname="indent">on</str>
<strname="wt"/>
<strname="version">2.2</str>
<strname="rows">10</str>
<strname="defType">edismax</str>
<strname="pf">name^0.5</str>
<strname="fl">*,score</str>
<strname="debugQuery">on</str>
<strname="start">0</str>
<strname="q">*:*</str>
<strname="qt"/>
<strname="fq"/>
</lst>
</lst>
<resultname="response"numFound="32"start="0"maxScore="0.14764866">
<doc>
<strname="id">GB18030TEST</str>
<strname="name">Test with some GB18030 encoded characters</str>
<arrname="features">
<str>No accents here</str>
<str>这是一个功能</str>
<str>This is a feature (translated)</str>
<str>这份文件是很有光泽</str>
<str>This document is very shiny (translated)</str>
</arr>
<floatname="price">0.0</float>
<strname="price_c">0,USD</str>
<boolname="inStock">true</bool>
<longname="_version_">1415830106215022592</long>
<floatname="score">0.14764866</float>
</doc>
...
</result>
<lstname="debug">
<strname="rawquerystring">*:*</str>
<strname="querystring">*:*</str>
<strname="parsedquery">
(+MatchAllDocsQuery(*:*) DisjunctionMaxQuery((name:"* : *"^0.5)))/no_coord
</str>
<strname="parsedquery_toString">+*:* (name:"* : *"^0.5)</str>
<lstname="explain">
<strname="GB18030TEST">
0.14764866 = (MATCH) sum of: 0.14764866 = (MATCH) MatchAllDocsQuery,
product of: 0.14764866 = queryNorm
</str>
</lst>
<strname="QParser">ExtendedDismaxQParser</str>
<nullname="altquerystring"/>
<nullname="boostfuncs"/>
...
</lst>
</lst>
</lst>
</response>
Re: Any filter to map mutiple tokens into one ?
Posted by Jack Krupansky <ja...@basetechnology.com>.
There's a miscommunication here somewhere. Is Solr 4.0 still passing "*:*"
to the analyzer? Show us the parsed query for "*:*", as well as the
debugQuery "explain" for the score.
I mean, "*:*" (MatchAllDocsQuery) has a "constant score", so there isn't any
way for it to be "suboptimal".
-- Jack Krupansky
-----Original Message-----
From: T. Kuro Kurosaka
Sent: Sunday, October 14, 2012 3:01 PM
To: solr-user@lucene.apache.org
Subject: Re: Any filter to map mutiple tokens into one ?
Jack,
I don't think SOLR-3261 describes this issue.
I ran the same experiment with Solr-3.6 and the score for all the
matches was 0.1626374.
The newly released Solr 4.0.0 also returns a suboptimal score of 0.14764866.
Kuro
On 10/12/12 2:03 PM, Jack Krupansky wrote:
> I don't have a Solr 3.5 to check, but SOLR-3261, which was fixed in
> Solr 3.6 may be your culprit.
>
> See:
> https://issues.apache.org/jira/browse/SOLR-3261
>
> So, try SOlr 3.6 or 3.6.1 or 4.0 to see if your issue goes away.
>
> -- Jack Krupansky
>
> -----Original Message----- From: T. Kuro Kurosaka
> Sent: Friday, October 12, 2012 3:15 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Any filter to map mutiple tokens into one ?
>
> Jack,
> It goes like this:
>
> http://myhost:8983/solr/select?indent=on&version=2.2&q=*%3A*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=&debugQuery=on
>
>
> and edismax is the default query parser in solrconfig.xml.
>
> There is a field named text_jpn that uses a Tokenizer that we developed
> as a product, which we can't share here.
>
> But I can simulate our situation using NGramTokenizer.
> After indexing the Solr sample docs normally, stop the Solr and insert:
>
> <fieldtype name="text_fake" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.NGramTokenizerFactory"
> maxGramSize="1"
> minGramSize="1" />
> </analyzer>
> </fieldtype>
>
> Replace the field definition for "name", for example:
> <field name="name" type="text_fake" indexed="true" stored="true"/>
>
> In solrconfig.xml, change the default search handler's definition like
> this:
> <str name="defType">edismax</str>
> <str name="pf">name^0.5</str>
> (I guess I could just have these in the URL.)
>
> Start Solr and give this URL:
>
> http://localhost:8983/solr/select?indent=on&version=2.2&q=*%3A*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=&debugQuery=on&explainOther=&hl.fl=
>
>
> Hopefully you'll see
> <floatname="score">0.3663672</float>
> and
> +MatchAllDocsQuery(*:*) DisjunctionMaxQuery((name:"* : *"^0.5))
>
> in the debug output.
>
> The score calculation should not be done when the query is *:* which has
> the special meaning, should it ?
> And even if the score calculation is done, "*:*" shouldn't be fed to
> Tokenizers, should it?
>
> On 10/12/12 9:44 AM, Jack Krupansky wrote:
>> Okay, let's back up. First, hold off mixing in your proposed solution
>> until after we understand the actual, original problem:
>>
>> 1. What is your field and field type (with analyzer details)?
>> 2. What is your query parser (defType)?
>> 3. What is your query request URL?
>> 4. What is the parsed query (add &debugQuery=true to your query
>> request)? (Actually, I think you gave us that)
>>
>> I just tried the following query with the fresh 4.0 release and it
>> works fine:
>>
>> http://localhost:8983/solr/collection1/select?q=*:*&wt=xml&debugQuery=true&defType=edismax
>>
>>
>>
>> <str name="rawquerystring">*:*</str>
>>
>> The parsed query is:
>>
>> <str name="parsedquery">(+MatchAllDocsQuery(*:*))/no_coord</str>
>>
>> And this was with the 4.0 example schema, adding *.xml and books.json
>> documents.
>>
>> If you could try your scenario with 4.0 that would be a help. If it's
>> a bug in 3.5 that is fixed now... oh well. I mean, feel free to check
>> the revision history for edismax since the 3.5 release.
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: T. Kuro Kurosaka
>> Sent: Friday, October 12, 2012 11:54 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Any filter to map mutiple tokens into one ?
>>
>> On 10/11/12 4:47 PM, Jack Krupansky wrote:
>>> The ":" which normally separates a field name from a term (or quoted
>>> string or parenthesized sub-query) is "parsed" by the query parser
>>> before analysis gets called, and "*:*" is recognized before analysis
>>> as well. So, any attempt to recreate "*:*" in analysis will be too
>>> late to affect query parsing and other pre-analysis processing.
>> That's why I suspect a bug in Solr. Tokenizer shouldn't play any roles
>> here but it is affecting the score calculation. I am seeing an evidence
>> that "*:*" is being passed to my tokenizer.
>> I'm trying to find a way to work around this by reconstructing "*:*" in
>> the analysis chain.
>>>
>>> But, what is it you are really trying to do? What's the real problem?
>>> (This sounds like a proverbial "XY Problem".)
>>>
>>> -- Jack Krupansky
>>>
>>> -----Original Message----- From: T. Kuro Kurosaka
>>> Sent: Thursday, October 11, 2012 7:35 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Any filter to map mutiple tokens into one ?
>>>
>>> I am looking for a way to fold a particular sequence of tokens into one
>>> token.
>>> Concretely, I'd like to detect a three-token sequence of "*", ":" and
>>> "*", and replace it with a token of the text "*:*".
>>> I tried SynonymFIlter but it seems it can only deal with a single input
>>> token. "* : * => *:*" seems to be interpreted
>>> as one input token of 5 characters "*", space, ":", space and "*".
>>>
>>> I'm using Solr 3.5.
>>>
>>> Background:
>>> My tokenizer separate the three character sequence "*:*" into 3 tokens
>>> of one character each.
>>> The edismax parser, when given the query "*:*", i.e. find every doc,
>>> seems to pass the entire string "*:*" to the query analyzer (I suspect
>>> a bug.),
>>> and feed the tokenized result to DisjunctionMaxQuery object,
>>> according to this debug output:
>>>
>>> <lst name="debug">
>>> <str name="rawquerystring">*:*</str>
>>> <str name="querystring">*:*</str>
>>> <str name="parsedquery">+MatchAllDocsQuery(*:*)
>>> DisjunctionMaxQuery((body:"* : *"~100^0.5 | title:"* :
>>> *"~100^1.2)~0.01)</str>
>>> <str name="parsedquery_toString">+*:* (body:"* : *"~100^0.5 |
>>> title:"* :
>>> *"~100^1.2)~0.01</str>
>>>
>>> Notice that there is a space between * and : in
>>> DisjunctionMaxQuery((body:"* : *" ....)
>>>
>>> Probably because of this, the hit score is as low as 0.109, while it is
>>> 1.000 if an analyzer that doesn't break "*:*" is used.
>>> So I'd like to stitch together "*", ":", "*" into "*:*" again to make
>>> DisjunctionMaxQuery happy.
>>>
>>>
>>> Thanks.
>>>
>>>
>>> T. "Kuro" Kurosaka
>>>
>>
>
Re: Any filter to map mutiple tokens into one ?
Posted by "T. Kuro Kurosaka" <ku...@basistech.com>.
Jack,
I don't think SOLR-3261 describes this issue.
I ran the same experiment with Solr-3.6 and the score for all the
matches was 0.1626374.
The newly released Solr 4.0.0 also returns a suboptimal score of 0.14764866.
Kuro
On 10/12/12 2:03 PM, Jack Krupansky wrote:
> I don't have a Solr 3.5 to check, but SOLR-3261, which was fixed in
> Solr 3.6 may be your culprit.
>
> See:
> https://issues.apache.org/jira/browse/SOLR-3261
>
> So, try SOlr 3.6 or 3.6.1 or 4.0 to see if your issue goes away.
>
> -- Jack Krupansky
>
> -----Original Message----- From: T. Kuro Kurosaka
> Sent: Friday, October 12, 2012 3:15 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Any filter to map mutiple tokens into one ?
>
> Jack,
> It goes like this:
>
> http://myhost:8983/solr/select?indent=on&version=2.2&q=*%3A*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=&debugQuery=on
>
>
> and edismax is the default query parser in solrconfig.xml.
>
> There is a field named text_jpn that uses a Tokenizer that we developed
> as a product, which we can't share here.
>
> But I can simulate our situation using NGramTokenizer.
> After indexing the Solr sample docs normally, stop the Solr and insert:
>
> <fieldtype name="text_fake" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.NGramTokenizerFactory"
> maxGramSize="1"
> minGramSize="1" />
> </analyzer>
> </fieldtype>
>
> Replace the field definition for "name", for example:
> <field name="name" type="text_fake" indexed="true" stored="true"/>
>
> In solrconfig.xml, change the default search handler's definition like
> this:
> <str name="defType">edismax</str>
> <str name="pf">name^0.5</str>
> (I guess I could just have these in the URL.)
>
> Start Solr and give this URL:
>
> http://localhost:8983/solr/select?indent=on&version=2.2&q=*%3A*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=&debugQuery=on&explainOther=&hl.fl=
>
>
> Hopefully you'll see
> <floatname="score">0.3663672</float>
> and
> +MatchAllDocsQuery(*:*) DisjunctionMaxQuery((name:"* : *"^0.5))
>
> in the debug output.
>
> The score calculation should not be done when the query is *:* which has
> the special meaning, should it ?
> And even if the score calculation is done, "*:*" shouldn't be fed to
> Tokenizers, should it?
>
> On 10/12/12 9:44 AM, Jack Krupansky wrote:
>> Okay, let's back up. First, hold off mixing in your proposed solution
>> until after we understand the actual, original problem:
>>
>> 1. What is your field and field type (with analyzer details)?
>> 2. What is your query parser (defType)?
>> 3. What is your query request URL?
>> 4. What is the parsed query (add &debugQuery=true to your query
>> request)? (Actually, I think you gave us that)
>>
>> I just tried the following query with the fresh 4.0 release and it
>> works fine:
>>
>> http://localhost:8983/solr/collection1/select?q=*:*&wt=xml&debugQuery=true&defType=edismax
>>
>>
>>
>> <str name="rawquerystring">*:*</str>
>>
>> The parsed query is:
>>
>> <str name="parsedquery">(+MatchAllDocsQuery(*:*))/no_coord</str>
>>
>> And this was with the 4.0 example schema, adding *.xml and books.json
>> documents.
>>
>> If you could try your scenario with 4.0 that would be a help. If it's
>> a bug in 3.5 that is fixed now... oh well. I mean, feel free to check
>> the revision history for edismax since the 3.5 release.
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: T. Kuro Kurosaka
>> Sent: Friday, October 12, 2012 11:54 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Any filter to map mutiple tokens into one ?
>>
>> On 10/11/12 4:47 PM, Jack Krupansky wrote:
>>> The ":" which normally separates a field name from a term (or quoted
>>> string or parenthesized sub-query) is "parsed" by the query parser
>>> before analysis gets called, and "*:*" is recognized before analysis
>>> as well. So, any attempt to recreate "*:*" in analysis will be too
>>> late to affect query parsing and other pre-analysis processing.
>> That's why I suspect a bug in Solr. Tokenizer shouldn't play any roles
>> here but it is affecting the score calculation. I am seeing an evidence
>> that "*:*" is being passed to my tokenizer.
>> I'm trying to find a way to work around this by reconstructing "*:*" in
>> the analysis chain.
>>>
>>> But, what is it you are really trying to do? What's the real problem?
>>> (This sounds like a proverbial "XY Problem".)
>>>
>>> -- Jack Krupansky
>>>
>>> -----Original Message----- From: T. Kuro Kurosaka
>>> Sent: Thursday, October 11, 2012 7:35 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Any filter to map mutiple tokens into one ?
>>>
>>> I am looking for a way to fold a particular sequence of tokens into one
>>> token.
>>> Concretely, I'd like to detect a three-token sequence of "*", ":" and
>>> "*", and replace it with a token of the text "*:*".
>>> I tried SynonymFIlter but it seems it can only deal with a single input
>>> token. "* : * => *:*" seems to be interpreted
>>> as one input token of 5 characters "*", space, ":", space and "*".
>>>
>>> I'm using Solr 3.5.
>>>
>>> Background:
>>> My tokenizer separate the three character sequence "*:*" into 3 tokens
>>> of one character each.
>>> The edismax parser, when given the query "*:*", i.e. find every doc,
>>> seems to pass the entire string "*:*" to the query analyzer (I suspect
>>> a bug.),
>>> and feed the tokenized result to DisjunctionMaxQuery object,
>>> according to this debug output:
>>>
>>> <lst name="debug">
>>> <str name="rawquerystring">*:*</str>
>>> <str name="querystring">*:*</str>
>>> <str name="parsedquery">+MatchAllDocsQuery(*:*)
>>> DisjunctionMaxQuery((body:"* : *"~100^0.5 | title:"* :
>>> *"~100^1.2)~0.01)</str>
>>> <str name="parsedquery_toString">+*:* (body:"* : *"~100^0.5 |
>>> title:"* :
>>> *"~100^1.2)~0.01</str>
>>>
>>> Notice that there is a space between * and : in
>>> DisjunctionMaxQuery((body:"* : *" ....)
>>>
>>> Probably because of this, the hit score is as low as 0.109, while it is
>>> 1.000 if an analyzer that doesn't break "*:*" is used.
>>> So I'd like to stitch together "*", ":", "*" into "*:*" again to make
>>> DisjunctionMaxQuery happy.
>>>
>>>
>>> Thanks.
>>>
>>>
>>> T. "Kuro" Kurosaka
>>>
>>
>
Re: Any filter to map mutiple tokens into one ?
Posted by Jack Krupansky <ja...@basetechnology.com>.
I don't have a Solr 3.5 to check, but SOLR-3261, which was fixed in Solr 3.6
may be your culprit.
See:
https://issues.apache.org/jira/browse/SOLR-3261
So, try SOlr 3.6 or 3.6.1 or 4.0 to see if your issue goes away.
-- Jack Krupansky
-----Original Message-----
From: T. Kuro Kurosaka
Sent: Friday, October 12, 2012 3:15 PM
To: solr-user@lucene.apache.org
Subject: Re: Any filter to map mutiple tokens into one ?
Jack,
It goes like this:
http://myhost:8983/solr/select?indent=on&version=2.2&q=*%3A*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=&debugQuery=on
and edismax is the default query parser in solrconfig.xml.
There is a field named text_jpn that uses a Tokenizer that we developed
as a product, which we can't share here.
But I can simulate our situation using NGramTokenizer.
After indexing the Solr sample docs normally, stop the Solr and insert:
<fieldtype name="text_fake" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.NGramTokenizerFactory"
maxGramSize="1"
minGramSize="1" />
</analyzer>
</fieldtype>
Replace the field definition for "name", for example:
<field name="name" type="text_fake" indexed="true" stored="true"/>
In solrconfig.xml, change the default search handler's definition like this:
<str name="defType">edismax</str>
<str name="pf">name^0.5</str>
(I guess I could just have these in the URL.)
Start Solr and give this URL:
http://localhost:8983/solr/select?indent=on&version=2.2&q=*%3A*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=&debugQuery=on&explainOther=&hl.fl=
Hopefully you'll see
<floatname="score">0.3663672</float>
and
+MatchAllDocsQuery(*:*) DisjunctionMaxQuery((name:"* : *"^0.5))
in the debug output.
The score calculation should not be done when the query is *:* which has
the special meaning, should it ?
And even if the score calculation is done, "*:*" shouldn't be fed to
Tokenizers, should it?
On 10/12/12 9:44 AM, Jack Krupansky wrote:
> Okay, let's back up. First, hold off mixing in your proposed solution
> until after we understand the actual, original problem:
>
> 1. What is your field and field type (with analyzer details)?
> 2. What is your query parser (defType)?
> 3. What is your query request URL?
> 4. What is the parsed query (add &debugQuery=true to your query
> request)? (Actually, I think you gave us that)
>
> I just tried the following query with the fresh 4.0 release and it
> works fine:
>
> http://localhost:8983/solr/collection1/select?q=*:*&wt=xml&debugQuery=true&defType=edismax
>
>
> <str name="rawquerystring">*:*</str>
>
> The parsed query is:
>
> <str name="parsedquery">(+MatchAllDocsQuery(*:*))/no_coord</str>
>
> And this was with the 4.0 example schema, adding *.xml and books.json
> documents.
>
> If you could try your scenario with 4.0 that would be a help. If it's
> a bug in 3.5 that is fixed now... oh well. I mean, feel free to check
> the revision history for edismax since the 3.5 release.
>
> -- Jack Krupansky
>
> -----Original Message----- From: T. Kuro Kurosaka
> Sent: Friday, October 12, 2012 11:54 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Any filter to map mutiple tokens into one ?
>
> On 10/11/12 4:47 PM, Jack Krupansky wrote:
>> The ":" which normally separates a field name from a term (or quoted
>> string or parenthesized sub-query) is "parsed" by the query parser
>> before analysis gets called, and "*:*" is recognized before analysis
>> as well. So, any attempt to recreate "*:*" in analysis will be too
>> late to affect query parsing and other pre-analysis processing.
> That's why I suspect a bug in Solr. Tokenizer shouldn't play any roles
> here but it is affecting the score calculation. I am seeing an evidence
> that "*:*" is being passed to my tokenizer.
> I'm trying to find a way to work around this by reconstructing "*:*" in
> the analysis chain.
>>
>> But, what is it you are really trying to do? What's the real problem?
>> (This sounds like a proverbial "XY Problem".)
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: T. Kuro Kurosaka
>> Sent: Thursday, October 11, 2012 7:35 PM
>> To: solr-user@lucene.apache.org
>> Subject: Any filter to map mutiple tokens into one ?
>>
>> I am looking for a way to fold a particular sequence of tokens into one
>> token.
>> Concretely, I'd like to detect a three-token sequence of "*", ":" and
>> "*", and replace it with a token of the text "*:*".
>> I tried SynonymFIlter but it seems it can only deal with a single input
>> token. "* : * => *:*" seems to be interpreted
>> as one input token of 5 characters "*", space, ":", space and "*".
>>
>> I'm using Solr 3.5.
>>
>> Background:
>> My tokenizer separate the three character sequence "*:*" into 3 tokens
>> of one character each.
>> The edismax parser, when given the query "*:*", i.e. find every doc,
>> seems to pass the entire string "*:*" to the query analyzer (I suspect
>> a bug.),
>> and feed the tokenized result to DisjunctionMaxQuery object,
>> according to this debug output:
>>
>> <lst name="debug">
>> <str name="rawquerystring">*:*</str>
>> <str name="querystring">*:*</str>
>> <str name="parsedquery">+MatchAllDocsQuery(*:*)
>> DisjunctionMaxQuery((body:"* : *"~100^0.5 | title:"* :
>> *"~100^1.2)~0.01)</str>
>> <str name="parsedquery_toString">+*:* (body:"* : *"~100^0.5 | title:"* :
>> *"~100^1.2)~0.01</str>
>>
>> Notice that there is a space between * and : in
>> DisjunctionMaxQuery((body:"* : *" ....)
>>
>> Probably because of this, the hit score is as low as 0.109, while it is
>> 1.000 if an analyzer that doesn't break "*:*" is used.
>> So I'd like to stitch together "*", ":", "*" into "*:*" again to make
>> DisjunctionMaxQuery happy.
>>
>>
>> Thanks.
>>
>>
>> T. "Kuro" Kurosaka
>>
>
Re: Any filter to map mutiple tokens into one ?
Posted by "T. Kuro Kurosaka" <ku...@basistech.com>.
Jack,
It goes like this:
http://myhost:8983/solr/select?indent=on&version=2.2&q=*%3A*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=&debugQuery=on
and edismax is the default query parser in solrconfig.xml.
There is a field named text_jpn that uses a Tokenizer that we developed
as a product, which we can't share here.
But I can simulate our situation using NGramTokenizer.
After indexing the Solr sample docs normally, stop the Solr and insert:
<fieldtype name="text_fake" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.NGramTokenizerFactory"
maxGramSize="1"
minGramSize="1" />
</analyzer>
</fieldtype>
Replace the field definition for "name", for example:
<field name="name" type="text_fake" indexed="true" stored="true"/>
In solrconfig.xml, change the default search handler's definition like this:
<str name="defType">edismax</str>
<str name="pf">name^0.5</str>
(I guess I could just have these in the URL.)
Start Solr and give this URL:
http://localhost:8983/solr/select?indent=on&version=2.2&q=*%3A*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=&debugQuery=on&explainOther=&hl.fl=
Hopefully you'll see
<floatname="score">0.3663672</float>
and
+MatchAllDocsQuery(*:*) DisjunctionMaxQuery((name:"* : *"^0.5))
in the debug output.
The score calculation should not be done when the query is *:* which has
the special meaning, should it ?
And even if the score calculation is done, "*:*" shouldn't be fed to
Tokenizers, should it?
On 10/12/12 9:44 AM, Jack Krupansky wrote:
> Okay, let's back up. First, hold off mixing in your proposed solution
> until after we understand the actual, original problem:
>
> 1. What is your field and field type (with analyzer details)?
> 2. What is your query parser (defType)?
> 3. What is your query request URL?
> 4. What is the parsed query (add &debugQuery=true to your query
> request)? (Actually, I think you gave us that)
>
> I just tried the following query with the fresh 4.0 release and it
> works fine:
>
> http://localhost:8983/solr/collection1/select?q=*:*&wt=xml&debugQuery=true&defType=edismax
>
>
> <str name="rawquerystring">*:*</str>
>
> The parsed query is:
>
> <str name="parsedquery">(+MatchAllDocsQuery(*:*))/no_coord</str>
>
> And this was with the 4.0 example schema, adding *.xml and books.json
> documents.
>
> If you could try your scenario with 4.0 that would be a help. If it's
> a bug in 3.5 that is fixed now... oh well. I mean, feel free to check
> the revision history for edismax since the 3.5 release.
>
> -- Jack Krupansky
>
> -----Original Message----- From: T. Kuro Kurosaka
> Sent: Friday, October 12, 2012 11:54 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Any filter to map mutiple tokens into one ?
>
> On 10/11/12 4:47 PM, Jack Krupansky wrote:
>> The ":" which normally separates a field name from a term (or quoted
>> string or parenthesized sub-query) is "parsed" by the query parser
>> before analysis gets called, and "*:*" is recognized before analysis
>> as well. So, any attempt to recreate "*:*" in analysis will be too
>> late to affect query parsing and other pre-analysis processing.
> That's why I suspect a bug in Solr. Tokenizer shouldn't play any roles
> here but it is affecting the score calculation. I am seeing an evidence
> that "*:*" is being passed to my tokenizer.
> I'm trying to find a way to work around this by reconstructing "*:*" in
> the analysis chain.
>>
>> But, what is it you are really trying to do? What's the real problem?
>> (This sounds like a proverbial "XY Problem".)
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: T. Kuro Kurosaka
>> Sent: Thursday, October 11, 2012 7:35 PM
>> To: solr-user@lucene.apache.org
>> Subject: Any filter to map mutiple tokens into one ?
>>
>> I am looking for a way to fold a particular sequence of tokens into one
>> token.
>> Concretely, I'd like to detect a three-token sequence of "*", ":" and
>> "*", and replace it with a token of the text "*:*".
>> I tried SynonymFIlter but it seems it can only deal with a single input
>> token. "* : * => *:*" seems to be interpreted
>> as one input token of 5 characters "*", space, ":", space and "*".
>>
>> I'm using Solr 3.5.
>>
>> Background:
>> My tokenizer separate the three character sequence "*:*" into 3 tokens
>> of one character each.
>> The edismax parser, when given the query "*:*", i.e. find every doc,
>> seems to pass the entire string "*:*" to the query analyzer (I suspect
>> a bug.),
>> and feed the tokenized result to DisjunctionMaxQuery object,
>> according to this debug output:
>>
>> <lst name="debug">
>> <str name="rawquerystring">*:*</str>
>> <str name="querystring">*:*</str>
>> <str name="parsedquery">+MatchAllDocsQuery(*:*)
>> DisjunctionMaxQuery((body:"* : *"~100^0.5 | title:"* :
>> *"~100^1.2)~0.01)</str>
>> <str name="parsedquery_toString">+*:* (body:"* : *"~100^0.5 | title:"* :
>> *"~100^1.2)~0.01</str>
>>
>> Notice that there is a space between * and : in
>> DisjunctionMaxQuery((body:"* : *" ....)
>>
>> Probably because of this, the hit score is as low as 0.109, while it is
>> 1.000 if an analyzer that doesn't break "*:*" is used.
>> So I'd like to stitch together "*", ":", "*" into "*:*" again to make
>> DisjunctionMaxQuery happy.
>>
>>
>> Thanks.
>>
>>
>> T. "Kuro" Kurosaka
>>
>
Re: Any filter to map mutiple tokens into one ?
Posted by Jack Krupansky <ja...@basetechnology.com>.
Okay, let's back up. First, hold off mixing in your proposed solution until
after we understand the actual, original problem:
1. What is your field and field type (with analyzer details)?
2. What is your query parser (defType)?
3. What is your query request URL?
4. What is the parsed query (add &debugQuery=true to your query request)?
(Actually, I think you gave us that)
I just tried the following query with the fresh 4.0 release and it works
fine:
http://localhost:8983/solr/collection1/select?q=*:*&wt=xml&debugQuery=true&defType=edismax
<str name="rawquerystring">*:*</str>
The parsed query is:
<str name="parsedquery">(+MatchAllDocsQuery(*:*))/no_coord</str>
And this was with the 4.0 example schema, adding *.xml and books.json
documents.
If you could try your scenario with 4.0 that would be a help. If it's a bug
in 3.5 that is fixed now... oh well. I mean, feel free to check the revision
history for edismax since the 3.5 release.
-- Jack Krupansky
-----Original Message-----
From: T. Kuro Kurosaka
Sent: Friday, October 12, 2012 11:54 AM
To: solr-user@lucene.apache.org
Subject: Re: Any filter to map mutiple tokens into one ?
On 10/11/12 4:47 PM, Jack Krupansky wrote:
> The ":" which normally separates a field name from a term (or quoted
> string or parenthesized sub-query) is "parsed" by the query parser before
> analysis gets called, and "*:*" is recognized before analysis as well. So,
> any attempt to recreate "*:*" in analysis will be too late to affect query
> parsing and other pre-analysis processing.
That's why I suspect a bug in Solr. Tokenizer shouldn't play any roles
here but it is affecting the score calculation. I am seeing an evidence
that "*:*" is being passed to my tokenizer.
I'm trying to find a way to work around this by reconstructing "*:*" in
the analysis chain.
>
> But, what is it you are really trying to do? What's the real problem?
> (This sounds like a proverbial "XY Problem".)
>
> -- Jack Krupansky
>
> -----Original Message----- From: T. Kuro Kurosaka
> Sent: Thursday, October 11, 2012 7:35 PM
> To: solr-user@lucene.apache.org
> Subject: Any filter to map mutiple tokens into one ?
>
> I am looking for a way to fold a particular sequence of tokens into one
> token.
> Concretely, I'd like to detect a three-token sequence of "*", ":" and
> "*", and replace it with a token of the text "*:*".
> I tried SynonymFIlter but it seems it can only deal with a single input
> token. "* : * => *:*" seems to be interpreted
> as one input token of 5 characters "*", space, ":", space and "*".
>
> I'm using Solr 3.5.
>
> Background:
> My tokenizer separate the three character sequence "*:*" into 3 tokens
> of one character each.
> The edismax parser, when given the query "*:*", i.e. find every doc,
> seems to pass the entire string "*:*" to the query analyzer (I suspect
> a bug.),
> and feed the tokenized result to DisjunctionMaxQuery object,
> according to this debug output:
>
> <lst name="debug">
> <str name="rawquerystring">*:*</str>
> <str name="querystring">*:*</str>
> <str name="parsedquery">+MatchAllDocsQuery(*:*)
> DisjunctionMaxQuery((body:"* : *"~100^0.5 | title:"* :
> *"~100^1.2)~0.01)</str>
> <str name="parsedquery_toString">+*:* (body:"* : *"~100^0.5 | title:"* :
> *"~100^1.2)~0.01</str>
>
> Notice that there is a space between * and : in
> DisjunctionMaxQuery((body:"* : *" ....)
>
> Probably because of this, the hit score is as low as 0.109, while it is
> 1.000 if an analyzer that doesn't break "*:*" is used.
> So I'd like to stitch together "*", ":", "*" into "*:*" again to make
> DisjunctionMaxQuery happy.
>
>
> Thanks.
>
>
> T. "Kuro" Kurosaka
>
Re: Any filter to map mutiple tokens into one ?
Posted by "T. Kuro Kurosaka" <ku...@basistech.com>.
On 10/11/12 4:47 PM, Jack Krupansky wrote:
> The ":" which normally separates a field name from a term (or quoted
> string or parenthesized sub-query) is "parsed" by the query parser
> before analysis gets called, and "*:*" is recognized before analysis
> as well. So, any attempt to recreate "*:*" in analysis will be too
> late to affect query parsing and other pre-analysis processing.
That's why I suspect a bug in Solr. Tokenizer shouldn't play any roles
here but it is affecting the score calculation. I am seeing an evidence
that "*:*" is being passed to my tokenizer.
I'm trying to find a way to work around this by reconstructing "*:*" in
the analysis chain.
>
> But, what is it you are really trying to do? What's the real problem?
> (This sounds like a proverbial "XY Problem".)
>
> -- Jack Krupansky
>
> -----Original Message----- From: T. Kuro Kurosaka
> Sent: Thursday, October 11, 2012 7:35 PM
> To: solr-user@lucene.apache.org
> Subject: Any filter to map mutiple tokens into one ?
>
> I am looking for a way to fold a particular sequence of tokens into one
> token.
> Concretely, I'd like to detect a three-token sequence of "*", ":" and
> "*", and replace it with a token of the text "*:*".
> I tried SynonymFIlter but it seems it can only deal with a single input
> token. "* : * => *:*" seems to be interpreted
> as one input token of 5 characters "*", space, ":", space and "*".
>
> I'm using Solr 3.5.
>
> Background:
> My tokenizer separate the three character sequence "*:*" into 3 tokens
> of one character each.
> The edismax parser, when given the query "*:*", i.e. find every doc,
> seems to pass the entire string "*:*" to the query analyzer (I suspect
> a bug.),
> and feed the tokenized result to DisjunctionMaxQuery object,
> according to this debug output:
>
> <lst name="debug">
> <str name="rawquerystring">*:*</str>
> <str name="querystring">*:*</str>
> <str name="parsedquery">+MatchAllDocsQuery(*:*)
> DisjunctionMaxQuery((body:"* : *"~100^0.5 | title:"* :
> *"~100^1.2)~0.01)</str>
> <str name="parsedquery_toString">+*:* (body:"* : *"~100^0.5 | title:"* :
> *"~100^1.2)~0.01</str>
>
> Notice that there is a space between * and : in
> DisjunctionMaxQuery((body:"* : *" ....)
>
> Probably because of this, the hit score is as low as 0.109, while it is
> 1.000 if an analyzer that doesn't break "*:*" is used.
> So I'd like to stitch together "*", ":", "*" into "*:*" again to make
> DisjunctionMaxQuery happy.
>
>
> Thanks.
>
>
> T. "Kuro" Kurosaka
>
Re: Any filter to map mutiple tokens into one ?
Posted by Jack Krupansky <ja...@basetechnology.com>.
The ":" which normally separates a field name from a term (or quoted string
or parenthesized sub-query) is "parsed" by the query parser before analysis
gets called, and "*:*" is recognized before analysis as well. So, any
attempt to recreate "*:*" in analysis will be too late to affect query
parsing and other pre-analysis processing.
But, what is it you are really trying to do? What's the real problem? (This
sounds like a proverbial "XY Problem".)
-- Jack Krupansky
-----Original Message-----
From: T. Kuro Kurosaka
Sent: Thursday, October 11, 2012 7:35 PM
To: solr-user@lucene.apache.org
Subject: Any filter to map mutiple tokens into one ?
I am looking for a way to fold a particular sequence of tokens into one
token.
Concretely, I'd like to detect a three-token sequence of "*", ":" and
"*", and replace it with a token of the text "*:*".
I tried SynonymFIlter but it seems it can only deal with a single input
token. "* : * => *:*" seems to be interpreted
as one input token of 5 characters "*", space, ":", space and "*".
I'm using Solr 3.5.
Background:
My tokenizer separate the three character sequence "*:*" into 3 tokens
of one character each.
The edismax parser, when given the query "*:*", i.e. find every doc,
seems to pass the entire string "*:*" to the query analyzer (I suspect
a bug.),
and feed the tokenized result to DisjunctionMaxQuery object,
according to this debug output:
<lst name="debug">
<str name="rawquerystring">*:*</str>
<str name="querystring">*:*</str>
<str name="parsedquery">+MatchAllDocsQuery(*:*)
DisjunctionMaxQuery((body:"* : *"~100^0.5 | title:"* :
*"~100^1.2)~0.01)</str>
<str name="parsedquery_toString">+*:* (body:"* : *"~100^0.5 | title:"* :
*"~100^1.2)~0.01</str>
Notice that there is a space between * and : in
DisjunctionMaxQuery((body:"* : *" ....)
Probably because of this, the hit score is as low as 0.109, while it is
1.000 if an analyzer that doesn't break "*:*" is used.
So I'd like to stitch together "*", ":", "*" into "*:*" again to make
DisjunctionMaxQuery happy.
Thanks.
T. "Kuro" Kurosaka