You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by tudor <tu...@gmail.com> on 2012/07/14 11:11:47 UTC
Query results vs. facets results
Hello,
I am new to Solr and I running some tests with our data in Solr. We are
using version 3.6 and the data is imported form a DB2 database using Solr's
DIH. We have defined a single entity in the db-data-config.xml, which is an
equivalent of the following query:
<entity name="connections"
query="
SELECT C.NAME,
F.CITY
FROM
NAME_CONNECTIONS AS C
JOIN NAME_DETAILS AS F
ON C.DETAILS_NAME = F.NAME"
>
</entity>
This might lead to some names appearing multiple times in the result set.
This is OK.
For the unique ID in the schema, we are using a solr.UUIDField:
<fieldType name="uuid" class="solr.UUIDField" indexed="true" />
<field name="id" type="uuid" indexed="true"
stored="true" default="NEW"/
All the searchable fields are declared as indexed and stored.
I am aware of the fact that this is a very crude configuration, but for the
tests that I am running it is fine.
The problem that I have is the different result counts that I receive when I
do equivalent queries for searching and faceting. For example, running the
following query
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=CITY:MILTON&fq=&start=0&rows=100&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=NAME&group.ngroups=true&group.truncate=true
yields
<int name="ngroups">134</int>
as a result, which is exactly what we expect.
On the other hand, running
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=NAME&group.truncate=true&facet=true&facet.field=CITY&group.ngroups=true
yields
<lst name="facet_counts">
<lst name="facet_queries"/>
<lst name="facet_fields">
<lst name="CITY">
<int name="MILTON">103</int>
I would expect to have the same number (134) in this facet result as well.
Could you please let me know why these two results are different?
Thank you,
Tudor
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-results-vs-facets-results-tp3994988.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query results vs. facets results
Posted by tudor <tu...@gmail.com>.
Hi Eric,
Thanks for the reply.
The query:
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=CITY:MILTON&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.ngroups=true&group.truncate=true&debugQuery=on
yields this in the debug section:
<lst name="debug"><str name="rawquerystring">CITY:MILTON</str>
<str name="querystring">CITY:MILTON</str>
<str name="parsedquery">CITY:MILTON</str>
<str name="parsedquery_toString">CITY:MILTON</str>
<str name="QParser">LuceneQParser</str>
in the explain section. There is no information about grouping.
Second query:
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.truncate=true&facet=true&facet.field=CITY&facet.missing=true&group.ngroups=true&debugQuery=on
yields this in the debug section:
<lst name="debug">
<str name="rawquerystring">*</str>
<str name="querystring">*</str>
<str name="parsedquery">ID:*</str>
<str name="parsedquery_toString">ID:*</str>
<str name="QParser">LuceneQParser</str>
To be honest, these do not tell me too much. I would like to see some
information about the grouping, since I believe this is where I am missing
something.
In the mean time, I have combined the two queries above, hoping to make some
sense out of the results. The following query filters all the entries with
the city name "MILTON" and groups together the ones with the same ID. Also,
the query facets the entries on city, grouping the ones with the same ID. So
the results numbers refer to the number of groups.
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq={!tag=dt}CITY:MILTON&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.truncate=true&facet=true&facet.field={!ex=dt}CITY&facet.missing=true&group.ngroups=true&debugQuery=on
yields the same (for me perplexing) results:
<lst name="grouped">
<lst name="ID">
<int name="matches">284</int>
<int name="ngroups">134</int>
(i.e.: fq says: 134 groups with CITY:MILTON)
...
<lst name="facet_counts">
<lst name="facet_queries"/>
<lst name="facet_fields">
...
<int name="MILTON">103</int>
(i.e.: faceted search says: 103 groups with CITY:MILTON)
I really believe that these different results have something to do with the
grouping that Solr makes, but I do not know how to dig into this.
Thank you and best regards,
Tudor
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-results-vs-facets-results-tp3995079p3995150.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query results vs. facets results
Posted by tudor <tu...@gmail.com>.
Erick Erickson wrote
>
> Ahhh, you need to look down another few lines. When you specify fq, there
> should be a section of the debug output like
> <arr name="filter_queries">
> .
> .
> .
> </arr>
>
> where the array is the parsed form of the filter queries. I was thinking
> about
> comparing that with the parsed form of the "q" parameter in the non-filter
> case to see what insight one could gain from that.
>
>
There is no "filter_queries" section because I do not use an fq in the first
two queries. I use one in the combined query, for which you can see the
output further below.
Erick Erickson wrote
>
>
> But there's already one difference, when you use *, you get
> <str name="parsedquery">ID:*</str>
>
> Is it possible that you have some documents that do NOT have an ID field?
> try *:* rather than just *. I'm guessing that your default search field is
> ID
> and you have some documents without an ID field. Not a good guess if ID
> is your <uniqueKey> though......
>
> Try q=*:* -ID:* and see if you get 31 docs.
>
>
All the entries have an ID, so q=*:* -ID:* yielded 0 results.
The ID could appear multiple times, that is the reason behind grouping of
results. Indeed, ID is the default search field.
Erick Erickson wrote
>
>
> Also note that if you _have_ specified ID as your <uniqueKey> _but_ you
> didn't
> re-index afterwards (actually, I'd blow away the entire
> <solrhome>/data directory
> and restart) you may have stale data in there that allowed documents to
> exist
> that do not have uniqueKey fields.....
>
>
For Solr's unique id I use a <fieldType name="uuid" class="solr.UUIDField"
indexed="true" /> field (which, of course, has a different name than the
default search ID), so it should not be a problem.
I have re-indexed the data, and I get somewhat a different result. This is
the query:
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*:*&fq={!tag=dt}CITY:MILTON&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=STR_ENTERPRISE_ID&group.truncate=true&facet=true&facet.field={!ex=dt}CITY&facet.missing=true&group.ngroups=true&debugQuery=on
And the results as well as the debug information:
<lst name="grouped">
<lst name="ID">
<int name="matches">284</int>
<int name="ngroups">134</int>
<arr name="groups"></arr>
...
<lst name="facet_counts">
<lst name="facet_queries"/>
<lst name="facet_fields">
<lst name="CITY">
...
<int name="MILTON">89</int>
...
<lst name="debug">
<str name="rawquerystring">*:*</str>
<str name="querystring">*:*</str>
<str name="parsedquery">MatchAllDocsQuery(*:*)</str>
<str name="parsedquery_toString">*:*</str>
<lst name="explain"></lst>
<str name="QParser">LuceneQParser</str>
<arr name="filter_queries">
<str>{!tag=dt}CITY:MILTON</str>
</arr><arr name="parsed_filter_queries">
<str>CITY:MILTON</str>
</arr>
<lst name="timing"></lst>
</lst>
So now fq says: 134 groups with CITY:MILTON and faceted search says: 83
groups with CITY:MILTON.
How can I see some information about the grouping in Solr?
Thanks Erick!
Regards,
Tudor
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-results-vs-facets-results-tp3995079p3995388.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query results vs. facets results
Posted by Erick Erickson <er...@gmail.com>.
Ahhh, you need to look down another few lines. When you specify fq, there
should be a section of the debug output like
<arr name="filter_queries">
.
.
.
</arr>
where the array is the parsed form of the filter queries. I was thinking about
comparing that with the parsed form of the "q" parameter in the non-filter
case to see what insight one could gain from that.
But there's already one difference, when you use *, you get
<str name="parsedquery">ID:*</str>
Is it possible that you have some documents that do NOT have an ID field?
try *:* rather than just *. I'm guessing that your default search field is ID
and you have some documents without an ID field. Not a good guess if ID
is your <uniqueKey> though......
Try q=*:* -ID:* and see if you get 31 docs.
Also note that if you _have_ specified ID as your <uniqueKey> _but_ you didn't
re-index afterwards (actually, I'd blow away the entire
<solrhome>/data directory
and restart) you may have stale data in there that allowed documents to exist
that do not have uniqueKey fields.....
Best
Erick
On Sun, Jul 15, 2012 at 4:49 PM, tudor <tu...@gmail.com> wrote:
> Hi Erick,
>
> Thanks for the reply.
>
> The query:
>
> http://localhost:8983/solr/db/select?indent=on&version=2.2&q=CITY:MILTON&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.ngroups=true&group.truncate=true&debugQuery=on
>
> yields this in the debug section:
>
> <lst name="debug"><str name="rawquerystring">CITY:MILTON</str>
> <str name="querystring">CITY:MILTON</str>
> <str name="parsedquery">CITY:MILTON</str>
> <str name="parsedquery_toString">CITY:MILTON</str>
> <str name="QParser">LuceneQParser</str>
>
> There is no information about grouping.
>
> Second query:
>
> http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.truncate=true&facet=true&facet.field=CITY&facet.missing=true&group.ngroups=true&debugQuery=on
>
> yields this in the debug section:
>
> <lst name="debug">
> <str name="rawquerystring">*</str>
> <str name="querystring">*</str>
> <str name="parsedquery">ID:*</str>
> <str name="parsedquery_toString">ID:*</str>
> <str name="QParser">LuceneQParser</str>
>
> To be honest, these do not tell me too much. I would like to see some
> information about the grouping, since I believe this is where I am missing
> something.
>
> In the mean time, I have combined the two queries above, hoping to make some
> sense out of the results. The following query filters all the entries with
> the city name "MILTON" and groups together the ones with the same ID. Also,
> the query facets the entries on city, grouping the ones with the same ID. So
> the results numbers refer to the number of groups.
>
> http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq={!tag=dt}CITY:MILTON&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.truncate=true&facet=true&facet.field={!ex=dt}CITY&facet.missing=true&group.ngroups=true&debugQuery=on
>
> yields the same (for me perplexing) results:
>
> <lst name="grouped">
> <lst name="ID">
> <int name="matches">284</int>
> <int name="ngroups">134</int>
>
> (i.e.: fq says: 134 groups with CITY:MILTON)
> ...
>
> <lst name="facet_counts">
> <lst name="facet_queries"/>
> <lst name="facet_fields">
> ...
> <int name="MILTON">103</int>
>
> (i.e.: faceted search says: 103 groups with CITY:MILTON)
>
> I really believe that these different results have something to do with the
> grouping that Solr makes, but I do not know how to dig into this.
>
> Thank you and best regards,
> Tudor
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Query-results-vs-facets-results-tp3995079p3995156.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query results vs. facets results
Posted by tudor <tu...@gmail.com>.
Hi Erick,
Thanks for the reply.
The query:
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=CITY:MILTON&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.ngroups=true&group.truncate=true&debugQuery=on
yields this in the debug section:
<lst name="debug"><str name="rawquerystring">CITY:MILTON</str>
<str name="querystring">CITY:MILTON</str>
<str name="parsedquery">CITY:MILTON</str>
<str name="parsedquery_toString">CITY:MILTON</str>
<str name="QParser">LuceneQParser</str>
There is no information about grouping.
Second query:
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.truncate=true&facet=true&facet.field=CITY&facet.missing=true&group.ngroups=true&debugQuery=on
yields this in the debug section:
<lst name="debug">
<str name="rawquerystring">*</str>
<str name="querystring">*</str>
<str name="parsedquery">ID:*</str>
<str name="parsedquery_toString">ID:*</str>
<str name="QParser">LuceneQParser</str>
To be honest, these do not tell me too much. I would like to see some
information about the grouping, since I believe this is where I am missing
something.
In the mean time, I have combined the two queries above, hoping to make some
sense out of the results. The following query filters all the entries with
the city name "MILTON" and groups together the ones with the same ID. Also,
the query facets the entries on city, grouping the ones with the same ID. So
the results numbers refer to the number of groups.
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq={!tag=dt}CITY:MILTON&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.truncate=true&facet=true&facet.field={!ex=dt}CITY&facet.missing=true&group.ngroups=true&debugQuery=on
yields the same (for me perplexing) results:
<lst name="grouped">
<lst name="ID">
<int name="matches">284</int>
<int name="ngroups">134</int>
(i.e.: fq says: 134 groups with CITY:MILTON)
...
<lst name="facet_counts">
<lst name="facet_queries"/>
<lst name="facet_fields">
...
<int name="MILTON">103</int>
(i.e.: faceted search says: 103 groups with CITY:MILTON)
I really believe that these different results have something to do with the
grouping that Solr makes, but I do not know how to dig into this.
Thank you and best regards,
Tudor
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-results-vs-facets-results-tp3995079p3995156.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query results vs. facets results
Posted by tudor <tu...@gmail.com>.
Hi Erick,
Thanks for the reply.
The query:
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=CITY:MILTON&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.ngroups=true&group.truncate=true&debugQuery=on
yields this in the debug section:
<lst name="debug"><str name="rawquerystring">CITY:MILTON</str>
<str name="querystring">CITY:MILTON</str>
<str name="parsedquery">CITY:MILTON</str>
<str name="parsedquery_toString">CITY:MILTON</str>
<str name="QParser">LuceneQParser</str>
in the explain section. There is no information about grouping.
Second query:
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.truncate=true&facet=true&facet.field=CITY&facet.missing=true&group.ngroups=true&debugQuery=on
yields this in the debug section:
<lst name="debug">
<str name="rawquerystring">*</str>
<str name="querystring">*</str>
<str name="parsedquery">ID:*</str>
<str name="parsedquery_toString">ID:*</str>
<str name="QParser">LuceneQParser</str>
To be honest, these do not tell me too much. I would like to see some
information about the grouping, since I believe this is where I am missing
something.
In the mean time, I have combined the two queries above, hoping to make some
sense out of the results. The following query filters all the entries with
the city name "MILTON" and groups together the ones with the same ID. Also,
the query facets the entries on city, grouping the ones with the same ID. So
the results numbers refer to the number of groups.
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq={!tag=dt}CITY:MILTON&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.truncate=true&facet=true&facet.field={!ex=dt}CITY&facet.missing=true&group.ngroups=true&debugQuery=on
yields the same (for me perplexing) results:
<lst name="grouped">
<lst name="ID">
<int name="matches">284</int>
<int name="ngroups">134</int>
(i.e.: fq says: 134 groups with CITY:MILTON)
...
<lst name="facet_counts">
<lst name="facet_queries"/>
<lst name="facet_fields">
...
<int name="MILTON">103</int>
(i.e.: faceted search says: 103 groups with CITY:MILTON)
I really believe that these different results have something to do with the
grouping that Solr makes, but I do not know how to dig into this.
Thank you and best regards,
Tudor
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-results-vs-facets-results-tp3995079p3995152.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query results vs. facets results
Posted by tudor <tu...@gmail.com>.
Hi Eric,
Thanks for the reply.
The query:
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=CITY:MILTON&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.ngroups=true&group.truncate=true&debugQuery=on
yields this in the debug section:
<lst name="debug"><str name="rawquerystring">CITY:MILTON</str>
<str name="querystring">CITY:MILTON</str>
<str name="parsedquery">CITY:MILTON</str>
<str name="parsedquery_toString">CITY:MILTON</str>
<str name="QParser">LuceneQParser</str>
There is no information about grouping.
Second query:
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.truncate=true&facet=true&facet.field=CITY&facet.missing=true&group.ngroups=true&debugQuery=on
yields this in the debug section:
<lst name="debug">
<str name="rawquerystring">*</str>
<str name="querystring">*</str>
<str name="parsedquery">ID:*</str>
<str name="parsedquery_toString">ID:*</str>
<str name="QParser">LuceneQParser</str>
To be honest, these do not tell me too much. I would like to see some
information about the grouping, since I believe this is where I am missing
something.
In the mean time, I have combined the two queries above, hoping to make some
sense out of the results. The following query filters all the entries with
the city name "MILTON" and groups together the ones with the same ID. Also,
the query facets the entries on city, grouping the ones with the same ID. So
the results numbers refer to the number of groups.
http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq={!tag=dt}CITY:MILTON&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=ID&group.truncate=true&facet=true&facet.field={!ex=dt}CITY&facet.missing=true&group.ngroups=true&debugQuery=on
yields the same (for me perplexing) results:
<lst name="grouped">
<lst name="ID">
<int name="matches">284</int>
<int name="ngroups">134</int>
(i.e.: fq says: 134 groups with CITY:MILTON)
...
<lst name="facet_counts">
<lst name="facet_queries"/>
<lst name="facet_fields">
...
<int name="MILTON">103</int>
(i.e.: faceted search says: 103 groups with CITY:MILTON)
I really believe that these different results have something to do with the
grouping that Solr makes, but I do not know how to dig into this.
Thank you and best regards,
Tudor
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-results-vs-facets-results-tp3995079p3995154.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query results vs. facets results
Posted by Erick Erickson <er...@gmail.com>.
q and fq queries don't necessarily run through the same query parser, see:
http://wiki.apache.org/solr/SimpleFacetParameters#facet.query_:_Arbitrary_Query_Faceting
So try adding &debugQuery=on to both queries you submitted. My guess
is that if you look at the parsed queries, you'll see something that explains
your differences. If not, paste the results back and we can take a look.
BTW, ignore all the "explain" bits for now, the important bit is the parsed form
of q and fq in your queries.
Best
Erick
On Sat, Jul 14, 2012 at 5:11 AM, tudor <tu...@gmail.com> wrote:
> Hello,
>
> I am new to Solr and I running some tests with our data in Solr. We are
> using version 3.6 and the data is imported form a DB2 database using Solr's
> DIH. We have defined a single entity in the db-data-config.xml, which is an
> equivalent of the following query:
> <entity name="connections"
> query="
> SELECT C.NAME,
> F.CITY
> FROM
> NAME_CONNECTIONS AS C
> JOIN NAME_DETAILS AS F
> ON C.DETAILS_NAME = F.NAME"
>>
> </entity>
>
> This might lead to some names appearing multiple times in the result set.
> This is OK.
>
> For the unique ID in the schema, we are using a solr.UUIDField:
>
> <fieldType name="uuid" class="solr.UUIDField" indexed="true" />
> <field name="id" type="uuid" indexed="true"
> stored="true" default="NEW"/
>
> All the searchable fields are declared as indexed and stored.
>
> I am aware of the fact that this is a very crude configuration, but for the
> tests that I am running it is fine.
>
> The problem that I have is the different result counts that I receive when I
> do equivalent queries for searching and faceting. For example, running the
> following query
>
> http://localhost:8983/solr/db/select?indent=on&version=2.2&q=CITY:MILTON&fq=&start=0&rows=100&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=NAME&group.ngroups=true&group.truncate=true
>
> yields
>
> <int name="ngroups">134</int>
>
> as a result, which is exactly what we expect.
>
> On the other hand, running
>
> http://localhost:8983/solr/db/select?indent=on&version=2.2&q=*&fq=&start=0&rows=10&fl=*&wt=&explainOther=&hl.fl=&group=true&group.field=NAME&group.truncate=true&facet=true&facet.field=CITY&group.ngroups=true
>
> yields
>
> <lst name="facet_counts">
> <lst name="facet_queries"/>
> <lst name="facet_fields">
> <lst name="CITY">
> <int name="MILTON">103</int>
>
> I would expect to have the same number (134) in this facet result as well.
> Could you please let me know why these two results are different?
>
> Thank you,
> Tudor
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Query-results-vs-facets-results-tp3994988.html
> Sent from the Solr - User mailing list archive at Nabble.com.