having solr generate and execute other related queries automatically

[gdeconto <ge...@topproducer.com>]

Scenario:

1. I have a query I want to execute; I would be using the results and facets
returned
2. I also have a couple of dozen other queries that are closely related to
the first query and to the facets returned by that query.  For each query, I
would only be using the total number of results documents (aka numFound)
3. steps 1 and 2 would occur for each page load/view

while I could run these queries individually, I am concerned about the
overhead of having to run +20 solr queries.

I was wondering if it was a better idea to extend the search or facet
component somehow so that:

1. I send solr the first query
2. solr executes the query
3. solr then creates (programatically, based on the parameters I pass in the
first query as well as some of the facet results) and executes +20 other
queries
4. solr finally returns me the results and facets of the first query, as
well as the counts/numFounds obtained from the +20 other queries

I was thinking that this would save me the time and roundtrip overhead of
+20 solr queries per page, but I am unsure of how to proceed.

hopefully this question is not too vague.  any help/example appreciated.
-- 
View this message in context: http://old.nabble.com/having-solr-generate-and-execute-other-related-queries-automatically-tp26327032p26327032.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: having solr generate and execute other related queries automatically

[gdeconto <ge...@topproducer.com>]

tpunder wrote:
> 
> Maybe I misunderstand what you are trying to do (or the facet.query
> feature).  If I did an initial query on my data-set that left me with the
> following questions:
> ...
> http://localhost:8983/solr/select/?q=*%3A*&start=0&rows=0&facet=on&facet.query=brand_id:1&facet.query=brand_id:2&facet.query=+%2Bbrand_id:5+%2Bcategory_id:4051
> ...
> 

Thanks for the reply Tim.

I can't provide you with an example as I dont have anything prototyped as
yet; I am still trying to work things thru in my head.  The +20 queries
would allow us to suggest other possibilities to users in a facet-like way
(but not returning the exact same info as facets).

With the technique you mention I would have to specify the list of query
params for each facet.query.  That would work for relatively simple queries. 
Unfortunately, the queries I was looking at doing would be fairly long (say
hundreds of AND/OR statements).   That said, I dont think solr would be able
to handle the query size I would end up with (at least not efficiently),
because the resulting query would consist of thousands of AND/OR statements
(isnt there a limit of sorts in Solr?)

I think that my best bet would be to extend the SearchComponent and perform
the additional query generation and execution in the extension.  That
approach should also allow me to have access to the facet values that the
base query would generate (which would allow me to generate and execute the
other queries).

thx again.
-- 
View this message in context: http://old.nabble.com/having-solr-generate-and-execute-other-related-queries-automatically-tp26327032p26343409.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: having solr generate and execute other related queries automatically

[Tim Underwood <ti...@gmail.com>]

>
>
> Unfortunately no.  the +20 queries are distinct from each other, even tho
> they share some of the original query parameters (and some facet
> information
> from the original query facets).
>
> what I was envisioning was something that works like a facet, but instead
> of
> returning information about the first query, it would return information
> about queries similar to the first query.
>

Maybe I misunderstand what you are trying to do (or the facet.query
feature).  If I did an initial query on my data-set that left me with the
following questions:

1. How many products are in brand 1?
2. How many products are in brand 2?
3. How many products are in brand 5 and category 4051?
4. etc...  (however many other arbitrary queries I want to get counts for)

I could use facet.query parameters to answer those with something like:

http://localhost:8983/solr/select/?q=*%3A*&start=0&rows=0&facet=on&facet.query=brand_id:1&facet.query=brand_id:2&facet.query=+%2Bbrand_id:5+%2Bcategory_id:4051

Where the parameters are:

q=*:*
start=0
rows=0
facet=on
facet.query=brand_id:1
facet.query=brand_id:2
facet.query=+brand_id:5 +category_id:4051

My response looks like:

<response>
<result name="response" numFound="926626" start="0"/>
<lst name="facet_counts">
 <lst name="facet_queries">
  <int name="brand_id:1">1450</int>
  <int name="brand_id:2">1047</int>
  <int name=" +brand_id:5 +category_id:4051">21</int>
 </lst>
 <lst name="facet_fields"/>
 <lst name="facet_dates"/>
</lst>
</response>


Are you talking about a different problem?  Do you have a simple example?

-Tim

Re: having solr generate and execute other related queries automatically

[gdeconto <ge...@topproducer.com>]

tpunder wrote:
> 
> Could you use the facet.query feature
> (http://wiki.apache.org/solr/SimpleFacetParameters#facet.query_:_Arbitrary_Query_Faceting)
> to reduce it to 2 queries?
> 
> So you'd:
> 
> 1. Send solr the first query
> 2. Solr executes and returns the query to you
> 3. You then use the facet results to create a 2nd query with your +20
> facet.query queries
> 3. Solr executes and returns the results of your +20 facet.query queries
> 
> Then you'd only have 2 roundtrips.
> 
> -Tim
> 

Unfortunately no.  the +20 queries are distinct from each other, even tho
they share some of the original query parameters (and some facet information
from the original query facets).

what I was envisioning was something that works like a facet, but instead of
returning information about the first query, it would return information
about queries similar to the first query.
-- 
View this message in context: http://old.nabble.com/having-solr-generate-and-execute-other-related-queries-automatically-tp26327032p26328314.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: having solr generate and execute other related queries automatically

[tpunder <ti...@gmail.com>]

Could you use the facet.query feature
(http://wiki.apache.org/solr/SimpleFacetParameters#facet.query_:_Arbitrary_Query_Faceting)
to reduce it to 2 queries?

So you'd:

1. Send solr the first query
2. Solr executes and returns the query to you
3. You then use the facet results to create a 2nd query with your +20
facet.query queries
3. Solr executes and returns the results of your +20 facet.query queries

Then you'd only have 2 roundtrips.

-Tim


gdeconto wrote:
> 
> Scenario:
> 
> 1. I have a query I want to execute; I would be using the results and
> facets returned
> 2. I also have a couple of dozen other queries that are closely related to
> the first query and to the facets returned by that query.  For each query,
> I would only be using the total number of results documents (aka numFound)
> 3. steps 1 and 2 would occur for each page load/view
> 
> while I could run these queries individually, I am concerned about the
> overhead of having to run +20 solr queries.
> 
> I was wondering if it was a better idea to extend the search or facet
> component somehow so that:
> 
> 1. I send solr the first query
> 2. solr executes the query
> 3. solr then creates (programatically, based on the parameters I pass in
> the first query as well as some of the facet results) and executes +20
> other queries
> 4. solr finally returns me the results and facets of the first query, as
> well as the counts/numFounds obtained from the +20 other queries
> 
> I was thinking that this would save me the time and roundtrip overhead of
> +20 solr queries per page, but I am unsure of how to proceed.
> 
> hopefully this question is not too vague.  any help/example appreciated.
> 

-- 
View this message in context: http://old.nabble.com/having-solr-generate-and-execute-other-related-queries-automatically-tp26327032p26328209.html
Sent from the Solr - User mailing list archive at Nabble.com.