You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Neil Hooey <nh...@gmail.com> on 2012/03/02 21:37:03 UTC
How can Solr do parallel query warming with and ?
I'm trying to get Solr to run warming queries in parallel with
listener events, but it always does them in sequence, pegging one CPU
while calculating facet counts.
Someone at Lucid Imagination suggested using multiple <listenever
event="firstSearcher"> tags, each with a single facet query in them,
but those are still done in parallel.
Is it possible to run warming queries in parallel, and if so, how?
I'm aware that you could run an external script that forks, but I'd
like to use Solr's native support for this if it exists.
Examples that don't work:
<!-- runs in sequence: multiple facet queries in a single <listener> -->
<query>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst><str name="q">*:*</str><str name="facet.field">field1</str></lst>
<lst><str name="q">*:*</str><str name="facet.field">field2</str></lst>
<lst><str name="q">*:*</str><str name="facet.field">field3</str></lst>
<lst><str name="q">*:*</str><str name="facet.field">field4</str></lst>
</arr>
</listener>
</query>
<!-- runs in sequence: queries distributed across separate <listener> tags -->
<query>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst><str name="q">*:*</str><str name="facet.field">field1</str></lst>
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst><str name="q">*:*</str><str name="facet.field">field2</str></lst>
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst><str name="q">*:*</str><str name="facet.field">field3</str></lst>
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst><str name="q">*:*</str><str name="facet.field">field4</str></lst>
</arr>
</listener>
</query>
RE: How can Solr do parallel query warming with and
?
Posted by Michael Ryan <mr...@moreover.com>.
https://issues.apache.org/jira/browse/SOLR-2548 may be of interest to you.
-Michael
Re: How can Solr do parallel query warming with and ?
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Neil,
Still is not clear whether it multi or singe valued fields that
defines usage or FieldCache or UnInvertedField, and per-segment reader vs
top-level reader.
The only concern I have about your approach is the waste of cpu for
calculate facets for huge *:* docsets. I guess you can try to find narrow
way to trigger field cache initialization. Perhaps
http://wiki.apache.org/solr/TermsComponent can be useful for it.
Regards
On Sat, Mar 3, 2012 at 11:58 PM, Neil Hooey <nh...@gmail.com> wrote:
> I need to have those queries trigger the generation of facet counts, which
> can take up to 5 minutes for all of them combined.
>
> If the facet counts aren't warmed, then the first query to ask for facet
> counts on a particular field will take several minutes to return results.
>
> On Sat, Mar 3, 2012 at 5:40 AM, Mikhail Khludnev <
> mkhludnev@griddynamics.com>
> wrote:
> > Neil,
> >
> > Would you mind if I ask what particularly do you want to warm by these
> > queries?
> >
> > Regards
> >
> > On Sat, Mar 3, 2012 at 12:37 AM, Neil Hooey <nh...@gmail.com> wrote:
> >
> >> I'm trying to get Solr to run warming queries in parallel with
> >> listener events, but it always does them in sequence, pegging one CPU
> >> while calculating facet counts.
> >>
> >> Someone at Lucid Imagination suggested using multiple <listenever
> >> event="firstSearcher"> tags, each with a single facet query in them,
> >> but those are still done in parallel.
> >>
> >> Is it possible to run warming queries in parallel, and if so, how?
> >>
> >> I'm aware that you could run an external script that forks, but I'd
> >> like to use Solr's native support for this if it exists.
> >>
> >> Examples that don't work:
> >>
> >> <!-- runs in sequence: multiple facet queries in a single <listener> -->
> >> <query>
> >> <listener event="firstSearcher" class="solr.QuerySenderListener">
> >> <arr name="queries">
> >> <lst><str name="q">*:*</str><str
> name="facet.field">field1</str></lst>
> >> <lst><str name="q">*:*</str><str
> name="facet.field">field2</str></lst>
> >> <lst><str name="q">*:*</str><str
> name="facet.field">field3</str></lst>
> >> <lst><str name="q">*:*</str><str
> name="facet.field">field4</str></lst>
> >> </arr>
> >> </listener>
> >> </query>
> >>
> >> <!-- runs in sequence: queries distributed across separate <listener>
> tags
> >> -->
> >> <query>
> >> <listener event="firstSearcher" class="solr.QuerySenderListener">
> >> <arr name="queries">
> >> <lst><str name="q">*:*</str><str
> name="facet.field">field1</str></lst>
> >> </arr>
> >> </listener>
> >> <listener event="firstSearcher" class="solr.QuerySenderListener">
> >> <arr name="queries">
> >> <lst><str name="q">*:*</str><str
> name="facet.field">field2</str></lst>
> >> </arr>
> >> </listener>
> >> <listener event="firstSearcher" class="solr.QuerySenderListener">
> >> <arr name="queries">
> >> <lst><str name="q">*:*</str><str
> name="facet.field">field3</str></lst>
> >> </arr>
> >> </listener>
> >> <listener event="firstSearcher" class="solr.QuerySenderListener">
> >> <arr name="queries">
> >> <lst><str name="q">*:*</str><str
> name="facet.field">field4</str></lst>
> >> </arr>
> >> </listener>
> >> </query>
> >>
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Lucid Certified
> > Apache Lucene/Solr Developer
> > Grid Dynamics
> >
> > <http://www.griddynamics.com>
> > <mk...@griddynamics.com>
>
--
Sincerely yours
Mikhail Khludnev
Lucid Certified
Apache Lucene/Solr Developer
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: How can Solr do parallel query warming with and ?
Posted by Neil Hooey <nh...@gmail.com>.
I need to have those queries trigger the generation of facet counts, which
can take up to 5 minutes for all of them combined.
If the facet counts aren't warmed, then the first query to ask for facet
counts on a particular field will take several minutes to return results.
On Sat, Mar 3, 2012 at 5:40 AM, Mikhail Khludnev <mk...@griddynamics.com>
wrote:
> Neil,
>
> Would you mind if I ask what particularly do you want to warm by these
> queries?
>
> Regards
>
> On Sat, Mar 3, 2012 at 12:37 AM, Neil Hooey <nh...@gmail.com> wrote:
>
>> I'm trying to get Solr to run warming queries in parallel with
>> listener events, but it always does them in sequence, pegging one CPU
>> while calculating facet counts.
>>
>> Someone at Lucid Imagination suggested using multiple <listenever
>> event="firstSearcher"> tags, each with a single facet query in them,
>> but those are still done in parallel.
>>
>> Is it possible to run warming queries in parallel, and if so, how?
>>
>> I'm aware that you could run an external script that forks, but I'd
>> like to use Solr's native support for this if it exists.
>>
>> Examples that don't work:
>>
>> <!-- runs in sequence: multiple facet queries in a single <listener> -->
>> <query>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <lst><str name="q">*:*</str><str
name="facet.field">field1</str></lst>
>> <lst><str name="q">*:*</str><str
name="facet.field">field2</str></lst>
>> <lst><str name="q">*:*</str><str
name="facet.field">field3</str></lst>
>> <lst><str name="q">*:*</str><str
name="facet.field">field4</str></lst>
>> </arr>
>> </listener>
>> </query>
>>
>> <!-- runs in sequence: queries distributed across separate <listener>
tags
>> -->
>> <query>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <lst><str name="q">*:*</str><str
name="facet.field">field1</str></lst>
>> </arr>
>> </listener>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <lst><str name="q">*:*</str><str
name="facet.field">field2</str></lst>
>> </arr>
>> </listener>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <lst><str name="q">*:*</str><str
name="facet.field">field3</str></lst>
>> </arr>
>> </listener>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <lst><str name="q">*:*</str><str
name="facet.field">field4</str></lst>
>> </arr>
>> </listener>
>> </query>
>>
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Lucid Certified
> Apache Lucene/Solr Developer
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
Re: How can Solr do parallel query warming with and ?
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Neil,
Would you mind if I ask what particularly do you want to warm by these
queries?
Regards
On Sat, Mar 3, 2012 at 12:37 AM, Neil Hooey <nh...@gmail.com> wrote:
> I'm trying to get Solr to run warming queries in parallel with
> listener events, but it always does them in sequence, pegging one CPU
> while calculating facet counts.
>
> Someone at Lucid Imagination suggested using multiple <listenever
> event="firstSearcher"> tags, each with a single facet query in them,
> but those are still done in parallel.
>
> Is it possible to run warming queries in parallel, and if so, how?
>
> I'm aware that you could run an external script that forks, but I'd
> like to use Solr's native support for this if it exists.
>
> Examples that don't work:
>
> <!-- runs in sequence: multiple facet queries in a single <listener> -->
> <query>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">*:*</str><str name="facet.field">field1</str></lst>
> <lst><str name="q">*:*</str><str name="facet.field">field2</str></lst>
> <lst><str name="q">*:*</str><str name="facet.field">field3</str></lst>
> <lst><str name="q">*:*</str><str name="facet.field">field4</str></lst>
> </arr>
> </listener>
> </query>
>
> <!-- runs in sequence: queries distributed across separate <listener> tags
> -->
> <query>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">*:*</str><str name="facet.field">field1</str></lst>
> </arr>
> </listener>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">*:*</str><str name="facet.field">field2</str></lst>
> </arr>
> </listener>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">*:*</str><str name="facet.field">field3</str></lst>
> </arr>
> </listener>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">*:*</str><str name="facet.field">field4</str></lst>
> </arr>
> </listener>
> </query>
>
--
Sincerely yours
Mikhail Khludnev
Lucid Certified
Apache Lucene/Solr Developer
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: How can Solr do parallel query warming with and ?
Posted by Lance Norskog <go...@gmail.com>.
The code does everything in single-threaded mode, but is coded to use
a multi-threaded Java ExecutorService. So, I've filed a request:
https://issues.apache.org/jira/browse/SOLR-3197
On Fri, Mar 2, 2012 at 12:40 PM, Neil Hooey <nh...@gmail.com> wrote:
>> Someone at Lucid Imagination suggested using multiple <listenever
>> event="firstSearcher"> tags, each with a single facet query in them,
>> but those are still done in parallel.
>
> I meant to say: "but those are still done in sequence".
>
>
> On Fri, Mar 2, 2012 at 3:37 PM, Neil Hooey <nh...@gmail.com> wrote:
>> I'm trying to get Solr to run warming queries in parallel with
>> listener events, but it always does them in sequence, pegging one CPU
>> while calculating facet counts.
>>
>> Someone at Lucid Imagination suggested using multiple <listenever
>> event="firstSearcher"> tags, each with a single facet query in them,
>> but those are still done in parallel.
>>
>> Is it possible to run warming queries in parallel, and if so, how?
>>
>> I'm aware that you could run an external script that forks, but I'd
>> like to use Solr's native support for this if it exists.
>>
>> Examples that don't work:
>>
>> <!-- runs in sequence: multiple facet queries in a single <listener> -->
>> <query>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <lst><str name="q">*:*</str><str name="facet.field">field1</str></lst>
>> <lst><str name="q">*:*</str><str name="facet.field">field2</str></lst>
>> <lst><str name="q">*:*</str><str name="facet.field">field3</str></lst>
>> <lst><str name="q">*:*</str><str name="facet.field">field4</str></lst>
>> </arr>
>> </listener>
>> </query>
>>
>> <!-- runs in sequence: queries distributed across separate <listener> tags -->
>> <query>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <lst><str name="q">*:*</str><str name="facet.field">field1</str></lst>
>> </arr>
>> </listener>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <lst><str name="q">*:*</str><str name="facet.field">field2</str></lst>
>> </arr>
>> </listener>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <lst><str name="q">*:*</str><str name="facet.field">field3</str></lst>
>> </arr>
>> </listener>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <lst><str name="q">*:*</str><str name="facet.field">field4</str></lst>
>> </arr>
>> </listener>
>> </query>
--
Lance Norskog
goksron@gmail.com
Re: How can Solr do parallel query warming with and ?
Posted by Neil Hooey <nh...@gmail.com>.
> Someone at Lucid Imagination suggested using multiple <listenever
> event="firstSearcher"> tags, each with a single facet query in them,
> but those are still done in parallel.
I meant to say: "but those are still done in sequence".
On Fri, Mar 2, 2012 at 3:37 PM, Neil Hooey <nh...@gmail.com> wrote:
> I'm trying to get Solr to run warming queries in parallel with
> listener events, but it always does them in sequence, pegging one CPU
> while calculating facet counts.
>
> Someone at Lucid Imagination suggested using multiple <listenever
> event="firstSearcher"> tags, each with a single facet query in them,
> but those are still done in parallel.
>
> Is it possible to run warming queries in parallel, and if so, how?
>
> I'm aware that you could run an external script that forks, but I'd
> like to use Solr's native support for this if it exists.
>
> Examples that don't work:
>
> <!-- runs in sequence: multiple facet queries in a single <listener> -->
> <query>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">*:*</str><str name="facet.field">field1</str></lst>
> <lst><str name="q">*:*</str><str name="facet.field">field2</str></lst>
> <lst><str name="q">*:*</str><str name="facet.field">field3</str></lst>
> <lst><str name="q">*:*</str><str name="facet.field">field4</str></lst>
> </arr>
> </listener>
> </query>
>
> <!-- runs in sequence: queries distributed across separate <listener> tags -->
> <query>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">*:*</str><str name="facet.field">field1</str></lst>
> </arr>
> </listener>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">*:*</str><str name="facet.field">field2</str></lst>
> </arr>
> </listener>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">*:*</str><str name="facet.field">field3</str></lst>
> </arr>
> </listener>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">*:*</str><str name="facet.field">field4</str></lst>
> </arr>
> </listener>
> </query>