solr warmup and reading the index into memory on startup?

[Nicolas Flacco <nf...@attinteractive.com>]

I'm seeing some problems warming up solr on startup. Currently warmup
consists of two parts- running queries on startup programmatically, and
then running a script to perform queries. The programmatic warmup seems to
warm up Solr fine in terms of making queries via the Solr admin tool, but
when I do a query programmatically, the first query basically takes 2-3m,
during which time I see tons of lucene index loading activity. I'm
assuming that the lucene index is not getting loaded into memory, so this
happens on the first query.

Is there a way to force Solr to load the index into memory on startup
apart from doing a query and waiting?

Programmatic warmup:

for(a bunch of queries){
  SolrQueryResponse rsp = new SolrQueryResponse();
  core.execute(req, rsp);
  NamedList values = rsp.getValues();
  // and iterate through the docs in these values
}

Also tried adding a *:* query into the warmup listener- this didn't help
either.

<listener event="firstSearcher" class="solr.FooWarmupListener">
             <str name="warmupFile">warmup_queries.txt</str>
      <arr name="queries">
      	<lst>
<str name="q">*:*</str></lst>
    </arr>

Re: solr warmup and reading the index into memory on startup?

[Erick Erickson <er...@gmail.com>]

You've got to add firstSearcher/newSearcher queries
that do the expensive parts. Where you're probably
taking a wrong turn is thinking that queries
"load the index into memory". Nothing of the sort
happens. q=*:* is particularly unhelpful since it's
a "constant score query" which basically short-circuits
any index loading.

What you want is to use queries that load the various
caches. Good candidates are queries that sort, facet,
etc. Try listing a query as firstSearcher that sorts and
facets as per the most popular uses of sorting and
faceting in our SolrJ program, that will probably help.

But 2-3 minutes is quite a long time. The number
of *unique* terms in the fields that you facet and
sort on heavily influence the time it takes to
get the caches warmed up, so you might examine
those.

P.S. dumping the time it takes to run each of
your "bunch of queries" will help pinpoint the ones
that would be good candidates for firstSearcher.

Best
Erick

On Fri, Feb 24, 2012 at 2:59 PM, Nicolas Flacco
<nf...@attinteractive.com> wrote:
> I'm seeing some problems warming up solr on startup. Currently warmup
> consists of two parts- running queries on startup programmatically, and
> then running a script to perform queries. The programmatic warmup seems to
> warm up Solr fine in terms of making queries via the Solr admin tool, but
> when I do a query programmatically, the first query basically takes 2-3m,
> during which time I see tons of lucene index loading activity. I'm
> assuming that the lucene index is not getting loaded into memory, so this
> happens on the first query.
>
> Is there a way to force Solr to load the index into memory on startup
> apart from doing a query and waiting?
>
> Programmatic warmup:
>
> for(a bunch of queries){
>  SolrQueryResponse rsp = new SolrQueryResponse();
>  core.execute(req, rsp);
>  NamedList values = rsp.getValues();
>  // and iterate through the docs in these values
> }
>
> Also tried adding a *:* query into the warmup listener- this didn't help
> either.
>
> <listener event="firstSearcher" class="solr.FooWarmupListener">
>             <str name="warmupFile">warmup_queries.txt</str>
>      <arr name="queries">
>        <lst>
> <str name="q">*:*</str></lst>
>    </arr>
>
>