You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by geeky2 <ge...@hotmail.com> on 2013/05/20 16:16:40 UTC
seeing lots of "autowarming" messages in log during DIH indexing
hello,
we are tracking down some performance issues with our DIH process.
not sure if this is related - but i am seeing tons of the messages below in
the logs during re-indexing of the core.
what do these messages mean?
2013-05-18 19:37:30,623 INFO [org.apache.solr.update.UpdateHandler]
(pool-11-thread-1) end_commit_flush
2013-05-18 19:37:30,623 INFO [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,624 INFO [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming result for Searcher@5b8d745 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,624 INFO [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,625 INFO [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming result for Searcher@5b8d745 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,625 INFO [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=1,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,628 INFO [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming result for Searcher@5b8d745 main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=3,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,628 INFO [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,628 INFO [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming result for Searcher@5b8d745 main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
thx
mark
--
View this message in context: http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: seeing lots of "autowarming" messages in log during DIH
indexing
Posted by geeky2 <ge...@hotmail.com>.
the DIH is launched via a script - called by a "cron like" scheduler.
clean, commit and optimize are all true.
thx
mark
#!/bin/bash
SERVER=$1
PORT=$2
CLEAN=$3
COMMIT=$4
OPTIMIZE=$5
COREPATH=$6
echo SERVER: $SERVER
echo PORT: $PORT
echo CLEAN: $CLEAN
echo COMMIT: $COMMIT
echo OPTIMIZE: $OPTIMIZE
echo COREPATH: $COREPATH
if [ $# != 6 ]; then
echo "USAGE: $0 [SERVER] [PORT] [CLEAN: true/false] [COMMIT:
true/false] [OPTIMIZE: true/false] [COREPATH] "
exit 1;
fi
...
--
View this message in context: http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649p4067477.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: seeing lots of "autowarming" messages in log during DIH indexing
Posted by Tim Vaillancourt <ti...@elementspace.com>.
Interesting.
In your scenario would you use commit=true, or commit=false, and do you use auto soft/hard commits?
Secondly, if you did use auto-soft/hard commits, how would they affect this scenario? I'm guessing even with commit=false, the autoCommits would be triggered either by time or max docs, which opens a searcher anyways. A total guess though.
I'm interested in doing full-imports without committing/opening new searchers until it is complete.
Cheers!
Tim
On 20/05/13 03:59 PM, shreejay wrote:
> geeky2 wrote
>> you mean i would add this switch to my script that kicks of the
>> dataimport?
>>
>> exmaple:
>>
>>
>> OUTPUT=$(curl -v
>> http://${SERVER}.intra.searshc.com:${PORT}/solrpartscat/${CORE}/dataimport
>> -F command=full-import -F clean=${CLEAN} -F commit=${COMMIT} -F
>> optimize=${OPTIMIZE} -F openSearcher=false)
> Yes. Thats correct
>
>
>
> geeky2 wrote
>> what needs to be done _AFTER_ the DIH finishes (if anything)?
>>
>> eg, does this need to be turned back on after the DIH has finished?
> Yes. You need to open the searcher to be able to search. Just run another
> commit with openSearcher = true , once your indexing process finishes.
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649p4064768.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: seeing lots of "autowarming" messages in log during DIH
indexing
Posted by shreejay <sh...@gmail.com>.
geeky2 wrote
> you mean i would add this switch to my script that kicks of the
> dataimport?
>
> exmaple:
>
>
> OUTPUT=$(curl -v
> http://${SERVER}.intra.searshc.com:${PORT}/solrpartscat/${CORE}/dataimport
> -F command=full-import -F clean=${CLEAN} -F commit=${COMMIT} -F
> optimize=${OPTIMIZE} -F openSearcher=false)
Yes. Thats correct
geeky2 wrote
> what needs to be done _AFTER_ the DIH finishes (if anything)?
>
> eg, does this need to be turned back on after the DIH has finished?
Yes. You need to open the searcher to be able to search. Just run another
commit with openSearcher = true , once your indexing process finishes.
--
View this message in context: http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649p4064768.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: seeing lots of "autowarming" messages in log during DIH
indexing
Posted by geeky2 <ge...@hotmail.com>.
you mean i would add this switch to my script that kicks of the dataimport?
exmaple:
OUTPUT=$(curl -v
http://${SERVER}.intra.searshc.com:${PORT}/solrpartscat/${CORE}/dataimport
-F command=full-import -F clean=${CLEAN} -F commit=${COMMIT} -F
optimize=${OPTIMIZE} -F openSearcher=false)
what needs to be done _AFTER_ the DIH finishes (if anything)?
eg, does this need to be turned back on after the DIH has finished?
--
View this message in context: http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649p4064695.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: seeing lots of "autowarming" messages in log during DIH
indexing
Posted by Shreejay <sh...@gmail.com>.
Every time a commit is done a new searcher is opened. In the solr config file caches are defined with a parameter called autowarm. Autowarm basically tries to copy the cache values from previous searcher into the current one. If you are doing a bulk update and do not care for searching till your indexing is over then you can specify openSearcher=false while doing a commit. That should speed up your indexing a lot.
--
Shreejay Nair
Sent from my mobile device. Please excuse brevity and typos.
On Monday, May 20, 2013 at 7:16, geeky2 wrote:
> hello,
>
> we are tracking down some performance issues with our DIH process.
>
> not sure if this is related - but i am seeing tons of the messages below in
> the logs during re-indexing of the core.
>
> what do these messages mean?
>
>
> 2013-05-18 19:37:30,623 INFO [org.apache.solr.update.UpdateHandler]
> (pool-11-thread-1) end_commit_flush
> 2013-05-18 19:37:30,623 INFO [org.apache.solr.search.SolrIndexSearcher]
> (pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
> main
> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> 2013-05-18 19:37:30,624 INFO [org.apache.solr.search.SolrIndexSearcher]
> (pool-10-thread-1) autowarming result for Searcher@5b8d745 main
> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> 2013-05-18 19:37:30,624 INFO [org.apache.solr.search.SolrIndexSearcher]
> (pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
> main
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> 2013-05-18 19:37:30,625 INFO [org.apache.solr.search.SolrIndexSearcher]
> (pool-10-thread-1) autowarming result for Searcher@5b8d745 main
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> 2013-05-18 19:37:30,625 INFO [org.apache.solr.search.SolrIndexSearcher]
> (pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
> main
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=1,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> 2013-05-18 19:37:30,628 INFO [org.apache.solr.search.SolrIndexSearcher]
> (pool-10-thread-1) autowarming result for Searcher@5b8d745 main
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=3,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> 2013-05-18 19:37:30,628 INFO [org.apache.solr.search.SolrIndexSearcher]
> (pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
> main
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> 2013-05-18 19:37:30,628 INFO [org.apache.solr.search.SolrIndexSearcher]
> (pool-10-thread-1) autowarming result for Searcher@5b8d745 main
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>
> thx
> mark
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>