You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Yonik Seeley <yo...@lucidimagination.com> on 2010/10/01 22:45:40 UTC
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
On Thu, Sep 30, 2010 at 4:52 PM, Renee Sun <re...@mcafee.com> wrote:
> - do you have any warming queries configured?
> ----> no, all autowarmingcount are set to 0 for all caches
Any static warming requests though (newSearcher / firstSearcher hooks
in solrconfig.xml)?
Is anything at all querying these cores while you are trying to start
everything up?
> - do the cores have documents already, and if so, how many per core?
> ----> yes, 130 cores total, 2,3 of them already have 1~2.4 million
> documents, others have about 50,000 documents
> - are you using the same schema & solrconfig, or did you upgrade?
> ----> yes, absolutely no change
> - have you tried finding out what is taking up all the memory (or
> all the CPU time)?
> ----> yes, JConsole shows after 70 cores are loaded in about 4 minutes, all
> 16GB memory are taken and rest of cores load extremely slow. The memory
> remain high and never dropped.
The question that we need to figure out is: what is taking the memory
in 1.4 that did not in 1.3?
Perhaps a memory profiler will point to where all of the extra memory
is being allocated?
-Yonik
http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Posted by Peter Karich <pe...@yahoo.de>.
just a blind shot (didn't read the full thread):
what is your maxWarmingSearchers settings? For large indices we set it
to 2 (maximum)
Regards,
Peter.
> just update on this issue...
>
> we turned off the new/first searchers (upgrade to Solr 1.4.1), and ran
> benchmark tests, there is no noticeable performance impact on the queries we
> perform comparing with Solr 1.3 benchmark tests WITH new/first searchers.
>
> Also the memory usage reduced by 5.5 GB after loading the cores with our
> data volume by turning these static warm caches off.
>
> We will take this approach in our production environment but meanwhile I am
> curious if this issue will be addressed: it seems the new/first searchers do
> not really buy any performance benefits because it uses so much memory,
> especially at core loading time.
>
> thanks
> Renee
>
>
>
--
http://jetwick.com twitter search prototype
Re: Upgrade to Solr 1.4, very slow at start up when loading all
cores
Posted by Chris Hostetter <ho...@fucit.org>.
: We will take this approach in our production environment but meanwhile I am
: curious if this issue will be addressed: it seems the new/first searchers do
: not really buy any performance benefits because it uses so much memory,
: especially at core loading time.
There's nothing inheriently wrong with using newSearcher/firstSearcher --
for many people they do in fact provide a perf improvement for "real"
users (at the cost of some initial time spent warming before those users
ever get access to the searcher)
As i udnerstand it from this thread, your issue is not actually the
firstSearcher/newSearcher -- your issue (per yonik's ocmments) is that
with per Segment sorting in 1.4, the FieldCache for some of your fields
requires a lot more ram in 1.4 then it would have been for Solr 1.3 --
which caused GC thrashing during initialization.
Even w/o using firstSearcher/newSearcher, all that RAM is still going to
be used if/when you do sort on those fields -- all removing the
firstSearcher/newSearcher queries on those fields has done for you is
delay when the time spent initializing those FieldCaches happens and when
that RAM first starts getting used.
It's posisbly you never actual sort on those fields, in which case
removing those warming queries completely is definitely the way to go --
but if you do sort on them then the warming queries can still legitimately
be helpful (in thta they pay the cost up front before a real user issues
queries)
As yonik mentioned the real "fix" for the amount of memory being used is
to switch to the TrieDateFields which use much more efficient FieldCache's
for sorting -- with that change you can probably start using the warming
queries again. (Depending on how you tested, you may not have noticed
much advantage to having them because you'll really only see the
advantages on the initial queries that do sorting -- those should show
huge outlier times w/o the warming queries, but once those poor unlucky
users have paid the price for initializing hte FieldCache, every one elses
sorts should be fast)
-Hoss
Re: Upgrade to Solr 1.4, very slow at start up when loading all
cores
Posted by Renee Sun <re...@mcafee.com>.
just update on this issue...
we turned off the new/first searchers (upgrade to Solr 1.4.1), and ran
benchmark tests, there is no noticeable performance impact on the queries we
perform comparing with Solr 1.3 benchmark tests WITH new/first searchers.
Also the memory usage reduced by 5.5 GB after loading the cores with our
data volume by turning these static warm caches off.
We will take this approach in our production environment but meanwhile I am
curious if this issue will be addressed: it seems the new/first searchers do
not really buy any performance benefits because it uses so much memory,
especially at core loading time.
thanks
Renee
--
View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1697609.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all
cores
Posted by Renee Sun <re...@mcafee.com>.
Hi Yonik,
I tried the fix suggested in your comments (using "solr.TrieDateField" ),
and it loaded up 130 cores in 1 minute, 1.3GB memory (a little more than 1GB
when turning off static warm cache, and much less than 6.5GB when use
'solr.DateField').
Will this have any impact on first query or performance?
I am about to run some benchmark test and compare with old data, will update
you.
Renee
--
View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1637176.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Posted by Lance Norskog <go...@gmail.com>.
Could Solr just load cores one at a time, waiting for loader events to
finish? Or continuously stage 2 or three simultaneously?
On Sat, Oct 2, 2010 at 7:08 AM, Yonik Seeley <yo...@lucidimagination.com> wrote:
> On Fri, Oct 1, 2010 at 5:42 PM, Renee Sun <re...@mcafee.com> wrote:
>> Hi Yonik,
>>
>> I attached the solrconfig.xml to you in previous post, and we do have
>> firstSearch and newSearch hook ups.
>>
>> I commented them out, all 130 cores loaded up in 1 minute, same as in solr
>> 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took
>> about 6.5GB for same amount of data.
>
> For other's reference: here is the warming query (it's the same for
> newSearcher too):
>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst>
> <str name="q">type:message</str>
> <str name="start">0</str>
> <str name="rows">10</str>
> <str name="sort">message_date desc</str>
> </lst>
> </arr>
> </listener>
>
> The sort field message_date is what will be taking up the memory.
>
> Starting with Lucene 2.9 (which is used in Solr 1.4), searching and
> sorting is per-segment.
> This is generally beneficial, but in this case I believe it is causing
> the extra memory usage because the same date value that would have
> been shared across all documents in the fieldcache is now repeated in
> each segment it is used in.
>
> One potential fix (that requires you to reindex) is to use the "date"
> fieldType as defined in the new 1.4 schema:
> <fieldType name="date" class="solr.TrieDateField" omitNorms="true"
> precisionStep="0" positionIncrementGap="0"/>
>
> This will use 8 bytes per document in your index, rather than 4 bytes
> per doc + an array of unique string-date values per index.
>
> Trunk (4.0-dev) is also much more efficient at storing string-based
> fields in the FieldCache - but that will only help you if you're
> comfortable with using development versions.
>
> -Yonik
> http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8
>
--
Lance Norskog
goksron@gmail.com
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Oct 1, 2010 at 5:42 PM, Renee Sun <re...@mcafee.com> wrote:
> Hi Yonik,
>
> I attached the solrconfig.xml to you in previous post, and we do have
> firstSearch and newSearch hook ups.
>
> I commented them out, all 130 cores loaded up in 1 minute, same as in solr
> 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took
> about 6.5GB for same amount of data.
For other's reference: here is the warming query (it's the same for
newSearcher too):
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<str name="q">type:message</str>
<str name="start">0</str>
<str name="rows">10</str>
<str name="sort">message_date desc</str>
</lst>
</arr>
</listener>
The sort field message_date is what will be taking up the memory.
Starting with Lucene 2.9 (which is used in Solr 1.4), searching and
sorting is per-segment.
This is generally beneficial, but in this case I believe it is causing
the extra memory usage because the same date value that would have
been shared across all documents in the fieldcache is now repeated in
each segment it is used in.
One potential fix (that requires you to reindex) is to use the "date"
fieldType as defined in the new 1.4 schema:
<fieldType name="date" class="solr.TrieDateField" omitNorms="true"
precisionStep="0" positionIncrementGap="0"/>
This will use 8 bytes per document in your index, rather than 4 bytes
per doc + an array of unique string-date values per index.
Trunk (4.0-dev) is also much more efficient at storing string-based
fields in the FieldCache - but that will only help you if you're
comfortable with using development versions.
-Yonik
http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Posted by Dennis Gearon <ge...@sbcglobal.net>.
VERY interesting. Looking forward to what happens on the first queries.
Dennis Gearon
Signature Warning
----------------
EARTH has a Right To Life,
otherwise we all die.
Read 'Hot, Flat, and Crowded'
Laugh at http://www.yert.com/film.php
--- On Fri, 10/1/10, Renee Sun <re...@mcafee.com> wrote:
> From: Renee Sun <re...@mcafee.com>
> Subject: Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
> To: solr-user@lucene.apache.org
> Date: Friday, October 1, 2010, 2:42 PM
>
> Hi Yonik,
>
> I attached the solrconfig.xml to you in previous post, and
> we do have
> firstSearch and newSearch hook ups.
>
> I commented them out, all 130 cores loaded up in 1 minute,
> same as in solr
> 1.3. total memory took about 1GB. Whereas in 1.3,
> with hook ups, it took
> about 6.5GB for same amount of data.
>
> I assuem the consequence of commenting out the static warm
> requests will be
> it will slow down first time we hit the core for query?
>
> thanks
> Renee
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1617263.html
> Sent from the Solr - User mailing list archive at
> Nabble.com.
>
Re: Upgrade to Solr 1.4, very slow at start up when loading all
cores
Posted by Renee Sun <re...@mcafee.com>.
Hi Yonik,
I attached the solrconfig.xml to you in previous post, and we do have
firstSearch and newSearch hook ups.
I commented them out, all 130 cores loaded up in 1 minute, same as in solr
1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took
about 6.5GB for same amount of data.
I assuem the consequence of commenting out the static warm requests will be
it will slow down first time we hit the core for query?
thanks
Renee
--
View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1617263.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all
cores
Posted by Renee Sun <re...@mcafee.com>.
http://lucene.472066.n3.nabble.com/file/n1617135/solrconfig.xml
solrconfig.xml
Hi Yonik,
I have uploaded our solrconfig.xml file for your reference.
we also tried 1.4.1, for same index data, it took about 30-55 minutes to
load up all 130 cores, it did not help at all.
There is no query running when we tried to upload the cores.
Since JConsole is not responding at all when this happens, I am not sure if
there is any command link memory profiler I can use to collect information,
any suggestions?
thanks
Renee
--
View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1617135.html
Sent from the Solr - User mailing list archive at Nabble.com.