You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Wei Zhao <dw...@gmail.com> on 2013/05/17 03:14:20 UTC
Solr 4 memory usage increase
We are migrating from Solr 3.5 to Solr 4.2.
After some performance testing, we found 4.2's memory usage is a lot higher
than 3.5. Our 12GM max heap process used to handle the test pretty well with
3.5. while, with 4.2, the same test runs into serious GC half way (20
minutes) into the test.
Anyone knows that something is significantly different from Solr 3.5 in
terms of memory usage?
We also notice on a slave, IndexeWriter class is actually taking significant
portion (around 3GB) of the heap. Why Solr opens a IndexWriter on a slave?
Is there a conf I can use to turn it off? I don't remember I saw such heap
usage by a similar class in Solr 3.5.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-memory-usage-increase-tp4064066.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4 memory usage increase
Posted by Wei Zhao <dw...@gmail.com>.
Here is the JVM info:
$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-memory-usage-increase-tp4064066p4064271.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4 memory usage increase
Posted by William Bell <bi...@gmail.com>.
Yeah how to turn off index writer ?
On Friday, May 17, 2013, Andre Bois-Crettez wrote:
> Can you explain your setup more ?
> ie. is it master/slave, indexing in parallel, etc ?
>
> We had to commit more often to reduce JVM memory usage due to
> transaction logs in SolrCloud mode, compared with previous setups
> without tlogs.
> update?commit=true&**openSearcher=false
>
>
> André
>
> On 05/17/2013 09:56 AM, J Mohamed Zahoor wrote:
>
>> I get the same issue in 1.7.0_09-b05 also.
>>
>> ./zahoor
>>
>>
>> On 17-May-2013, at 12:07 PM, Walter Underwood<wu...@wunderwood.org>
>> wrote:
>>
>> It is past time to get off of Java 6. That is dead. End of life. No more
>>> updates, not even for security bugs.
>>>
>>> What version of Java 6? Some earlier versions had bad bugs that Solr
>>> would run into. We hit them in prod until we upgraded.
>>>
>>> wunder
>>>
>>> On May 16, 2013, at 11:28 PM, Wei Zhao wrote:
>>>
>>> No, exactly the same JVM of Java6
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://lucene.472066.n3.**
>>>> nabble.com/Solr-4-memory-**usage-increase-**tp4064066p4064108.html<http://lucene.472066.n3.nabble.com/Solr-4-memory-usage-increase-tp4064066p4064108.html>
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>
>> --
>> André Bois-Crettez
>>
>> Search technology, Kelkoo
>> http://www.kelkoo.com/
>>
>
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 8, rue du Sentier 75002 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à
> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
> destinataire de ce message, merci de le détruire et d'en avertir
> l'expéditeur.
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
Re: Solr 4 memory usage increase
Posted by Chris Hostetter <ho...@fucit.org>.
: We have master/slave setup. We disabled autocommits/autosoftcommits. So the
: slave only replicates from master and serve query. Master does all the
: indexing and commit every 5 minutes. Slave polls master every 2.5 minutes
: and does replication.
Details matter...
Are you using hte exact same configs that you had with 3.5, or did you
copy the configs from 4.0 and then modify them?
The distinction is important, because if you "disabled
autocommits/autosofcommits" from the 4.0 example configs, w/o also
disabling hte updateLog, then you'll probably see the updateLog eat up a
lot of RAM.
When asking a general question, ie: "It uses more RAM" you need to provide
a lot of details about your usage for anyone to even attempt to make a
guess about your problem.
It would probably help folks help you if you provided your actual configs,
specifics about the number of documents, size of index, types of queries,
etc....
Please review in it's entirety...
https://wiki.apache.org/solr/UsingMailingLists
-Hoss
Re: Solr 4 memory usage increase
Posted by Wei Zhao <dw...@gmail.com>.
We have master/slave setup. We disabled autocommits/autosoftcommits. So the
slave only replicates from master and serve query. Master does all the
indexing and commit every 5 minutes. Slave polls master every 2.5 minutes
and does replication.
Both tests with Solr 3.5 and 4.2 was run with the same setup and both with
master/slave replication running.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-memory-usage-increase-tp4064066p4064275.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4 memory usage increase
Posted by Andre Bois-Crettez <an...@kelkoo.com>.
Can you explain your setup more ?
ie. is it master/slave, indexing in parallel, etc ?
We had to commit more often to reduce JVM memory usage due to
transaction logs in SolrCloud mode, compared with previous setups
without tlogs.
update?commit=true&openSearcher=false
André
On 05/17/2013 09:56 AM, J Mohamed Zahoor wrote:
> I get the same issue in 1.7.0_09-b05 also.
>
> ./zahoor
>
>
> On 17-May-2013, at 12:07 PM, Walter Underwood<wu...@wunderwood.org> wrote:
>
>> It is past time to get off of Java 6. That is dead. End of life. No more updates, not even for security bugs.
>>
>> What version of Java 6? Some earlier versions had bad bugs that Solr would run into. We hit them in prod until we upgraded.
>>
>> wunder
>>
>> On May 16, 2013, at 11:28 PM, Wei Zhao wrote:
>>
>>> No, exactly the same JVM of Java6
>>>
>>>
>>>
>>> --
>>> View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-memory-usage-increase-tp4064066p4064108.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> André Bois-Crettez
>
> Search technology, Kelkoo
> http://www.kelkoo.com/
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris
Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.
Re: Solr 4 memory usage increase
Posted by J Mohamed Zahoor <za...@indix.com>.
I get the same issue in 1.7.0_09-b05 also.
./zahoor
On 17-May-2013, at 12:07 PM, Walter Underwood <wu...@wunderwood.org> wrote:
> It is past time to get off of Java 6. That is dead. End of life. No more updates, not even for security bugs.
>
> What version of Java 6? Some earlier versions had bad bugs that Solr would run into. We hit them in prod until we upgraded.
>
> wunder
>
> On May 16, 2013, at 11:28 PM, Wei Zhao wrote:
>
>> No, exactly the same JVM of Java6
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-memory-usage-increase-tp4064066p4064108.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
Re: Solr 4 memory usage increase
Posted by Walter Underwood <wu...@wunderwood.org>.
It is past time to get off of Java 6. That is dead. End of life. No more updates, not even for security bugs.
What version of Java 6? Some earlier versions had bad bugs that Solr would run into. We hit them in prod until we upgraded.
wunder
On May 16, 2013, at 11:28 PM, Wei Zhao wrote:
> No, exactly the same JVM of Java6
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-memory-usage-increase-tp4064066p4064108.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4 memory usage increase
Posted by Wei Zhao <dw...@gmail.com>.
No, exactly the same JVM of Java6
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-memory-usage-increase-tp4064066p4064108.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4 memory usage increase
Posted by William Bell <bi...@gmail.com>.
Did you switch the JVM too?
On Thu, May 16, 2013 at 7:14 PM, Wei Zhao <dw...@gmail.com> wrote:
> We are migrating from Solr 3.5 to Solr 4.2.
>
> After some performance testing, we found 4.2's memory usage is a lot higher
> than 3.5. Our 12GM max heap process used to handle the test pretty well
> with
> 3.5. while, with 4.2, the same test runs into serious GC half way (20
> minutes) into the test.
>
> Anyone knows that something is significantly different from Solr 3.5 in
> terms of memory usage?
>
> We also notice on a slave, IndexeWriter class is actually taking
> significant
> portion (around 3GB) of the heap. Why Solr opens a IndexWriter on a slave?
> Is there a conf I can use to turn it off? I don't remember I saw such heap
> usage by a similar class in Solr 3.5.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-4-memory-usage-increase-tp4064066.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076