You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by yaswanth kumar <ya...@gmail.com> on 2020/09/01 11:50:33 UTC
Understanding Solr heap %
Can someone make me understand on how the value % on the column Heap is calculated.
I did created a new solr cloud with 3 solr nodes and one zookeeper, its not yet live neither interms of indexing or searching, but I do see some spikes in the HEAP column against nodes when I refresh the page multiple times. Its like almost going to 95% (sometimes) and then coming down to 50%
Solr version: 8.2
Zookeeper: 3.4
JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually RAM size on the node is 16GB)
Basically need to understand if I need to worry about this heap % which was quite altering before making it live? or is that quite normal, because this is new UI change on solr cloud is kind of new to us as we used to have solr 5 version before and this UI component doesn't exists then.
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com
Sent from my iPhone
Re: Understanding Solr heap %
Posted by Joe Doupnik <jr...@netlab1.net>.
That's good. I think I need to mention one other point about this
matter. It is feeding files into Tika (in my case) is paced to avoid
overloads. That is done in my crawler by having a small adjustable pause
(~100ms) after each file submission, and then longer ones (1-3 sec)
after every 100 and 1000 submissions. Also the crawler is set to run at
a lower priority than Solr, thus giving preference to Solr.
In the end we ought to run experiments to find and verify working
values.
Thanks,
Joe D.
On 02/09/2020 03:40, yaswanth kumar wrote:
> I got some understanding now about my actual question.. thanks all for your valuable theories
>
> Sent from my iPhone
>
>> On Sep 1, 2020, at 2:01 PM, Joe Doupnik <jr...@netlab1.net> wrote:
>>
>> As I have not received the follow-on message to mine I will cut&paste it below.
>> My comments on that are the numbers are the numbers. More importantly, I have run large imports ~0.5M docs and I have watched as that progresses. My crawler paces material into Solr. Memory usage (Linux "top") shows cyclic small rises and falls, peaking at about 2GB as the crawler introduces 1 and 3 second pauses every hundred and thousand submissions.. The test shown in my original message is sufficient to show the nature of Solr versions and the choice of garbage collector, and other folks can do similar experiments on their gear. The quoted tests are indeed representative of large and small amounts of various kinds of documents, and I say that based on much experience observing the details.
>> Quibble about GC names if you wish, but please do see those experimental results. Also note the difference in our SOLR_HEAP values: 2GB in my work, 8GB in yours. I have found 2GB to work well for importing small and very large collections (of many file varieties).
>> Thanks,
>> Joe D.
>>> This is misleading and not particularly good advice.
>>>
>>> Solr 8 does NOT contain G1. G1GC is a feature of the JVM. We’ve been using
>>> it with Java 8 and Solr 6.6.2 for a few years.
>>>
>>> A test with eighty documents doesn’t test anything. Try a million documents to
>>> get Solr memory usage warmed up.
>>>
>>> GC_TUNE has been in the solr.in.sh file for a long time. Here are the settings
>>> we use with Java 8. We have about 120 hosts running Solr in six prod clusters.
>>>
>>> SOLR_HEAP=8g
>>> # Use G1 GC -- wunder 2017-01-23
>>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>>> GC_TUNE=" \
>>> -XX:+UseG1GC \
>>> -XX:+ParallelRefProcEnabled \
>>> -XX:G1HeapRegionSize=8m \
>>> -XX:MaxGCPauseMillis=200 \
>>> -XX:+UseLargePages \
>>> -XX:+AggressiveOpts \
>>> "
>>>
>>> wunder
>>> Walter Underwood
>>> wunder@wunderwood.org
>>> http://observer.wunderwood.org/ (my blog)
>>> On 01/09/2020 16:39, Joe Doupnik wrote:
>>> Erick states this correctly. To give some numbers from my experiences, here are two slides from my presentation about installing Solr (https://netlab1.net/, locate item "Solr/Lucene Search Service"):
>>>> <hbifonfjanlomngl.png>
>>>> <phnahkoblmojphjo.png>
>>> Thus we see a) experiments are the key, just as Erick says, and b) the choice of garbage collection algorithm plays a major role.
>>> In my setup I assigned SOLR_HEAP to be 2048m, SOLR_OPTS has -Xss1024k, plus stock GC_TUNE values. Your "memorage" may vary.
>>> Thanks,
>>> Joe D.
>>>
>>>> On 01/09/2020 15:33, Erick Erickson wrote:
>>>> You want to run with the smallest heap you can due to Lucene’s use of MMapDirectory,
>>>> see the excellent:
>>>>
>>>> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>>>
>>>> There’s also little reason to have different Xms and Xmx values, that just means you’ll
>>>> eventually move a bunch of memory around as the heap expands, I usually set them both
>>>> to the same value.
>>>>
>>>> How to determine what “the smallest heap you can” is? Unfortunately there’s no good way
>>>> outside of stress-testing your application with less and less memory until you have problems,
>>>> then add some extra…
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>>>> On Sep 1, 2020, at 10:27 AM, Dominique Bejean <do...@eolya.fr> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> As all Java applications the Heap memory is regularly cleaned by the
>>>>>> garbage collector (some young items moved to the old generation heap zone
>>>>>> and unused old items removed from the old generation heap zone). This
>>>>>> causes heap usage to continuously grow and reduce.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Dominique
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Le mar. 1 sept. 2020 à 13:50, yaswanth kumar <ya...@gmail.com> a
>>>>>> écrit :
>>>>>>
>>>>>> Can someone make me understand on how the value % on the column Heap is
>>>>>> calculated.
>>>>>>
>>>>>> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
>>>>>> not yet live neither interms of indexing or searching, but I do see some
>>>>>> spikes in the HEAP column against nodes when I refresh the page multiple
>>>>>> times. Its like almost going to 95% (sometimes) and then coming down to 50%
>>>>>>
>>>>>> Solr version: 8.2
>>>>>> Zookeeper: 3.4
>>>>>>
>>>>>> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
>>>>>> RAM size on the node is 16GB)
>>>>>>
>>>>>> Basically need to understand if I need to worry about this heap % which
>>>>>> was quite altering before making it live? or is that quite normal, because
>>>>>> this is new UI change on solr cloud is kind of new to us as we used to have
>>>>>> solr 5 version before and this UI component doesn't exists then.
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Yaswanth Kumar Konathala.
>>>>>> yaswanthcse@gmail.com
>>>>>>
>>>>>> Sent from my iPhone
Re: Understanding Solr heap %
Posted by yaswanth kumar <ya...@gmail.com>.
I got some understanding now about my actual question.. thanks all for your valuable theories
Sent from my iPhone
> On Sep 1, 2020, at 2:01 PM, Joe Doupnik <jr...@netlab1.net> wrote:
>
> As I have not received the follow-on message to mine I will cut&paste it below.
> My comments on that are the numbers are the numbers. More importantly, I have run large imports ~0.5M docs and I have watched as that progresses. My crawler paces material into Solr. Memory usage (Linux "top") shows cyclic small rises and falls, peaking at about 2GB as the crawler introduces 1 and 3 second pauses every hundred and thousand submissions.. The test shown in my original message is sufficient to show the nature of Solr versions and the choice of garbage collector, and other folks can do similar experiments on their gear. The quoted tests are indeed representative of large and small amounts of various kinds of documents, and I say that based on much experience observing the details.
> Quibble about GC names if you wish, but please do see those experimental results. Also note the difference in our SOLR_HEAP values: 2GB in my work, 8GB in yours. I have found 2GB to work well for importing small and very large collections (of many file varieties).
> Thanks,
> Joe D.
>> This is misleading and not particularly good advice.
>>
>> Solr 8 does NOT contain G1. G1GC is a feature of the JVM. We’ve been using
>> it with Java 8 and Solr 6.6.2 for a few years.
>>
>> A test with eighty documents doesn’t test anything. Try a million documents to
>> get Solr memory usage warmed up.
>>
>> GC_TUNE has been in the solr.in.sh file for a long time. Here are the settings
>> we use with Java 8. We have about 120 hosts running Solr in six prod clusters.
>>
>> SOLR_HEAP=8g
>> # Use G1 GC -- wunder 2017-01-23
>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>> GC_TUNE=" \
>> -XX:+UseG1GC \
>> -XX:+ParallelRefProcEnabled \
>> -XX:G1HeapRegionSize=8m \
>> -XX:MaxGCPauseMillis=200 \
>> -XX:+UseLargePages \
>> -XX:+AggressiveOpts \
>> "
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>
>> On 01/09/2020 16:39, Joe Doupnik wrote:
>> Erick states this correctly. To give some numbers from my experiences, here are two slides from my presentation about installing Solr (https://netlab1.net/, locate item "Solr/Lucene Search Service"):
>>> <hbifonfjanlomngl.png>
>>
>>> <phnahkoblmojphjo.png>
>>
>> Thus we see a) experiments are the key, just as Erick says, and b) the choice of garbage collection algorithm plays a major role.
>> In my setup I assigned SOLR_HEAP to be 2048m, SOLR_OPTS has -Xss1024k, plus stock GC_TUNE values. Your "memorage" may vary.
>> Thanks,
>> Joe D.
>>
>>> On 01/09/2020 15:33, Erick Erickson wrote:
>>> You want to run with the smallest heap you can due to Lucene’s use of MMapDirectory,
>>> see the excellent:
>>>
>>> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>>
>>> There’s also little reason to have different Xms and Xmx values, that just means you’ll
>>> eventually move a bunch of memory around as the heap expands, I usually set them both
>>> to the same value.
>>>
>>> How to determine what “the smallest heap you can” is? Unfortunately there’s no good way
>>> outside of stress-testing your application with less and less memory until you have problems,
>>> then add some extra…
>>>
>>> Best,
>>> Erick
>>>
>>>>> On Sep 1, 2020, at 10:27 AM, Dominique Bejean <do...@eolya.fr> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> As all Java applications the Heap memory is regularly cleaned by the
>>>>> garbage collector (some young items moved to the old generation heap zone
>>>>> and unused old items removed from the old generation heap zone). This
>>>>> causes heap usage to continuously grow and reduce.
>>>>>
>>>>> Regards
>>>>>
>>>>> Dominique
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Le mar. 1 sept. 2020 à 13:50, yaswanth kumar <ya...@gmail.com> a
>>>>> écrit :
>>>>>
>>>>> Can someone make me understand on how the value % on the column Heap is
>>>>> calculated.
>>>>>
>>>>> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
>>>>> not yet live neither interms of indexing or searching, but I do see some
>>>>> spikes in the HEAP column against nodes when I refresh the page multiple
>>>>> times. Its like almost going to 95% (sometimes) and then coming down to 50%
>>>>>
>>>>> Solr version: 8.2
>>>>> Zookeeper: 3.4
>>>>>
>>>>> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
>>>>> RAM size on the node is 16GB)
>>>>>
>>>>> Basically need to understand if I need to worry about this heap % which
>>>>> was quite altering before making it live? or is that quite normal, because
>>>>> this is new UI change on solr cloud is kind of new to us as we used to have
>>>>> solr 5 version before and this UI component doesn't exists then.
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Yaswanth Kumar Konathala.
>>>>> yaswanthcse@gmail.com
>>>>>
>>>>> Sent from my iPhone
>>
>
Re: Understanding Solr heap %
Posted by Joe Doupnik <jr...@netlab1.net>.
As I have not received the follow-on message to mine I will
cut&paste it below.
My comments on that are the numbers are the numbers. More
importantly, I have run large imports ~0.5M docs and I have watched as
that progresses. My crawler paces material into Solr. Memory usage
(Linux "top") shows cyclic small rises and falls, peaking at about 2GB
as the crawler introduces 1 and 3 second pauses every hundred and
thousand submissions.. The test shown in my original message is
sufficient to show the nature of Solr versions and the choice of garbage
collector, and other folks can do similar experiments on their gear. The
quoted tests are indeed representative of large and small amounts of
various kinds of documents, and I say that based on much experience
observing the details.
Quibble about GC names if you wish, but please do see those
experimental results. Also note the difference in our SOLR_HEAP values:
2GB in my work, 8GB in yours. I have found 2GB to work well for
importing small and very large collections (of many file varieties).
Thanks,
Joe D.
> This is misleading and not particularly good advice.
>
> Solr 8 does NOT contain G1. G1GC is a feature of the JVM. We’ve been using
> it with Java 8 and Solr 6.6.2 for a few years.
>
> A test with eighty documents doesn’t test anything. Try a million documents to
> get Solr memory usage warmed up.
>
> GC_TUNE has been in the solr.in.sh file for a long time. Here are the settings
> we use with Java 8. We have about 120 hosts running Solr in six prod clusters.
>
> SOLR_HEAP=8g
> # Use G1 GC -- wunder 2017-01-23
> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
On 01/09/2020 16:39, Joe Doupnik wrote:
> Erick states this correctly. To give some numbers from my
> experiences, here are two slides from my presentation about installing
> Solr (https://netlab1.net/, locate item "Solr/Lucene Search Service"):
>
>
> Thus we see a) experiments are the key, just as Erick says, and b)
> the choice of garbage collection algorithm plays a major role.
> In my setup I assigned SOLR_HEAP to be 2048m, SOLR_OPTS has
> -Xss1024k, plus stock GC_TUNE values. Your "memorage" may vary.
> Thanks,
> Joe D.
>
> On 01/09/2020 15:33, Erick Erickson wrote:
>> You want to run with the smallest heap you can due to Lucene’s use of MMapDirectory,
>> see the excellent:
>>
>> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>
>> There’s also little reason to have different Xms and Xmx values, that just means you’ll
>> eventually move a bunch of memory around as the heap expands, I usually set them both
>> to the same value.
>>
>> How to determine what “the smallest heap you can” is? Unfortunately there’s no good way
>> outside of stress-testing your application with less and less memory until you have problems,
>> then add some extra…
>>
>> Best,
>> Erick
>>
>>> On Sep 1, 2020, at 10:27 AM, Dominique Bejean<do...@eolya.fr> wrote:
>>>
>>> Hi,
>>>
>>> As all Java applications the Heap memory is regularly cleaned by the
>>> garbage collector (some young items moved to the old generation heap zone
>>> and unused old items removed from the old generation heap zone). This
>>> causes heap usage to continuously grow and reduce.
>>>
>>> Regards
>>>
>>> Dominique
>>>
>>>
>>>
>>>
>>> Le mar. 1 sept. 2020 à 13:50, yaswanth kumar<ya...@gmail.com> a
>>> écrit :
>>>
>>>> Can someone make me understand on how the value % on the column Heap is
>>>> calculated.
>>>>
>>>> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
>>>> not yet live neither interms of indexing or searching, but I do see some
>>>> spikes in the HEAP column against nodes when I refresh the page multiple
>>>> times. Its like almost going to 95% (sometimes) and then coming down to 50%
>>>>
>>>> Solr version: 8.2
>>>> Zookeeper: 3.4
>>>>
>>>> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
>>>> RAM size on the node is 16GB)
>>>>
>>>> Basically need to understand if I need to worry about this heap % which
>>>> was quite altering before making it live? or is that quite normal, because
>>>> this is new UI change on solr cloud is kind of new to us as we used to have
>>>> solr 5 version before and this UI component doesn't exists then.
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Yaswanth Kumar Konathala.
>>>> yaswanthcse@gmail.com
>>>>
>>>> Sent from my iPhone
>
Re: Understanding Solr heap %
Posted by Walter Underwood <wu...@wunderwood.org>.
This is misleading and not particularly good advice.
Solr 8 does NOT contain G1. G1GC is a feature of the JVM. We’ve been using
it with Java 8 and Solr 6.6.2 for a few years.
A test with eighty documents doesn’t test anything. Try a million documents to
get Solr memory usage warmed up.
GC_TUNE has been in the solr.in.sh file for a long time. Here are the settings
we use with Java 8. We have about 120 hosts running Solr in six prod clusters.
SOLR_HEAP=8g
# Use G1 GC -- wunder 2017-01-23
# Settings from https://wiki.apache.org/solr/ShawnHeisey
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=200 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \
"
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Sep 1, 2020, at 8:39 AM, Joe Doupnik <jr...@netlab1.net> wrote:
>
> Erick states this correctly. To give some numbers from my experiences, here are two slides from my presentation about installing Solr (https://netlab1.net/ <https://netlab1.net/>, locate item "Solr/Lucene Search Service"):
>> <hbifonfjanlomngl.png>
>
>> <phnahkoblmojphjo.png>
>
> Thus we see a) experiments are the key, just as Erick says, and b) the choice of garbage collection algorithm plays a major role.
> In my setup I assigned SOLR_HEAP to be 2048m, SOLR_OPTS has -Xss1024k, plus stock GC_TUNE values. Your "memorage" may vary.
> Thanks,
> Joe D.
>
> On 01/09/2020 15:33, Erick Erickson wrote:
>> You want to run with the smallest heap you can due to Lucene’s use of MMapDirectory,
>> see the excellent:
>>
>> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html <https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html>
>>
>> There’s also little reason to have different Xms and Xmx values, that just means you’ll
>> eventually move a bunch of memory around as the heap expands, I usually set them both
>> to the same value.
>>
>> How to determine what “the smallest heap you can” is? Unfortunately there’s no good way
>> outside of stress-testing your application with less and less memory until you have problems,
>> then add some extra…
>>
>> Best,
>> Erick
>>
>>> On Sep 1, 2020, at 10:27 AM, Dominique Bejean <do...@eolya.fr> <ma...@eolya.fr> wrote:
>>>
>>> Hi,
>>>
>>> As all Java applications the Heap memory is regularly cleaned by the
>>> garbage collector (some young items moved to the old generation heap zone
>>> and unused old items removed from the old generation heap zone). This
>>> causes heap usage to continuously grow and reduce.
>>>
>>> Regards
>>>
>>> Dominique
>>>
>>>
>>>
>>>
>>> Le mar. 1 sept. 2020 à 13:50, yaswanth kumar <ya...@gmail.com> <ma...@gmail.com> a
>>> écrit :
>>>
>>>> Can someone make me understand on how the value % on the column Heap is
>>>> calculated.
>>>>
>>>> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
>>>> not yet live neither interms of indexing or searching, but I do see some
>>>> spikes in the HEAP column against nodes when I refresh the page multiple
>>>> times. Its like almost going to 95% (sometimes) and then coming down to 50%
>>>>
>>>> Solr version: 8.2
>>>> Zookeeper: 3.4
>>>>
>>>> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
>>>> RAM size on the node is 16GB)
>>>>
>>>> Basically need to understand if I need to worry about this heap % which
>>>> was quite altering before making it live? or is that quite normal, because
>>>> this is new UI change on solr cloud is kind of new to us as we used to have
>>>> solr 5 version before and this UI component doesn't exists then.
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Yaswanth Kumar Konathala.
>>>> yaswanthcse@gmail.com <ma...@gmail.com>
>>>>
>>>> Sent from my iPhone
>
Re: Understanding Solr heap %
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
You should _not_ set "-XX:G1HeapRegionSize=n" , because:
"... The goal is to have around 2048 regions based on the minimum Java heap size...."
The value of G1HeapRegionSize is automatically calculated upon start up of the JVM.
The parameter "-XX:MaxGCPauseMillis=200" is the default.
Where is the sense of explicitly setting a default parameter to its default value?
Regards
Bernd
Am 01.09.20 um 18:00 schrieb Walter Underwood:
> This is misleading and not particularly good advice.
>
> Solr 8 does NOT contain G1. G1GC is a feature of the JVM. We’ve been using
> it with Java 8 and Solr 6.6.2 for a few years.
>
> A test with eighty documents doesn’t test anything. Try a million documents to
> get Solr memory usage warmed up.
>
> GC_TUNE has been in the solr.in.sh file for a long time. Here are the settings
> we use with Java 8. We have about 120 hosts running Solr in six prod clusters.
>
> SOLR_HEAP=8g
> # Use G1 GC -- wunder 2017-01-23
> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>> On Sep 1, 2020, at 8:39 AM, Joe Doupnik <jr...@netlab1.net> wrote:
>>
>> Erick states this correctly. To give some numbers from my experiences, here are two slides from my presentation about installing Solr (https://netlab1.net/ <https://netlab1.net/>, locate item "Solr/Lucene Search Service"):
>>> <hbifonfjanlomngl.png>
>>
>>> <phnahkoblmojphjo.png>
>>
>> Thus we see a) experiments are the key, just as Erick says, and b) the choice of garbage collection algorithm plays a major role.
>> In my setup I assigned SOLR_HEAP to be 2048m, SOLR_OPTS has -Xss1024k, plus stock GC_TUNE values. Your "memorage" may vary.
>> Thanks,
>> Joe D.
>>
>> On 01/09/2020 15:33, Erick Erickson wrote:
>>> You want to run with the smallest heap you can due to Lucene’s use of MMapDirectory,
>>> see the excellent:
>>>
>>> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html <https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html>
>>>
>>> There’s also little reason to have different Xms and Xmx values, that just means you’ll
>>> eventually move a bunch of memory around as the heap expands, I usually set them both
>>> to the same value.
>>>
>>> How to determine what “the smallest heap you can” is? Unfortunately there’s no good way
>>> outside of stress-testing your application with less and less memory until you have problems,
>>> then add some extra…
>>>
>>> Best,
>>> Erick
>>>
>>>> On Sep 1, 2020, at 10:27 AM, Dominique Bejean <do...@eolya.fr> <ma...@eolya.fr> wrote:
>>>>
>>>> Hi,
>>>>
>>>> As all Java applications the Heap memory is regularly cleaned by the
>>>> garbage collector (some young items moved to the old generation heap zone
>>>> and unused old items removed from the old generation heap zone). This
>>>> causes heap usage to continuously grow and reduce.
>>>>
>>>> Regards
>>>>
>>>> Dominique
>>>>
>>>>
>>>>
>>>>
>>>> Le mar. 1 sept. 2020 à 13:50, yaswanth kumar <ya...@gmail.com> <ma...@gmail.com> a
>>>> écrit :
>>>>
>>>>> Can someone make me understand on how the value % on the column Heap is
>>>>> calculated.
>>>>>
>>>>> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
>>>>> not yet live neither interms of indexing or searching, but I do see some
>>>>> spikes in the HEAP column against nodes when I refresh the page multiple
>>>>> times. Its like almost going to 95% (sometimes) and then coming down to 50%
>>>>>
>>>>> Solr version: 8.2
>>>>> Zookeeper: 3.4
>>>>>
>>>>> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
>>>>> RAM size on the node is 16GB)
>>>>>
>>>>> Basically need to understand if I need to worry about this heap % which
>>>>> was quite altering before making it live? or is that quite normal, because
>>>>> this is new UI change on solr cloud is kind of new to us as we used to have
>>>>> solr 5 version before and this UI component doesn't exists then.
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Yaswanth Kumar Konathala.
>>>>> yaswanthcse@gmail.com <ma...@gmail.com>
>>>>>
>>>>> Sent from my iPhone
>>
>
>
Re: Understanding Solr heap %
Posted by Joe Doupnik <jr...@netlab1.net>.
Erick states this correctly. To give some numbers from my
experiences, here are two slides from my presentation about installing
Solr (https://netlab1.net/, locate item "Solr/Lucene Search Service"):
Thus we see a) experiments are the key, just as Erick says, and b)
the choice of garbage collection algorithm plays a major role.
In my setup I assigned SOLR_HEAP to be 2048m, SOLR_OPTS has
-Xss1024k, plus stock GC_TUNE values. Your "memorage" may vary.
Thanks,
Joe D.
On 01/09/2020 15:33, Erick Erickson wrote:
> You want to run with the smallest heap you can due to Lucene’s use of MMapDirectory,
> see the excellent:
>
> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> There’s also little reason to have different Xms and Xmx values, that just means you’ll
> eventually move a bunch of memory around as the heap expands, I usually set them both
> to the same value.
>
> How to determine what “the smallest heap you can” is? Unfortunately there’s no good way
> outside of stress-testing your application with less and less memory until you have problems,
> then add some extra…
>
> Best,
> Erick
>
>> On Sep 1, 2020, at 10:27 AM, Dominique Bejean <do...@eolya.fr> wrote:
>>
>> Hi,
>>
>> As all Java applications the Heap memory is regularly cleaned by the
>> garbage collector (some young items moved to the old generation heap zone
>> and unused old items removed from the old generation heap zone). This
>> causes heap usage to continuously grow and reduce.
>>
>> Regards
>>
>> Dominique
>>
>>
>>
>>
>> Le mar. 1 sept. 2020 à 13:50, yaswanth kumar <ya...@gmail.com> a
>> écrit :
>>
>>> Can someone make me understand on how the value % on the column Heap is
>>> calculated.
>>>
>>> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
>>> not yet live neither interms of indexing or searching, but I do see some
>>> spikes in the HEAP column against nodes when I refresh the page multiple
>>> times. Its like almost going to 95% (sometimes) and then coming down to 50%
>>>
>>> Solr version: 8.2
>>> Zookeeper: 3.4
>>>
>>> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
>>> RAM size on the node is 16GB)
>>>
>>> Basically need to understand if I need to worry about this heap % which
>>> was quite altering before making it live? or is that quite normal, because
>>> this is new UI change on solr cloud is kind of new to us as we used to have
>>> solr 5 version before and this UI component doesn't exists then.
>>>
>>> --
>>> Thanks & Regards,
>>> Yaswanth Kumar Konathala.
>>> yaswanthcse@gmail.com
>>>
>>> Sent from my iPhone
Re: Understanding Solr heap %
Posted by Erick Erickson <er...@gmail.com>.
You want to run with the smallest heap you can due to Lucene’s use of MMapDirectory,
see the excellent:
https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
There’s also little reason to have different Xms and Xmx values, that just means you’ll
eventually move a bunch of memory around as the heap expands, I usually set them both
to the same value.
How to determine what “the smallest heap you can” is? Unfortunately there’s no good way
outside of stress-testing your application with less and less memory until you have problems,
then add some extra…
Best,
Erick
> On Sep 1, 2020, at 10:27 AM, Dominique Bejean <do...@eolya.fr> wrote:
>
> Hi,
>
> As all Java applications the Heap memory is regularly cleaned by the
> garbage collector (some young items moved to the old generation heap zone
> and unused old items removed from the old generation heap zone). This
> causes heap usage to continuously grow and reduce.
>
> Regards
>
> Dominique
>
>
>
>
> Le mar. 1 sept. 2020 à 13:50, yaswanth kumar <ya...@gmail.com> a
> écrit :
>
>> Can someone make me understand on how the value % on the column Heap is
>> calculated.
>>
>> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
>> not yet live neither interms of indexing or searching, but I do see some
>> spikes in the HEAP column against nodes when I refresh the page multiple
>> times. Its like almost going to 95% (sometimes) and then coming down to 50%
>>
>> Solr version: 8.2
>> Zookeeper: 3.4
>>
>> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
>> RAM size on the node is 16GB)
>>
>> Basically need to understand if I need to worry about this heap % which
>> was quite altering before making it live? or is that quite normal, because
>> this is new UI change on solr cloud is kind of new to us as we used to have
>> solr 5 version before and this UI component doesn't exists then.
>>
>> --
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> yaswanthcse@gmail.com
>>
>> Sent from my iPhone
Re: Understanding Solr heap %
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
As all Java applications the Heap memory is regularly cleaned by the
garbage collector (some young items moved to the old generation heap zone
and unused old items removed from the old generation heap zone). This
causes heap usage to continuously grow and reduce.
Regards
Dominique
Le mar. 1 sept. 2020 à 13:50, yaswanth kumar <ya...@gmail.com> a
écrit :
> Can someone make me understand on how the value % on the column Heap is
> calculated.
>
> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
> not yet live neither interms of indexing or searching, but I do see some
> spikes in the HEAP column against nodes when I refresh the page multiple
> times. Its like almost going to 95% (sometimes) and then coming down to 50%
>
> Solr version: 8.2
> Zookeeper: 3.4
>
> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
> RAM size on the node is 16GB)
>
> Basically need to understand if I need to worry about this heap % which
> was quite altering before making it live? or is that quite normal, because
> this is new UI change on solr cloud is kind of new to us as we used to have
> solr 5 version before and this UI component doesn't exists then.
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
>
> Sent from my iPhone