RE: SOLR uses too much CPU and GC is also weird on Windows server

[Jaan Arjasepp <Ja...@coop.ee>]

Thanks for all for helping to think about it, but eventually found out that code was basically single record deleting/adding records. After it was batched up, then everything is back to normal. Funny thing is that 6.0.0. handled these requests somehow, but newer version did not.
Anyway, we will observe this and try to improve our code as well.

Best regards,
Jaan

-----Original Message-----
From: Erick Erickson <er...@gmail.com> 
Sent: 28 October 2020 17:18
To: solr-user@lucene.apache.org
Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server

DocValues=true are usually only used for “primitive” types, string, numerics, booleans and the like, specifically _not_ text-based.

I say “usually” because there’s a special “SortableTextField” where it does make some sense to have a text-based field have docValues, but that’s intended for relatively short fields. For example you want to sort on a title field. And probably not something you’re working with.

There’s not much we can say from this distance I’m afraid. I think I’d focus on the memory requirements, maybe take a heap dump and see what’s using memory.

Did you restart Solr _after_ turning off indexing? I ask because that would help determine which side the problem is on, indexing or querying. It does sound like querying though.

As for docValues in general, if you want to be really brave, you can set uninvertible=false for all your fields where docValues=false. When you facet on such a field, you won’t get anything back. If you sort on such a field, you’ll get an error message back. That should test if somehow not having docValues is the root of your problem. Do this on a test system of course ;) I think this is a low-probability issue, but it’s a mystery anyway so...

Updating shouldn’t be that much of a problem either, and if you still see high CPU with indexing turned off, that eliminates indexing as a candidate.

Is there any chance you changed your schema at all and didn’t delete your entire index and add all your documents back? There are a lot of ways things can go wrong if that’s the case. You had to reindex from scratch when you went to 8x from 6x, I’m wondering if during that process the schema changed without starting over. I’m grasping at straws here…

I’d also seriously consider going to 8.6.3. We only make point releases when there’s something serious. Looking through lucene/CHANGES.txt, there is one memory leak fix in 8.6.2. I’d expect a gradual buildup of heap if that were what you’re seeing, but you never know.

As for having docValues=false, that would cut down on the size of the index on disk and speed up indexing some, but in terms of memory usage or CPU usage when querying, unless the docValues structures are _needed_, they’re never read into OS RAM by MMapDirectory… The question really is whether you ever, intentionally or not, do “something” that would be more efficient with docValues. That’s where setting uninvertible=false whenever you set docValues=false makes sense, things will show up if your assumption that you don’t need docValues is false.

Best,
Erick


> On Oct 28, 2020, at 9:29 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
> 
> Hi all,
> 
> Its me again. Anyway, I did a little research and we tried different things and well, some questions I want to ask and some things that I found.
> 
> Well after monitoring my system with VirtualVM, I found that GC jumping is from 0.5GB to 2.5GB and it has 4GB of memory for now, so it should not be an issue anymore or what? But will observe it a bit as it might rise I guess a bit.
> 
> Next thing we found or are thinking about is that writing on a disk might be an issue, we turned off the indexing and some other stuff, but I would say, it did not save much still.
> I also did go through all the schema fields, not that much really. They are all docValues=true. Also I must say they are all automatically generated, so no manual working there except one field, but this also has docValue=true. Just curious, if the field is not a string/text, can it be docValue=false or still better to have true? And as for uninversion, then we are not using much facets nor other specific things in query, just simple queries. 
> 
> Though I must say we are updating documents quite a bunch, but usage of CPU for being so high, not sure about that. Older version seemed not using CPU so much...
> 
> I am a bit running out of ideas and hoping that this will continue to work, but I dont like the CPU usage even over night, when nobody uses it. We will try to figure out the issue here and I hope I can ask more questions when in doubt or out of ideas. Also I must admit, solr is really new for me personally.
> 
> Jaan
> 
> -----Original Message-----
> From: Walter Underwood <wu...@wunderwood.org>
> Sent: 27 October 2020 18:44
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR uses too much CPU and GC is also weird on Windows 
> server
> 
> That first graph shows a JVM that does not have enough heap for the program it is running. Look at the bottom of the dips. That is the amount of memory still in use after a full GC.
> 
> You want those dips to drop to about half of the available heap, so I’d immediately increase that heap to 4G. That might not be enough, so you’ll need to to watch that graph after the increase.
> 
> I’ve been using 8G heaps with Solr since version 1.2. We run this config with Java 8 on over 100 machines. We do not do any faceting, which can take more memory.
> 
> SOLR_HEAP=8g
> # Use G1 GC  -- wunder 2017-01-23
> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
>> On Oct 27, 2020, at 12:48 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>> 
>> Hello,
>> 
>> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
>> We use it on a Windows Server 2019.
>> Java version is 11
>> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
>> Document sizes are not big, I guess. We only use one core.
>> Document stats are here:
>> Num Docs: 3627341
>> Max Doc: 4981019
>> Heap Memory Usage: 434400
>> Deleted Docs: 1353678
>> Version: 15999036
>> Segment Count: 30
>> 
>> The size of index is 2.66GB
>> 
>> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
>> If needed more information about background of the system, I am happy to help.
>> 
>> 
>> But now to the issue I am having.
>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
>> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>> 
>> Heap memory usage via JMX and jconsole -> 
>> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/vie
>> w ?usp=sharing As you can see, it starts of normal, but then goes 
>> crazy and it has been like this over night.
>> 
>> This is overall monitoring graphs, as you can see CPU is working hard 
>> or hardly working. -> 
>> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/vie
>> w ?usp=sharing VM summary can be found here -> 
>> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/vie
>> w ?usp=sharing And finally to have better and quick overview of the 
>> SOLR executing parameters that I have -> 
>> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/vie
>> w
>> ?usp=sharing
>> 
>> If you can point me what I have to do to make it work, then I appreciate it a lot.
>> 
>> Thank you in advance.
>> 
>> Best regards,
>> Jaan
>> 
>> 
> 

Re: SOLR uses too much CPU and GC is also weird on Windows server

[Erick Erickson <er...@gmail.com>]

What this sounds like is that somehow you were committing after every update in 8x but not in your 6x code. How that would have been change is anybody’s guess ;).

It’s vaguely possible that your client is committing and you had IgnoreCommitOptimizeUpdateProcessorFactory defined in your update chain in 6x but not 8x.

The other thing would be if your commit interval was much shorter in 8x than 6x or if your autowarm parameters were significantly different.

That said, this is still a mystery, glad you found an answer.

Thanks for getting back to us on this, this is useful information to have.

Best,
Erick

> On Nov 2, 2020, at 7:50 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
> 
> Thanks for all for helping to think about it, but eventually found out that code was basically single record deleting/adding records. After it was batched up, then everything is back to normal. Funny thing is that 6.0.0. handled these requests somehow, but newer version did not.
> Anyway, we will observe this and try to improve our code as well.
> 
> Best regards,
> Jaan
> 
> -----Original Message-----
> From: Erick Erickson <er...@gmail.com> 
> Sent: 28 October 2020 17:18
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
> 
> DocValues=true are usually only used for “primitive” types, string, numerics, booleans and the like, specifically _not_ text-based.
> 
> I say “usually” because there’s a special “SortableTextField” where it does make some sense to have a text-based field have docValues, but that’s intended for relatively short fields. For example you want to sort on a title field. And probably not something you’re working with.
> 
> There’s not much we can say from this distance I’m afraid. I think I’d focus on the memory requirements, maybe take a heap dump and see what’s using memory.
> 
> Did you restart Solr _after_ turning off indexing? I ask because that would help determine which side the problem is on, indexing or querying. It does sound like querying though.
> 
> As for docValues in general, if you want to be really brave, you can set uninvertible=false for all your fields where docValues=false. When you facet on such a field, you won’t get anything back. If you sort on such a field, you’ll get an error message back. That should test if somehow not having docValues is the root of your problem. Do this on a test system of course ;) I think this is a low-probability issue, but it’s a mystery anyway so...
> 
> Updating shouldn’t be that much of a problem either, and if you still see high CPU with indexing turned off, that eliminates indexing as a candidate.
> 
> Is there any chance you changed your schema at all and didn’t delete your entire index and add all your documents back? There are a lot of ways things can go wrong if that’s the case. You had to reindex from scratch when you went to 8x from 6x, I’m wondering if during that process the schema changed without starting over. I’m grasping at straws here…
> 
> I’d also seriously consider going to 8.6.3. We only make point releases when there’s something serious. Looking through lucene/CHANGES.txt, there is one memory leak fix in 8.6.2. I’d expect a gradual buildup of heap if that were what you’re seeing, but you never know.
> 
> As for having docValues=false, that would cut down on the size of the index on disk and speed up indexing some, but in terms of memory usage or CPU usage when querying, unless the docValues structures are _needed_, they’re never read into OS RAM by MMapDirectory… The question really is whether you ever, intentionally or not, do “something” that would be more efficient with docValues. That’s where setting uninvertible=false whenever you set docValues=false makes sense, things will show up if your assumption that you don’t need docValues is false.
> 
> Best,
> Erick
> 
> 
>> On Oct 28, 2020, at 9:29 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>> 
>> Hi all,
>> 
>> Its me again. Anyway, I did a little research and we tried different things and well, some questions I want to ask and some things that I found.
>> 
>> Well after monitoring my system with VirtualVM, I found that GC jumping is from 0.5GB to 2.5GB and it has 4GB of memory for now, so it should not be an issue anymore or what? But will observe it a bit as it might rise I guess a bit.
>> 
>> Next thing we found or are thinking about is that writing on a disk might be an issue, we turned off the indexing and some other stuff, but I would say, it did not save much still.
>> I also did go through all the schema fields, not that much really. They are all docValues=true. Also I must say they are all automatically generated, so no manual working there except one field, but this also has docValue=true. Just curious, if the field is not a string/text, can it be docValue=false or still better to have true? And as for uninversion, then we are not using much facets nor other specific things in query, just simple queries. 
>> 
>> Though I must say we are updating documents quite a bunch, but usage of CPU for being so high, not sure about that. Older version seemed not using CPU so much...
>> 
>> I am a bit running out of ideas and hoping that this will continue to work, but I dont like the CPU usage even over night, when nobody uses it. We will try to figure out the issue here and I hope I can ask more questions when in doubt or out of ideas. Also I must admit, solr is really new for me personally.
>> 
>> Jaan
>> 
>> -----Original Message-----
>> From: Walter Underwood <wu...@wunderwood.org>
>> Sent: 27 October 2020 18:44
>> To: solr-user@lucene.apache.org
>> Subject: Re: SOLR uses too much CPU and GC is also weird on Windows 
>> server
>> 
>> That first graph shows a JVM that does not have enough heap for the program it is running. Look at the bottom of the dips. That is the amount of memory still in use after a full GC.
>> 
>> You want those dips to drop to about half of the available heap, so I’d immediately increase that heap to 4G. That might not be enough, so you’ll need to to watch that graph after the increase.
>> 
>> I’ve been using 8G heaps with Solr since version 1.2. We run this config with Java 8 on over 100 machines. We do not do any faceting, which can take more memory.
>> 
>> SOLR_HEAP=8g
>> # Use G1 GC  -- wunder 2017-01-23
>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>> GC_TUNE=" \
>> -XX:+UseG1GC \
>> -XX:+ParallelRefProcEnabled \
>> -XX:G1HeapRegionSize=8m \
>> -XX:MaxGCPauseMillis=200 \
>> -XX:+UseLargePages \
>> -XX:+AggressiveOpts \
>> "
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>>> On Oct 27, 2020, at 12:48 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>>> 
>>> Hello,
>>> 
>>> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
>>> We use it on a Windows Server 2019.
>>> Java version is 11
>>> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
>>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
>>> Document sizes are not big, I guess. We only use one core.
>>> Document stats are here:
>>> Num Docs: 3627341
>>> Max Doc: 4981019
>>> Heap Memory Usage: 434400
>>> Deleted Docs: 1353678
>>> Version: 15999036
>>> Segment Count: 30
>>> 
>>> The size of index is 2.66GB
>>> 
>>> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
>>> If needed more information about background of the system, I am happy to help.
>>> 
>>> 
>>> But now to the issue I am having.
>>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
>>> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>>> 
>>> Heap memory usage via JMX and jconsole -> 
>>> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/vie
>>> w ?usp=sharing As you can see, it starts of normal, but then goes 
>>> crazy and it has been like this over night.
>>> 
>>> This is overall monitoring graphs, as you can see CPU is working hard 
>>> or hardly working. -> 
>>> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/vie
>>> w ?usp=sharing VM summary can be found here -> 
>>> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/vie
>>> w ?usp=sharing And finally to have better and quick overview of the 
>>> SOLR executing parameters that I have -> 
>>> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/vie
>>> w
>>> ?usp=sharing
>>> 
>>> If you can point me what I have to do to make it work, then I appreciate it a lot.
>>> 
>>> Thank you in advance.
>>> 
>>> Best regards,
>>> Jaan
>>> 
>>> 
>> 
>