You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jaan Arjasepp <Ja...@coop.ee> on 2020/10/27 07:48:06 UTC
SOLR uses too much CPU and GC is also weird on Windows server
Hello,
We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
We use it on a Windows Server 2019.
Java version is 11
Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
Document sizes are not big, I guess. We only use one core.
Document stats are here:
Num Docs: 3627341
Max Doc: 4981019
Heap Memory Usage: 434400
Deleted Docs: 1353678
Version: 15999036
Segment Count: 30
The size of index is 2.66GB
While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
If needed more information about background of the system, I am happy to help.
But now to the issue I am having.
If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
Heap memory usage via JMX and jconsole -> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view?usp=sharing
As you can see, it starts of normal, but then goes crazy and it has been like this over night.
This is overall monitoring graphs, as you can see CPU is working hard or hardly working. -> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view?usp=sharing
VM summary can be found here -> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view?usp=sharing
And finally to have better and quick overview of the SOLR executing parameters that I have -> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view?usp=sharing
If you can point me what I have to do to make it work, then I appreciate it a lot.
Thank you in advance.
Best regards,
Jaan
Re: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Erick Erickson <er...@gmail.com>.
What this sounds like is that somehow you were committing after every update in 8x but not in your 6x code. How that would have been change is anybody’s guess ;).
It’s vaguely possible that your client is committing and you had IgnoreCommitOptimizeUpdateProcessorFactory defined in your update chain in 6x but not 8x.
The other thing would be if your commit interval was much shorter in 8x than 6x or if your autowarm parameters were significantly different.
That said, this is still a mystery, glad you found an answer.
Thanks for getting back to us on this, this is useful information to have.
Best,
Erick
> On Nov 2, 2020, at 7:50 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>
> Thanks for all for helping to think about it, but eventually found out that code was basically single record deleting/adding records. After it was batched up, then everything is back to normal. Funny thing is that 6.0.0. handled these requests somehow, but newer version did not.
> Anyway, we will observe this and try to improve our code as well.
>
> Best regards,
> Jaan
>
> -----Original Message-----
> From: Erick Erickson <er...@gmail.com>
> Sent: 28 October 2020 17:18
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
>
> DocValues=true are usually only used for “primitive” types, string, numerics, booleans and the like, specifically _not_ text-based.
>
> I say “usually” because there’s a special “SortableTextField” where it does make some sense to have a text-based field have docValues, but that’s intended for relatively short fields. For example you want to sort on a title field. And probably not something you’re working with.
>
> There’s not much we can say from this distance I’m afraid. I think I’d focus on the memory requirements, maybe take a heap dump and see what’s using memory.
>
> Did you restart Solr _after_ turning off indexing? I ask because that would help determine which side the problem is on, indexing or querying. It does sound like querying though.
>
> As for docValues in general, if you want to be really brave, you can set uninvertible=false for all your fields where docValues=false. When you facet on such a field, you won’t get anything back. If you sort on such a field, you’ll get an error message back. That should test if somehow not having docValues is the root of your problem. Do this on a test system of course ;) I think this is a low-probability issue, but it’s a mystery anyway so...
>
> Updating shouldn’t be that much of a problem either, and if you still see high CPU with indexing turned off, that eliminates indexing as a candidate.
>
> Is there any chance you changed your schema at all and didn’t delete your entire index and add all your documents back? There are a lot of ways things can go wrong if that’s the case. You had to reindex from scratch when you went to 8x from 6x, I’m wondering if during that process the schema changed without starting over. I’m grasping at straws here…
>
> I’d also seriously consider going to 8.6.3. We only make point releases when there’s something serious. Looking through lucene/CHANGES.txt, there is one memory leak fix in 8.6.2. I’d expect a gradual buildup of heap if that were what you’re seeing, but you never know.
>
> As for having docValues=false, that would cut down on the size of the index on disk and speed up indexing some, but in terms of memory usage or CPU usage when querying, unless the docValues structures are _needed_, they’re never read into OS RAM by MMapDirectory… The question really is whether you ever, intentionally or not, do “something” that would be more efficient with docValues. That’s where setting uninvertible=false whenever you set docValues=false makes sense, things will show up if your assumption that you don’t need docValues is false.
>
> Best,
> Erick
>
>
>> On Oct 28, 2020, at 9:29 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>>
>> Hi all,
>>
>> Its me again. Anyway, I did a little research and we tried different things and well, some questions I want to ask and some things that I found.
>>
>> Well after monitoring my system with VirtualVM, I found that GC jumping is from 0.5GB to 2.5GB and it has 4GB of memory for now, so it should not be an issue anymore or what? But will observe it a bit as it might rise I guess a bit.
>>
>> Next thing we found or are thinking about is that writing on a disk might be an issue, we turned off the indexing and some other stuff, but I would say, it did not save much still.
>> I also did go through all the schema fields, not that much really. They are all docValues=true. Also I must say they are all automatically generated, so no manual working there except one field, but this also has docValue=true. Just curious, if the field is not a string/text, can it be docValue=false or still better to have true? And as for uninversion, then we are not using much facets nor other specific things in query, just simple queries.
>>
>> Though I must say we are updating documents quite a bunch, but usage of CPU for being so high, not sure about that. Older version seemed not using CPU so much...
>>
>> I am a bit running out of ideas and hoping that this will continue to work, but I dont like the CPU usage even over night, when nobody uses it. We will try to figure out the issue here and I hope I can ask more questions when in doubt or out of ideas. Also I must admit, solr is really new for me personally.
>>
>> Jaan
>>
>> -----Original Message-----
>> From: Walter Underwood <wu...@wunderwood.org>
>> Sent: 27 October 2020 18:44
>> To: solr-user@lucene.apache.org
>> Subject: Re: SOLR uses too much CPU and GC is also weird on Windows
>> server
>>
>> That first graph shows a JVM that does not have enough heap for the program it is running. Look at the bottom of the dips. That is the amount of memory still in use after a full GC.
>>
>> You want those dips to drop to about half of the available heap, so I’d immediately increase that heap to 4G. That might not be enough, so you’ll need to to watch that graph after the increase.
>>
>> I’ve been using 8G heaps with Solr since version 1.2. We run this config with Java 8 on over 100 machines. We do not do any faceting, which can take more memory.
>>
>> SOLR_HEAP=8g
>> # Use G1 GC -- wunder 2017-01-23
>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>> GC_TUNE=" \
>> -XX:+UseG1GC \
>> -XX:+ParallelRefProcEnabled \
>> -XX:G1HeapRegionSize=8m \
>> -XX:MaxGCPauseMillis=200 \
>> -XX:+UseLargePages \
>> -XX:+AggressiveOpts \
>> "
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>> On Oct 27, 2020, at 12:48 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>>>
>>> Hello,
>>>
>>> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
>>> We use it on a Windows Server 2019.
>>> Java version is 11
>>> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
>>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
>>> Document sizes are not big, I guess. We only use one core.
>>> Document stats are here:
>>> Num Docs: 3627341
>>> Max Doc: 4981019
>>> Heap Memory Usage: 434400
>>> Deleted Docs: 1353678
>>> Version: 15999036
>>> Segment Count: 30
>>>
>>> The size of index is 2.66GB
>>>
>>> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
>>> If needed more information about background of the system, I am happy to help.
>>>
>>>
>>> But now to the issue I am having.
>>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
>>> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>>>
>>> Heap memory usage via JMX and jconsole ->
>>> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/vie
>>> w ?usp=sharing As you can see, it starts of normal, but then goes
>>> crazy and it has been like this over night.
>>>
>>> This is overall monitoring graphs, as you can see CPU is working hard
>>> or hardly working. ->
>>> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/vie
>>> w ?usp=sharing VM summary can be found here ->
>>> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/vie
>>> w ?usp=sharing And finally to have better and quick overview of the
>>> SOLR executing parameters that I have ->
>>> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/vie
>>> w
>>> ?usp=sharing
>>>
>>> If you can point me what I have to do to make it work, then I appreciate it a lot.
>>>
>>> Thank you in advance.
>>>
>>> Best regards,
>>> Jaan
>>>
>>>
>>
>
RE: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Jaan Arjasepp <Ja...@coop.ee>.
Thanks for all for helping to think about it, but eventually found out that code was basically single record deleting/adding records. After it was batched up, then everything is back to normal. Funny thing is that 6.0.0. handled these requests somehow, but newer version did not.
Anyway, we will observe this and try to improve our code as well.
Best regards,
Jaan
-----Original Message-----
From: Erick Erickson <er...@gmail.com>
Sent: 28 October 2020 17:18
To: solr-user@lucene.apache.org
Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
DocValues=true are usually only used for “primitive” types, string, numerics, booleans and the like, specifically _not_ text-based.
I say “usually” because there’s a special “SortableTextField” where it does make some sense to have a text-based field have docValues, but that’s intended for relatively short fields. For example you want to sort on a title field. And probably not something you’re working with.
There’s not much we can say from this distance I’m afraid. I think I’d focus on the memory requirements, maybe take a heap dump and see what’s using memory.
Did you restart Solr _after_ turning off indexing? I ask because that would help determine which side the problem is on, indexing or querying. It does sound like querying though.
As for docValues in general, if you want to be really brave, you can set uninvertible=false for all your fields where docValues=false. When you facet on such a field, you won’t get anything back. If you sort on such a field, you’ll get an error message back. That should test if somehow not having docValues is the root of your problem. Do this on a test system of course ;) I think this is a low-probability issue, but it’s a mystery anyway so...
Updating shouldn’t be that much of a problem either, and if you still see high CPU with indexing turned off, that eliminates indexing as a candidate.
Is there any chance you changed your schema at all and didn’t delete your entire index and add all your documents back? There are a lot of ways things can go wrong if that’s the case. You had to reindex from scratch when you went to 8x from 6x, I’m wondering if during that process the schema changed without starting over. I’m grasping at straws here…
I’d also seriously consider going to 8.6.3. We only make point releases when there’s something serious. Looking through lucene/CHANGES.txt, there is one memory leak fix in 8.6.2. I’d expect a gradual buildup of heap if that were what you’re seeing, but you never know.
As for having docValues=false, that would cut down on the size of the index on disk and speed up indexing some, but in terms of memory usage or CPU usage when querying, unless the docValues structures are _needed_, they’re never read into OS RAM by MMapDirectory… The question really is whether you ever, intentionally or not, do “something” that would be more efficient with docValues. That’s where setting uninvertible=false whenever you set docValues=false makes sense, things will show up if your assumption that you don’t need docValues is false.
Best,
Erick
> On Oct 28, 2020, at 9:29 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>
> Hi all,
>
> Its me again. Anyway, I did a little research and we tried different things and well, some questions I want to ask and some things that I found.
>
> Well after monitoring my system with VirtualVM, I found that GC jumping is from 0.5GB to 2.5GB and it has 4GB of memory for now, so it should not be an issue anymore or what? But will observe it a bit as it might rise I guess a bit.
>
> Next thing we found or are thinking about is that writing on a disk might be an issue, we turned off the indexing and some other stuff, but I would say, it did not save much still.
> I also did go through all the schema fields, not that much really. They are all docValues=true. Also I must say they are all automatically generated, so no manual working there except one field, but this also has docValue=true. Just curious, if the field is not a string/text, can it be docValue=false or still better to have true? And as for uninversion, then we are not using much facets nor other specific things in query, just simple queries.
>
> Though I must say we are updating documents quite a bunch, but usage of CPU for being so high, not sure about that. Older version seemed not using CPU so much...
>
> I am a bit running out of ideas and hoping that this will continue to work, but I dont like the CPU usage even over night, when nobody uses it. We will try to figure out the issue here and I hope I can ask more questions when in doubt or out of ideas. Also I must admit, solr is really new for me personally.
>
> Jaan
>
> -----Original Message-----
> From: Walter Underwood <wu...@wunderwood.org>
> Sent: 27 October 2020 18:44
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR uses too much CPU and GC is also weird on Windows
> server
>
> That first graph shows a JVM that does not have enough heap for the program it is running. Look at the bottom of the dips. That is the amount of memory still in use after a full GC.
>
> You want those dips to drop to about half of the available heap, so I’d immediately increase that heap to 4G. That might not be enough, so you’ll need to to watch that graph after the increase.
>
> I’ve been using 8G heaps with Solr since version 1.2. We run this config with Java 8 on over 100 machines. We do not do any faceting, which can take more memory.
>
> SOLR_HEAP=8g
> # Use G1 GC -- wunder 2017-01-23
> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>> On Oct 27, 2020, at 12:48 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>>
>> Hello,
>>
>> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
>> We use it on a Windows Server 2019.
>> Java version is 11
>> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
>> Document sizes are not big, I guess. We only use one core.
>> Document stats are here:
>> Num Docs: 3627341
>> Max Doc: 4981019
>> Heap Memory Usage: 434400
>> Deleted Docs: 1353678
>> Version: 15999036
>> Segment Count: 30
>>
>> The size of index is 2.66GB
>>
>> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
>> If needed more information about background of the system, I am happy to help.
>>
>>
>> But now to the issue I am having.
>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
>> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>>
>> Heap memory usage via JMX and jconsole ->
>> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/vie
>> w ?usp=sharing As you can see, it starts of normal, but then goes
>> crazy and it has been like this over night.
>>
>> This is overall monitoring graphs, as you can see CPU is working hard
>> or hardly working. ->
>> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/vie
>> w ?usp=sharing VM summary can be found here ->
>> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/vie
>> w ?usp=sharing And finally to have better and quick overview of the
>> SOLR executing parameters that I have ->
>> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/vie
>> w
>> ?usp=sharing
>>
>> If you can point me what I have to do to make it work, then I appreciate it a lot.
>>
>> Thank you in advance.
>>
>> Best regards,
>> Jaan
>>
>>
>
Re: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Erick Erickson <er...@gmail.com>.
DocValues=true are usually only used for “primitive” types, string, numerics, booleans and the like, specifically _not_ text-based.
I say “usually” because there’s a special “SortableTextField” where it does make some sense to have a text-based field have docValues, but that’s intended for relatively short fields. For example you want to sort on a title field. And probably not something you’re working with.
There’s not much we can say from this distance I’m afraid. I think I’d focus on the memory requirements, maybe take a heap dump and see what’s using memory.
Did you restart Solr _after_ turning off indexing? I ask because that would help determine which side the problem is on, indexing or querying. It does sound like querying though.
As for docValues in general, if you want to be really brave, you can set uninvertible=false for all your fields where docValues=false. When you facet on such a field, you won’t get anything back. If you sort on such a field, you’ll get an error message back. That should test if somehow not having docValues is the root of your problem. Do this on a test system of course ;) I think this is a low-probability issue, but it’s a mystery anyway so...
Updating shouldn’t be that much of a problem either, and if you still see high CPU with indexing turned off, that eliminates indexing as a candidate.
Is there any chance you changed your schema at all and didn’t delete your entire index and add all your documents back? There are a lot of ways things can go wrong if that’s the case. You had to reindex from scratch when you went to 8x from 6x, I’m wondering if during that process the schema changed without starting over. I’m grasping at straws here…
I’d also seriously consider going to 8.6.3. We only make point releases when there’s something serious. Looking through lucene/CHANGES.txt, there is one memory leak fix in 8.6.2. I’d expect a gradual buildup of heap if that were what you’re seeing, but you never know.
As for having docValues=false, that would cut down on the size of the index on disk and speed up indexing some, but in terms of memory usage or CPU usage when querying, unless the docValues structures are _needed_, they’re never read into OS RAM by MMapDirectory… The question really is whether you ever, intentionally or not, do “something” that would be more efficient with docValues. That’s where setting uninvertible=false whenever you set docValues=false makes sense, things will show up if your assumption that you don’t need docValues is false.
Best,
Erick
> On Oct 28, 2020, at 9:29 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>
> Hi all,
>
> Its me again. Anyway, I did a little research and we tried different things and well, some questions I want to ask and some things that I found.
>
> Well after monitoring my system with VirtualVM, I found that GC jumping is from 0.5GB to 2.5GB and it has 4GB of memory for now, so it should not be an issue anymore or what? But will observe it a bit as it might rise I guess a bit.
>
> Next thing we found or are thinking about is that writing on a disk might be an issue, we turned off the indexing and some other stuff, but I would say, it did not save much still.
> I also did go through all the schema fields, not that much really. They are all docValues=true. Also I must say they are all automatically generated, so no manual working there except one field, but this also has docValue=true. Just curious, if the field is not a string/text, can it be docValue=false or still better to have true? And as for uninversion, then we are not using much facets nor other specific things in query, just simple queries.
>
> Though I must say we are updating documents quite a bunch, but usage of CPU for being so high, not sure about that. Older version seemed not using CPU so much...
>
> I am a bit running out of ideas and hoping that this will continue to work, but I dont like the CPU usage even over night, when nobody uses it. We will try to figure out the issue here and I hope I can ask more questions when in doubt or out of ideas. Also I must admit, solr is really new for me personally.
>
> Jaan
>
> -----Original Message-----
> From: Walter Underwood <wu...@wunderwood.org>
> Sent: 27 October 2020 18:44
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
>
> That first graph shows a JVM that does not have enough heap for the program it is running. Look at the bottom of the dips. That is the amount of memory still in use after a full GC.
>
> You want those dips to drop to about half of the available heap, so I’d immediately increase that heap to 4G. That might not be enough, so you’ll need to to watch that graph after the increase.
>
> I’ve been using 8G heaps with Solr since version 1.2. We run this config with Java 8 on over 100 machines. We do not do any faceting, which can take more memory.
>
> SOLR_HEAP=8g
> # Use G1 GC -- wunder 2017-01-23
> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>> On Oct 27, 2020, at 12:48 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>>
>> Hello,
>>
>> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
>> We use it on a Windows Server 2019.
>> Java version is 11
>> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
>> Document sizes are not big, I guess. We only use one core.
>> Document stats are here:
>> Num Docs: 3627341
>> Max Doc: 4981019
>> Heap Memory Usage: 434400
>> Deleted Docs: 1353678
>> Version: 15999036
>> Segment Count: 30
>>
>> The size of index is 2.66GB
>>
>> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
>> If needed more information about background of the system, I am happy to help.
>>
>>
>> But now to the issue I am having.
>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
>> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>>
>> Heap memory usage via JMX and jconsole ->
>> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view
>> ?usp=sharing As you can see, it starts of normal, but then goes crazy
>> and it has been like this over night.
>>
>> This is overall monitoring graphs, as you can see CPU is working hard
>> or hardly working. ->
>> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view
>> ?usp=sharing VM summary can be found here ->
>> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view
>> ?usp=sharing And finally to have better and quick overview of the SOLR
>> executing parameters that I have ->
>> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view
>> ?usp=sharing
>>
>> If you can point me what I have to do to make it work, then I appreciate it a lot.
>>
>> Thank you in advance.
>>
>> Best regards,
>> Jaan
>>
>>
>
Re: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Walter Underwood <wu...@wunderwood.org>.
Double the heap.
All that CPU is the GC trying to free up space.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Oct 28, 2020, at 6:29 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>
> Hi all,
>
> Its me again. Anyway, I did a little research and we tried different things and well, some questions I want to ask and some things that I found.
>
> Well after monitoring my system with VirtualVM, I found that GC jumping is from 0.5GB to 2.5GB and it has 4GB of memory for now, so it should not be an issue anymore or what? But will observe it a bit as it might rise I guess a bit.
>
> Next thing we found or are thinking about is that writing on a disk might be an issue, we turned off the indexing and some other stuff, but I would say, it did not save much still.
> I also did go through all the schema fields, not that much really. They are all docValues=true. Also I must say they are all automatically generated, so no manual working there except one field, but this also has docValue=true. Just curious, if the field is not a string/text, can it be docValue=false or still better to have true? And as for uninversion, then we are not using much facets nor other specific things in query, just simple queries.
>
> Though I must say we are updating documents quite a bunch, but usage of CPU for being so high, not sure about that. Older version seemed not using CPU so much...
>
> I am a bit running out of ideas and hoping that this will continue to work, but I dont like the CPU usage even over night, when nobody uses it. We will try to figure out the issue here and I hope I can ask more questions when in doubt or out of ideas. Also I must admit, solr is really new for me personally.
>
> Jaan
>
> -----Original Message-----
> From: Walter Underwood <wu...@wunderwood.org>
> Sent: 27 October 2020 18:44
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
>
> That first graph shows a JVM that does not have enough heap for the program it is running. Look at the bottom of the dips. That is the amount of memory still in use after a full GC.
>
> You want those dips to drop to about half of the available heap, so I’d immediately increase that heap to 4G. That might not be enough, so you’ll need to to watch that graph after the increase.
>
> I’ve been using 8G heaps with Solr since version 1.2. We run this config with Java 8 on over 100 machines. We do not do any faceting, which can take more memory.
>
> SOLR_HEAP=8g
> # Use G1 GC -- wunder 2017-01-23
> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>> On Oct 27, 2020, at 12:48 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>>
>> Hello,
>>
>> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
>> We use it on a Windows Server 2019.
>> Java version is 11
>> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
>> Document sizes are not big, I guess. We only use one core.
>> Document stats are here:
>> Num Docs: 3627341
>> Max Doc: 4981019
>> Heap Memory Usage: 434400
>> Deleted Docs: 1353678
>> Version: 15999036
>> Segment Count: 30
>>
>> The size of index is 2.66GB
>>
>> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
>> If needed more information about background of the system, I am happy to help.
>>
>>
>> But now to the issue I am having.
>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
>> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>>
>> Heap memory usage via JMX and jconsole ->
>> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view
>> ?usp=sharing As you can see, it starts of normal, but then goes crazy
>> and it has been like this over night.
>>
>> This is overall monitoring graphs, as you can see CPU is working hard
>> or hardly working. ->
>> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view
>> ?usp=sharing VM summary can be found here ->
>> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view
>> ?usp=sharing And finally to have better and quick overview of the SOLR
>> executing parameters that I have ->
>> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view
>> ?usp=sharing
>>
>> If you can point me what I have to do to make it work, then I appreciate it a lot.
>>
>> Thank you in advance.
>>
>> Best regards,
>> Jaan
>>
>>
>
RE: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Jaan Arjasepp <Ja...@coop.ee>.
Hi all,
Its me again. Anyway, I did a little research and we tried different things and well, some questions I want to ask and some things that I found.
Well after monitoring my system with VirtualVM, I found that GC jumping is from 0.5GB to 2.5GB and it has 4GB of memory for now, so it should not be an issue anymore or what? But will observe it a bit as it might rise I guess a bit.
Next thing we found or are thinking about is that writing on a disk might be an issue, we turned off the indexing and some other stuff, but I would say, it did not save much still.
I also did go through all the schema fields, not that much really. They are all docValues=true. Also I must say they are all automatically generated, so no manual working there except one field, but this also has docValue=true. Just curious, if the field is not a string/text, can it be docValue=false or still better to have true? And as for uninversion, then we are not using much facets nor other specific things in query, just simple queries.
Though I must say we are updating documents quite a bunch, but usage of CPU for being so high, not sure about that. Older version seemed not using CPU so much...
I am a bit running out of ideas and hoping that this will continue to work, but I dont like the CPU usage even over night, when nobody uses it. We will try to figure out the issue here and I hope I can ask more questions when in doubt or out of ideas. Also I must admit, solr is really new for me personally.
Jaan
-----Original Message-----
From: Walter Underwood <wu...@wunderwood.org>
Sent: 27 October 2020 18:44
To: solr-user@lucene.apache.org
Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
That first graph shows a JVM that does not have enough heap for the program it is running. Look at the bottom of the dips. That is the amount of memory still in use after a full GC.
You want those dips to drop to about half of the available heap, so I’d immediately increase that heap to 4G. That might not be enough, so you’ll need to to watch that graph after the increase.
I’ve been using 8G heaps with Solr since version 1.2. We run this config with Java 8 on over 100 machines. We do not do any faceting, which can take more memory.
SOLR_HEAP=8g
# Use G1 GC -- wunder 2017-01-23
# Settings from https://wiki.apache.org/solr/ShawnHeisey
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=200 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \
"
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Oct 27, 2020, at 12:48 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>
> Hello,
>
> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
> We use it on a Windows Server 2019.
> Java version is 11
> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
> Document sizes are not big, I guess. We only use one core.
> Document stats are here:
> Num Docs: 3627341
> Max Doc: 4981019
> Heap Memory Usage: 434400
> Deleted Docs: 1353678
> Version: 15999036
> Segment Count: 30
>
> The size of index is 2.66GB
>
> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
> If needed more information about background of the system, I am happy to help.
>
>
> But now to the issue I am having.
> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>
> Heap memory usage via JMX and jconsole ->
> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view
> ?usp=sharing As you can see, it starts of normal, but then goes crazy
> and it has been like this over night.
>
> This is overall monitoring graphs, as you can see CPU is working hard
> or hardly working. ->
> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view
> ?usp=sharing VM summary can be found here ->
> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view
> ?usp=sharing And finally to have better and quick overview of the SOLR
> executing parameters that I have ->
> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view
> ?usp=sharing
>
> If you can point me what I have to do to make it work, then I appreciate it a lot.
>
> Thank you in advance.
>
> Best regards,
> Jaan
>
>
Re: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Walter Underwood <wu...@wunderwood.org>.
That first graph shows a JVM that does not have enough heap for the
program it is running. Look at the bottom of the dips. That is the amount
of memory still in use after a full GC.
You want those dips to drop to about half of the available heap, so I’d
immediately increase that heap to 4G. That might not be enough, so
you’ll need to to watch that graph after the increase.
I’ve been using 8G heaps with Solr since version 1.2. We run this config
with Java 8 on over 100 machines. We do not do any faceting, which
can take more memory.
SOLR_HEAP=8g
# Use G1 GC -- wunder 2017-01-23
# Settings from https://wiki.apache.org/solr/ShawnHeisey
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=200 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \
"
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Oct 27, 2020, at 12:48 AM, Jaan Arjasepp <Ja...@coop.ee> wrote:
>
> Hello,
>
> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
> We use it on a Windows Server 2019.
> Java version is 11
> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
> Document sizes are not big, I guess. We only use one core.
> Document stats are here:
> Num Docs: 3627341
> Max Doc: 4981019
> Heap Memory Usage: 434400
> Deleted Docs: 1353678
> Version: 15999036
> Segment Count: 30
>
> The size of index is 2.66GB
>
> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
> If needed more information about background of the system, I am happy to help.
>
>
> But now to the issue I am having.
> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>
> Heap memory usage via JMX and jconsole -> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view?usp=sharing
> As you can see, it starts of normal, but then goes crazy and it has been like this over night.
>
> This is overall monitoring graphs, as you can see CPU is working hard or hardly working. -> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view?usp=sharing
> VM summary can be found here -> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view?usp=sharing
> And finally to have better and quick overview of the SOLR executing parameters that I have -> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view?usp=sharing
>
> If you can point me what I have to do to make it work, then I appreciate it a lot.
>
> Thank you in advance.
>
> Best regards,
> Jaan
>
>
RE: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Jaan Arjasepp <Ja...@coop.ee>.
I found one little difference from old solrconfig and new one.
It is in requestDispatchers section
It does not have this, but we had this in old configuration. Maybe it helps, I will see.
<requestParsers enableRemoteStreaming="true"
multipartUploadLimitInKB="2048000"
formdataUploadLimitInKB="2048"
addHttpRequestToContext="false"/>
Jaan
-----Original Message-----
From: Jaan Arjasepp <Ja...@coop.ee>
Sent: 27 October 2020 14:05
To: solr-user@lucene.apache.org
Subject: RE: SOLR uses too much CPU and GC is also weird on Windows server
Hi Emir,
I checked the solrconfig.xml file and we dont even use fieldValueCache. Also are you saying, I should check the schema and all the fields in the old solr and the new one to see if they match or contain similar settings? What does this uninverted value means? How to check this?
As for filtercache, it is default setting, should I change it?
I mean if you refer to these issues, then I guess it is either changing configuration or schema?
I will add my solrconfig.xml without comments so it is less data here, this is pretty default, nothing changed here:
<?xml version="1.0" encoding="UTF-8" ?>
<config>
<luceneMatchVersion>8.6.1</luceneMatchVersion>
<dataDir>${solr.data.dir:}</dataDir>
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
<codecFactory class="solr.SchemaCodecFactory"/>
<indexConfig>
<lockType>${solr.lock.type:native}</lockType>
</indexConfig>
<jmx />
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog>
<str name="dir">${solr.ulog.dir:}</str>
<int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
</updateLog>
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
</autoSoftCommit>
</updateHandler>
<query>
<maxBooleanClauses>${solr.max.booleanClauses:1024}</maxBooleanClauses>
<filterCache class="solr.FastLRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<queryResultCache class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<documentCache class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<cache name="perSegFilter"
class="solr.search.LRUCache"
size="10"
initialSize="0"
autowarmCount="10"
regenerator="solr.NoOpRegenerator" />
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<queryResultWindowSize>20</queryResultWindowSize>
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
</arr>
</listener>
<useColdSearcher>false</useColdSearcher>
</query>
<requestDispatcher>
<httpCaching never304="true" />
</requestDispatcher>
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
</lst>
</requestHandler>
<requestHandler name="/query" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="wt">json</str>
<str name="indent">true</str>
</lst>
</requestHandler>
<initParams path="/update/**,/query,/select,/spell">
<lst name="defaults">
<str name="df">_text_</str>
</lst>
</initParams>
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">text_general</str>
<lst name="spellchecker">
<str name="name">default</str>
<str name="field">_text_</str>
<str name="classname">solr.DirectSolrSpellChecker</str>
<str name="distanceMeasure">internal</str>
<float name="accuracy">0.5</float>
<int name="maxEdits">2</int>
<int name="minPrefix">1</int>
<int name="maxInspections">5</int>
<int name="minQueryLength">4</int>
<float name="maxQueryFrequency">0.01</float>
</lst>
</searchComponent>
<requestHandler name="/spell" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="spellcheck.dictionary">default</str>
<str name="spellcheck">on</str>
<str name="spellcheck.extendedResults">true</str>
<str name="spellcheck.count">10</str>
<str name="spellcheck.alternativeTermCount">5</str>
<str name="spellcheck.maxResultsForSuggest">5</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.collateExtendedResults">true</str>
<str name="spellcheck.maxCollationTries">10</str>
<str name="spellcheck.maxCollations">5</str>
</lst>
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
<searchComponent name="terms" class="solr.TermsComponent"/>
<requestHandler name="/terms" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<bool name="terms">true</bool>
<bool name="distrib">false</bool>
</lst>
<arr name="components">
<str>terms</str>
</arr>
</requestHandler>
<searchComponent class="solr.HighlightComponent" name="highlight">
<highlighting>
<fragmenter name="gap"
default="true"
class="solr.highlight.GapFragmenter">
<lst name="defaults">
<int name="hl.fragsize">100</int>
</lst>
</fragmenter>
<fragmenter name="regex"
class="solr.highlight.RegexFragmenter">
<lst name="defaults">
<int name="hl.fragsize">70</int>
<float name="hl.regex.slop">0.5</float>
<str name="hl.regex.pattern">[-\w ,/\n\"']{20,200}</str>
</lst>
</fragmenter>
<formatter name="html"
default="true"
class="solr.highlight.HtmlFormatter">
<lst name="defaults">
<str name="hl.simple.pre"><![CDATA[<em>]]></str>
<str name="hl.simple.post"><![CDATA[</em>]]></str>
</lst>
</formatter>
<encoder name="html"
class="solr.highlight.HtmlEncoder" />
<fragListBuilder name="simple"
class="solr.highlight.SimpleFragListBuilder"/>
<fragListBuilder name="single"
class="solr.highlight.SingleFragListBuilder"/>
<fragListBuilder name="weighted"
default="true"
class="solr.highlight.WeightedFragListBuilder"/>
<fragmentsBuilder name="default"
default="true"
class="solr.highlight.ScoreOrderFragmentsBuilder">
</fragmentsBuilder>
<fragmentsBuilder name="colored"
class="solr.highlight.ScoreOrderFragmentsBuilder">
<lst name="defaults">
<str name="hl.tag.pre"><![CDATA[
<b style="background:yellow">,<b style="background:lawgreen">,
<b style="background:aquamarine">,<b style="background:magenta">,
<b style="background:palegreen">,<b style="background:coral">,
<b style="background:wheat">,<b style="background:khaki">,
<b style="background:lime">,<b style="background:deepskyblue">]]></str>
<str name="hl.tag.post"><![CDATA[</b>]]></str>
</lst>
</fragmentsBuilder>
<boundaryScanner name="default"
default="true"
class="solr.highlight.SimpleBoundaryScanner">
<lst name="defaults">
<str name="hl.bs.maxScan">10</str>
<str name="hl.bs.chars">.,!? 	 </str>
</lst>
</boundaryScanner>
<boundaryScanner name="breakIterator"
class="solr.highlight.BreakIteratorBoundaryScanner">
<lst name="defaults">
<str name="hl.bs.type">WORD</str>
<str name="hl.bs.language">en</str>
<str name="hl.bs.country">US</str>
</lst>
</boundaryScanner>
</highlighting>
</searchComponent>
<updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/>
<updateProcessor class="solr.RemoveBlankFieldUpdateProcessorFactory" name="remove-blank"/>
<updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating">
<str name="pattern">[^\w-\.]</str>
<str name="replacement">_</str>
</updateProcessor>
<updateProcessor class="solr.ParseBooleanFieldUpdateProcessorFactory" name="parse-boolean"/>
<updateProcessor class="solr.ParseLongFieldUpdateProcessorFactory" name="parse-long"/>
<updateProcessor class="solr.ParseDoubleFieldUpdateProcessorFactory" name="parse-double"/>
<updateProcessor class="solr.ParseDateFieldUpdateProcessorFactory" name="parse-date">
<arr name="format">
<str>yyyy-MM-dd['T'[HH:mm[:ss[.SSS]][z</str>
<str>yyyy-MM-dd['T'[HH:mm[:ss[,SSS]][z</str>
<str>yyyy-MM-dd HH:mm[:ss[.SSS]][z</str>
<str>yyyy-MM-dd HH:mm[:ss[,SSS]][z</str>
<str>[EEE, ]dd MMM yyyy HH:mm[:ss] z</str>
<str>EEEE, dd-MMM-yy HH:mm:ss z</str>
<str>EEE MMM ppd HH:mm:ss [z ]yyyy</str>
</arr>
</updateProcessor>
<updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
<lst name="typeMapping">
<str name="valueClass">java.lang.String</str>
<str name="fieldType">text_general</str>
<lst name="copyField">
<str name="dest">*_str</str>
<int name="maxChars">256</int>
</lst>
<bool name="default">true</bool>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Boolean</str>
<str name="fieldType">booleans</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.util.Date</str>
<str name="fieldType">pdates</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Long</str>
<str name="valueClass">java.lang.Integer</str>
<str name="fieldType">plongs</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Number</str>
<str name="fieldType">pdoubles</str>
</lst>
</updateProcessor>
<updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}"
processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
<processor class="solr.LogUpdateProcessorFactory"/>
<processor class="solr.DistributedUpdateProcessorFactory"/>
<processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
<queryResponseWriter name="json" class="solr.JSONResponseWriter">
<str name="content-type">text/plain; charset=UTF-8</str>
</queryResponseWriter>
</config>
-----Original Message-----
From: Emir Arnautović <em...@sematext.com>
Sent: 27 October 2020 10:42
To: solr-user@lucene.apache.org
Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
Hi Jaan,
It can be several things:
caches
fieldCache/fieldValueCache - it can be that you you are missing doc values on some fields that are used for faceting/sorting/functions and that uninverted field structures are eating your memory.
filterCache - you’ve changed setting for filter caches and set it to some large value heavy queries return a lot of documents facet on high cardinality fields deep pagination
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 27 Oct 2020, at 08:48, Jaan Arjasepp <Ja...@coop.ee> wrote:
>
> Hello,
>
> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
> We use it on a Windows Server 2019.
> Java version is 11
> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
> Document sizes are not big, I guess. We only use one core.
> Document stats are here:
> Num Docs: 3627341
> Max Doc: 4981019
> Heap Memory Usage: 434400
> Deleted Docs: 1353678
> Version: 15999036
> Segment Count: 30
>
> The size of index is 2.66GB
>
> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
> If needed more information about background of the system, I am happy to help.
>
>
> But now to the issue I am having.
> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>
> Heap memory usage via JMX and jconsole ->
> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view
> ?usp=sharing As you can see, it starts of normal, but then goes crazy
> and it has been like this over night.
>
> This is overall monitoring graphs, as you can see CPU is working hard
> or hardly working. ->
> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view
> ?usp=sharing VM summary can be found here ->
> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view
> ?usp=sharing And finally to have better and quick overview of the SOLR
> executing parameters that I have ->
> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view
> ?usp=sharing
>
> If you can point me what I have to do to make it work, then I appreciate it a lot.
>
> Thank you in advance.
>
> Best regards,
> Jaan
>
>
RE: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Jaan Arjasepp <Ja...@coop.ee>.
Hi Emir,
I checked the solrconfig.xml file and we dont even use fieldValueCache. Also are you saying, I should check the schema and all the fields in the old solr and the new one to see if they match or contain similar settings? What does this uninverted value means? How to check this?
As for filtercache, it is default setting, should I change it?
I mean if you refer to these issues, then I guess it is either changing configuration or schema?
I will add my solrconfig.xml without comments so it is less data here, this is pretty default, nothing changed here:
<?xml version="1.0" encoding="UTF-8" ?>
<config>
<luceneMatchVersion>8.6.1</luceneMatchVersion>
<dataDir>${solr.data.dir:}</dataDir>
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
<codecFactory class="solr.SchemaCodecFactory"/>
<indexConfig>
<lockType>${solr.lock.type:native}</lockType>
</indexConfig>
<jmx />
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog>
<str name="dir">${solr.ulog.dir:}</str>
<int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
</updateLog>
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
</autoSoftCommit>
</updateHandler>
<query>
<maxBooleanClauses>${solr.max.booleanClauses:1024}</maxBooleanClauses>
<filterCache class="solr.FastLRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<queryResultCache class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<documentCache class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<cache name="perSegFilter"
class="solr.search.LRUCache"
size="10"
initialSize="0"
autowarmCount="10"
regenerator="solr.NoOpRegenerator" />
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<queryResultWindowSize>20</queryResultWindowSize>
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
</arr>
</listener>
<useColdSearcher>false</useColdSearcher>
</query>
<requestDispatcher>
<httpCaching never304="true" />
</requestDispatcher>
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
</lst>
</requestHandler>
<requestHandler name="/query" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="wt">json</str>
<str name="indent">true</str>
</lst>
</requestHandler>
<initParams path="/update/**,/query,/select,/spell">
<lst name="defaults">
<str name="df">_text_</str>
</lst>
</initParams>
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">text_general</str>
<lst name="spellchecker">
<str name="name">default</str>
<str name="field">_text_</str>
<str name="classname">solr.DirectSolrSpellChecker</str>
<str name="distanceMeasure">internal</str>
<float name="accuracy">0.5</float>
<int name="maxEdits">2</int>
<int name="minPrefix">1</int>
<int name="maxInspections">5</int>
<int name="minQueryLength">4</int>
<float name="maxQueryFrequency">0.01</float>
</lst>
</searchComponent>
<requestHandler name="/spell" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="spellcheck.dictionary">default</str>
<str name="spellcheck">on</str>
<str name="spellcheck.extendedResults">true</str>
<str name="spellcheck.count">10</str>
<str name="spellcheck.alternativeTermCount">5</str>
<str name="spellcheck.maxResultsForSuggest">5</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.collateExtendedResults">true</str>
<str name="spellcheck.maxCollationTries">10</str>
<str name="spellcheck.maxCollations">5</str>
</lst>
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
<searchComponent name="terms" class="solr.TermsComponent"/>
<requestHandler name="/terms" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<bool name="terms">true</bool>
<bool name="distrib">false</bool>
</lst>
<arr name="components">
<str>terms</str>
</arr>
</requestHandler>
<searchComponent class="solr.HighlightComponent" name="highlight">
<highlighting>
<fragmenter name="gap"
default="true"
class="solr.highlight.GapFragmenter">
<lst name="defaults">
<int name="hl.fragsize">100</int>
</lst>
</fragmenter>
<fragmenter name="regex"
class="solr.highlight.RegexFragmenter">
<lst name="defaults">
<int name="hl.fragsize">70</int>
<float name="hl.regex.slop">0.5</float>
<str name="hl.regex.pattern">[-\w ,/\n\"']{20,200}</str>
</lst>
</fragmenter>
<formatter name="html"
default="true"
class="solr.highlight.HtmlFormatter">
<lst name="defaults">
<str name="hl.simple.pre"><![CDATA[<em>]]></str>
<str name="hl.simple.post"><![CDATA[</em>]]></str>
</lst>
</formatter>
<encoder name="html"
class="solr.highlight.HtmlEncoder" />
<fragListBuilder name="simple"
class="solr.highlight.SimpleFragListBuilder"/>
<fragListBuilder name="single"
class="solr.highlight.SingleFragListBuilder"/>
<fragListBuilder name="weighted"
default="true"
class="solr.highlight.WeightedFragListBuilder"/>
<fragmentsBuilder name="default"
default="true"
class="solr.highlight.ScoreOrderFragmentsBuilder">
</fragmentsBuilder>
<fragmentsBuilder name="colored"
class="solr.highlight.ScoreOrderFragmentsBuilder">
<lst name="defaults">
<str name="hl.tag.pre"><![CDATA[
<b style="background:yellow">,<b style="background:lawgreen">,
<b style="background:aquamarine">,<b style="background:magenta">,
<b style="background:palegreen">,<b style="background:coral">,
<b style="background:wheat">,<b style="background:khaki">,
<b style="background:lime">,<b style="background:deepskyblue">]]></str>
<str name="hl.tag.post"><![CDATA[</b>]]></str>
</lst>
</fragmentsBuilder>
<boundaryScanner name="default"
default="true"
class="solr.highlight.SimpleBoundaryScanner">
<lst name="defaults">
<str name="hl.bs.maxScan">10</str>
<str name="hl.bs.chars">.,!? 	 </str>
</lst>
</boundaryScanner>
<boundaryScanner name="breakIterator"
class="solr.highlight.BreakIteratorBoundaryScanner">
<lst name="defaults">
<str name="hl.bs.type">WORD</str>
<str name="hl.bs.language">en</str>
<str name="hl.bs.country">US</str>
</lst>
</boundaryScanner>
</highlighting>
</searchComponent>
<updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/>
<updateProcessor class="solr.RemoveBlankFieldUpdateProcessorFactory" name="remove-blank"/>
<updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating">
<str name="pattern">[^\w-\.]</str>
<str name="replacement">_</str>
</updateProcessor>
<updateProcessor class="solr.ParseBooleanFieldUpdateProcessorFactory" name="parse-boolean"/>
<updateProcessor class="solr.ParseLongFieldUpdateProcessorFactory" name="parse-long"/>
<updateProcessor class="solr.ParseDoubleFieldUpdateProcessorFactory" name="parse-double"/>
<updateProcessor class="solr.ParseDateFieldUpdateProcessorFactory" name="parse-date">
<arr name="format">
<str>yyyy-MM-dd['T'[HH:mm[:ss[.SSS]][z</str>
<str>yyyy-MM-dd['T'[HH:mm[:ss[,SSS]][z</str>
<str>yyyy-MM-dd HH:mm[:ss[.SSS]][z</str>
<str>yyyy-MM-dd HH:mm[:ss[,SSS]][z</str>
<str>[EEE, ]dd MMM yyyy HH:mm[:ss] z</str>
<str>EEEE, dd-MMM-yy HH:mm:ss z</str>
<str>EEE MMM ppd HH:mm:ss [z ]yyyy</str>
</arr>
</updateProcessor>
<updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
<lst name="typeMapping">
<str name="valueClass">java.lang.String</str>
<str name="fieldType">text_general</str>
<lst name="copyField">
<str name="dest">*_str</str>
<int name="maxChars">256</int>
</lst>
<bool name="default">true</bool>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Boolean</str>
<str name="fieldType">booleans</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.util.Date</str>
<str name="fieldType">pdates</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Long</str>
<str name="valueClass">java.lang.Integer</str>
<str name="fieldType">plongs</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Number</str>
<str name="fieldType">pdoubles</str>
</lst>
</updateProcessor>
<updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}"
processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
<processor class="solr.LogUpdateProcessorFactory"/>
<processor class="solr.DistributedUpdateProcessorFactory"/>
<processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
<queryResponseWriter name="json" class="solr.JSONResponseWriter">
<str name="content-type">text/plain; charset=UTF-8</str>
</queryResponseWriter>
</config>
-----Original Message-----
From: Emir Arnautović <em...@sematext.com>
Sent: 27 October 2020 10:42
To: solr-user@lucene.apache.org
Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
Hi Jaan,
It can be several things:
caches
fieldCache/fieldValueCache - it can be that you you are missing doc values on some fields that are used for faceting/sorting/functions and that uninverted field structures are eating your memory.
filterCache - you’ve changed setting for filter caches and set it to some large value heavy queries return a lot of documents facet on high cardinality fields deep pagination
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 27 Oct 2020, at 08:48, Jaan Arjasepp <Ja...@coop.ee> wrote:
>
> Hello,
>
> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
> We use it on a Windows Server 2019.
> Java version is 11
> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
> Document sizes are not big, I guess. We only use one core.
> Document stats are here:
> Num Docs: 3627341
> Max Doc: 4981019
> Heap Memory Usage: 434400
> Deleted Docs: 1353678
> Version: 15999036
> Segment Count: 30
>
> The size of index is 2.66GB
>
> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
> If needed more information about background of the system, I am happy to help.
>
>
> But now to the issue I am having.
> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>
> Heap memory usage via JMX and jconsole ->
> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view
> ?usp=sharing As you can see, it starts of normal, but then goes crazy
> and it has been like this over night.
>
> This is overall monitoring graphs, as you can see CPU is working hard
> or hardly working. ->
> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view
> ?usp=sharing VM summary can be found here ->
> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view
> ?usp=sharing And finally to have better and quick overview of the SOLR
> executing parameters that I have ->
> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view
> ?usp=sharing
>
> If you can point me what I have to do to make it work, then I appreciate it a lot.
>
> Thank you in advance.
>
> Best regards,
> Jaan
>
>
Re: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Emir Arnautović <em...@sematext.com>.
Hi Jaan,
You can also check in admin console in caches the sizes of field* caches. That will tell you if some field needs docValues=true.
Regards,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 27 Oct 2020, at 14:36, Jaan Arjasepp <Ja...@coop.ee> wrote:
>
> Hi Erick,
>
> Thanks for this information, I will look into it.
> Main changes were regarding parsing the results JSON got from solr, not the queries or updates.
>
> Jaan
>
> P.S. configuration change about requestParser was not it.
>
>
> -----Original Message-----
> From: Erick Erickson <erickerickson@gmail.com <ma...@gmail.com>>
> Sent: 27 October 2020 15:03
> To: solr-user@lucene.apache.org <ma...@lucene.apache.org>
> Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
>
> Jean:
>
> The basic search uses an “inverted index”, which is basically a list of terms and the documents they appear in, e.g.
> my - 1, 4, 9, 12
> dog - 4, 8, 10
>
> So the word “my” appears in docs 1, 4, 9 and 12, and “dog” appears in 4, 8, 10. Makes it easy to search for my AND dog for instance, obviously both appear in doc 4.
>
> But that’s a lousy structure for faceting, where you have a list of documents and are trying to find the terms it has to count them up. For that, you want to “uninvert” the above structure,
> 1 - my
> 4 - my dog
> 8 - dog
> 9 - my
> 10 - dog
> 12 - my
>
> From there, it’s easy to say “count the distinct terms for docs 1 and 4 and put them in a bucket”, giving facet counts like
>
> my (2)
> dog (1)
>
> If docValues=true, then the second structure is built at index time and occupies memory at run time out in MMapDirectory space, i.e. _not_ on the heap.
>
> If docValues=false, the second structure is built _on_ the heap when it’s needed, adding to GC, memory pressure, CPU utilization etc.
>
> So one theory is that when you upgraded your system (and you did completely rebuild your corpus, right?) you inadvertently changed the docValues property for one or more fields that you facet, group, sort, or use function queries on and Solr is doing all the extra work of uninverting the field that it didn’t have to before.
>
> To answer that, you need to go through your schema and insure that docValues=true is set for any field you facet, group, sort, or use function queries on. If you do change this value, you need to blow away your index so there are no segments and index all your documents again.
>
> But that theory has problems:
> 1> why should Solr run for a while and then go crazy? It’d have to be
> 1> that the query that
> triggers uninversion is uncommon.
> 2> docValues defaults to true for simple types in recent schemas.
> 2> Perhaps you pulled
> over an old definition from your former schema?
>
>
> One other thing: you mention a bit of custom code you needed to change. I always try to investigate that first. Is it possible to
> 1> reproduce the problem no a non-prod system
> 2> see what happens if you take the custom code out?
>
> Best,
> Erick
>
>
>> On Oct 27, 2020, at 4:42 AM, Emir Arnautović <em...@sematext.com> wrote:
>>
>> Hi Jaan,
>> It can be several things:
>> caches
>> fieldCache/fieldValueCache - it can be that you you are missing doc values on some fields that are used for faceting/sorting/functions and that uninverted field structures are eating your memory.
>> filterCache - you’ve changed setting for filter caches and set it to
>> some large value heavy queries return a lot of documents facet on high
>> cardinality fields deep pagination
>>
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection Solr &
>> Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>>
>>> On 27 Oct 2020, at 08:48, Jaan Arjasepp <Ja...@coop.ee> wrote:
>>>
>>> Hello,
>>>
>>> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
>>> We use it on a Windows Server 2019.
>>> Java version is 11
>>> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
>>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
>>> Document sizes are not big, I guess. We only use one core.
>>> Document stats are here:
>>> Num Docs: 3627341
>>> Max Doc: 4981019
>>> Heap Memory Usage: 434400
>>> Deleted Docs: 1353678
>>> Version: 15999036
>>> Segment Count: 30
>>>
>>> The size of index is 2.66GB
>>>
>>> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
>>> If needed more information about background of the system, I am happy to help.
>>>
>>>
>>> But now to the issue I am having.
>>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
>>> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>>>
>>> Heap memory usage via JMX and jconsole ->
>>> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/vie
>>> w?usp=sharing As you can see, it starts of normal, but then goes
>>> crazy and it has been like this over night.
>>>
>>> This is overall monitoring graphs, as you can see CPU is working hard
>>> or hardly working. ->
>>> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/vie
>>> w?usp=sharing VM summary can be found here ->
>>> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/vie
>>> w?usp=sharing And finally to have better and quick overview of the
>>> SOLR executing parameters that I have ->
>>> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/vie
>>> w?usp=sharing
>>>
>>> If you can point me what I have to do to make it work, then I appreciate it a lot.
>>>
>>> Thank you in advance.
>>>
>>> Best regards,
>>> Jaan
RE: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Jaan Arjasepp <Ja...@coop.ee>.
Hi Erick,
Thanks for this information, I will look into it.
Main changes were regarding parsing the results JSON got from solr, not the queries or updates.
Jaan
P.S. configuration change about requestParser was not it.
-----Original Message-----
From: Erick Erickson <er...@gmail.com>
Sent: 27 October 2020 15:03
To: solr-user@lucene.apache.org
Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
Jean:
The basic search uses an “inverted index”, which is basically a list of terms and the documents they appear in, e.g.
my - 1, 4, 9, 12
dog - 4, 8, 10
So the word “my” appears in docs 1, 4, 9 and 12, and “dog” appears in 4, 8, 10. Makes it easy to search for my AND dog for instance, obviously both appear in doc 4.
But that’s a lousy structure for faceting, where you have a list of documents and are trying to find the terms it has to count them up. For that, you want to “uninvert” the above structure,
1 - my
4 - my dog
8 - dog
9 - my
10 - dog
12 - my
From there, it’s easy to say “count the distinct terms for docs 1 and 4 and put them in a bucket”, giving facet counts like
my (2)
dog (1)
If docValues=true, then the second structure is built at index time and occupies memory at run time out in MMapDirectory space, i.e. _not_ on the heap.
If docValues=false, the second structure is built _on_ the heap when it’s needed, adding to GC, memory pressure, CPU utilization etc.
So one theory is that when you upgraded your system (and you did completely rebuild your corpus, right?) you inadvertently changed the docValues property for one or more fields that you facet, group, sort, or use function queries on and Solr is doing all the extra work of uninverting the field that it didn’t have to before.
To answer that, you need to go through your schema and insure that docValues=true is set for any field you facet, group, sort, or use function queries on. If you do change this value, you need to blow away your index so there are no segments and index all your documents again.
But that theory has problems:
1> why should Solr run for a while and then go crazy? It’d have to be
1> that the query that
triggers uninversion is uncommon.
2> docValues defaults to true for simple types in recent schemas.
2> Perhaps you pulled
over an old definition from your former schema?
One other thing: you mention a bit of custom code you needed to change. I always try to investigate that first. Is it possible to
1> reproduce the problem no a non-prod system
2> see what happens if you take the custom code out?
Best,
Erick
> On Oct 27, 2020, at 4:42 AM, Emir Arnautović <em...@sematext.com> wrote:
>
> Hi Jaan,
> It can be several things:
> caches
> fieldCache/fieldValueCache - it can be that you you are missing doc values on some fields that are used for faceting/sorting/functions and that uninverted field structures are eating your memory.
> filterCache - you’ve changed setting for filter caches and set it to
> some large value heavy queries return a lot of documents facet on high
> cardinality fields deep pagination
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr &
> Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 27 Oct 2020, at 08:48, Jaan Arjasepp <Ja...@coop.ee> wrote:
>>
>> Hello,
>>
>> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
>> We use it on a Windows Server 2019.
>> Java version is 11
>> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
>> Document sizes are not big, I guess. We only use one core.
>> Document stats are here:
>> Num Docs: 3627341
>> Max Doc: 4981019
>> Heap Memory Usage: 434400
>> Deleted Docs: 1353678
>> Version: 15999036
>> Segment Count: 30
>>
>> The size of index is 2.66GB
>>
>> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
>> If needed more information about background of the system, I am happy to help.
>>
>>
>> But now to the issue I am having.
>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
>> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>>
>> Heap memory usage via JMX and jconsole ->
>> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/vie
>> w?usp=sharing As you can see, it starts of normal, but then goes
>> crazy and it has been like this over night.
>>
>> This is overall monitoring graphs, as you can see CPU is working hard
>> or hardly working. ->
>> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/vie
>> w?usp=sharing VM summary can be found here ->
>> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/vie
>> w?usp=sharing And finally to have better and quick overview of the
>> SOLR executing parameters that I have ->
>> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/vie
>> w?usp=sharing
>>
>> If you can point me what I have to do to make it work, then I appreciate it a lot.
>>
>> Thank you in advance.
>>
>> Best regards,
>> Jaan
>>
>>
>
Re: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Erick Erickson <er...@gmail.com>.
Jean:
The basic search uses an “inverted index”, which is basically a list of terms and the documents they appear in, e.g.
my - 1, 4, 9, 12
dog - 4, 8, 10
So the word “my” appears in docs 1, 4, 9 and 12, and “dog” appears in 4, 8, 10. Makes
it easy to search for
my AND dog
for instance, obviously both appear in doc 4.
But that’s a lousy structure for faceting, where you have a list of documents and are trying to
find the terms it has to count them up. For that, you want to “uninvert” the above structure,
1 - my
4 - my dog
8 - dog
9 - my
10 - dog
12 - my
From there, it’s easy to say “count the distinct terms for docs 1 and 4 and put them in a bucket”,
giving facet counts like
my (2)
dog (1)
If docValues=true, then the second structure is built at index time and occupies memory at
run time out in MMapDirectory space, i.e. _not_ on the heap.
If docValues=false, the second structure is built _on_ the heap when it’s needed, adding to
GC, memory pressure, CPU utilization etc.
So one theory is that when you upgraded your system (and you did completely rebuild your
corpus, right?) you inadvertently changed the docValues property for one or more fields that you
facet, group, sort, or use function queries on and Solr is doing all the extra work of
uninverting the field that it didn’t have to before.
To answer that, you need to go through your schema and insure that docValues=true is
set for any field you facet, group, sort, or use function queries on. If you do change
this value, you need to blow away your index so there are no segments and index
all your documents again.
But that theory has problems:
1> why should Solr run for a while and then go crazy? It’d have to be that the query that
triggers uninversion is uncommon.
2> docValues defaults to true for simple types in recent schemas. Perhaps you pulled
over an old definition from your former schema?
One other thing: you mention a bit of custom code you needed to change. I always try to
investigate that first. Is it possible to
1> reproduce the problem no a non-prod system
2> see what happens if you take the custom code out?
Best,
Erick
> On Oct 27, 2020, at 4:42 AM, Emir Arnautović <em...@sematext.com> wrote:
>
> Hi Jaan,
> It can be several things:
> caches
> fieldCache/fieldValueCache - it can be that you you are missing doc values on some fields that are used for faceting/sorting/functions and that uninverted field structures are eating your memory.
> filterCache - you’ve changed setting for filter caches and set it to some large value
> heavy queries
> return a lot of documents
> facet on high cardinality fields
> deep pagination
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 27 Oct 2020, at 08:48, Jaan Arjasepp <Ja...@coop.ee> wrote:
>>
>> Hello,
>>
>> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
>> We use it on a Windows Server 2019.
>> Java version is 11
>> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
>> Document sizes are not big, I guess. We only use one core.
>> Document stats are here:
>> Num Docs: 3627341
>> Max Doc: 4981019
>> Heap Memory Usage: 434400
>> Deleted Docs: 1353678
>> Version: 15999036
>> Segment Count: 30
>>
>> The size of index is 2.66GB
>>
>> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
>> If needed more information about background of the system, I am happy to help.
>>
>>
>> But now to the issue I am having.
>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
>> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>>
>> Heap memory usage via JMX and jconsole -> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view?usp=sharing
>> As you can see, it starts of normal, but then goes crazy and it has been like this over night.
>>
>> This is overall monitoring graphs, as you can see CPU is working hard or hardly working. -> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view?usp=sharing
>> VM summary can be found here -> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view?usp=sharing
>> And finally to have better and quick overview of the SOLR executing parameters that I have -> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view?usp=sharing
>>
>> If you can point me what I have to do to make it work, then I appreciate it a lot.
>>
>> Thank you in advance.
>>
>> Best regards,
>> Jaan
>>
>>
>
Re: SOLR uses too much CPU and GC is also weird on Windows server
Posted by Emir Arnautović <em...@sematext.com>.
Hi Jaan,
It can be several things:
caches
fieldCache/fieldValueCache - it can be that you you are missing doc values on some fields that are used for faceting/sorting/functions and that uninverted field structures are eating your memory.
filterCache - you’ve changed setting for filter caches and set it to some large value
heavy queries
return a lot of documents
facet on high cardinality fields
deep pagination
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 27 Oct 2020, at 08:48, Jaan Arjasepp <Ja...@coop.ee> wrote:
>
> Hello,
>
> We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1.
> We use it on a Windows Server 2019.
> Java version is 11
> Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but it ran out of memory and stopped responding. Not sure if it was the issue. When older version, it managed fine with 512MB.
> SOLR is not in a cloud mode, but in solo mode as we use it internally and it does not have too many request nor indexing actually.
> Document sizes are not big, I guess. We only use one core.
> Document stats are here:
> Num Docs: 3627341
> Max Doc: 4981019
> Heap Memory Usage: 434400
> Deleted Docs: 1353678
> Version: 15999036
> Segment Count: 30
>
> The size of index is 2.66GB
>
> While making upgrade we had to modify one field and a bit of code that uses it. Thats basically it. It works.
> If needed more information about background of the system, I am happy to help.
>
>
> But now to the issue I am having.
> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not high, heap usage seem normal. All is good, but then suddenly, the heap usage goes crazy, going up and down, up and down and CPU rises to 50-60% of the usage. Also I noticed over the weekend, when there are no writing usage, the CPU remains low and decent. I can try it this weekend again to see if and how this works out.
> Also it seems to me, that after 4-5 days of working like this, it stops responding, but needs to be confirmed with more heap also.
>
> Heap memory usage via JMX and jconsole -> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view?usp=sharing
> As you can see, it starts of normal, but then goes crazy and it has been like this over night.
>
> This is overall monitoring graphs, as you can see CPU is working hard or hardly working. -> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view?usp=sharing
> VM summary can be found here -> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view?usp=sharing
> And finally to have better and quick overview of the SOLR executing parameters that I have -> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view?usp=sharing
>
> If you can point me what I have to do to make it work, then I appreciate it a lot.
>
> Thank you in advance.
>
> Best regards,
> Jaan
>
>