Re: High CPU utilization on Upgrading to Solr Version 6.3

[Atita Arora <at...@gmail.com>]

Hi All ,

Just thought of giving quick update on this.
So we were able to *knock down this issue by using jvisualvm* which comes
with java .
So , we enabled monitoring  through jmx and the CPU profiling showed (as
attached in one of my previous emails) *Highlighting taking maximum
processing.*
Mysteriously , this was happening in highlighting-> merge which was invoked
through when we enabled *mergecontiguous=true* I'm still surprised as to
turning this only property false, resolved the issue and we happily went
live last week.

Later , as I found the code for this particular property is causing endless
recursions as I traced.

Please guide / share if you may have any other thoughts.

Thanks,
Atita



On Fri, Jul 28, 2017 at 7:18 PM, Shawn Heisey <ap...@elyograg.org> wrote:

> On 7/27/2017 1:30 AM, Atita Arora wrote:
> > What OS is Solr running on?  I'm only asking because some additional
> > information I'm after has different gathering methods depending on OS.
> > Other questions:
> >
> > /*OpenJDK 64-Bit Server VM (25.141-b16) for linux-amd64 JRE
> > (1.8.0_141-b16), built on Jul 20 2017 21:47:59 by "mockbuild" with gcc
> > 4.4.7 20120313 (Red Hat 4.4.7-18)*/
> > /*Memory: 4k page, physical 264477520k(92198808k free), swap 0k(0k
> free)*/
>
> Linux is the easiest to get good information from.  Run the "top"
> program in a commandline session.  Press shift-M to sort by memory size,
> and grab a screenshot.  Share that screenshot with a file sharing site
> and give us the URL.
>
> > Is there only one Solr process per machine, or more than one?
> > /*On an average yes , one solr process per machine , however , we do
> > have a machine (where this log is taken) has two solr processes
> > (master and slave)*/
>
> Running a master and a slave on one machine does nothing for
> redundancy.  They need to be on separate machines for that to really
> help.  As for multiple processes per machine, tou can have many indexes
> in one Solr instance -- you don't need more than one in most cases.
>
> > How many total documents are managed by one machine?
> > */About 220945 per machine ( and double for this machine as it has
> > instance of master as well as other slave)/*
> >
> > How big is all the index data managed by one machine?
> > */The index is about 4G./*
>
> If less than a quarter of a million documents results in a 4GB index,
> those documents must be ENORMOUS, or else there is something strange
> going on.
>
> > What is the max heap on each Solr process?
> > */Max heap is 25G for each Solr Process. (Xms 25g Xmx 25g)/*
> > */
> > /*
> > The reason of choosing RAMDirectory was that it was used in the
> > similar manner while the production Solr was on Version 4.3.2, so no
> > particular reason but just replicated how it was working , never
> > thought this may give troubles.
>
> Set up the slaves just like the masters, with
> NRTCachingDirectoryFactory.  For a couple hundred thousand docs, you
> probably only need a 2GB heap, possibly even less.
>
> > I had included a pastebin of GC snapshot (the complete log was too big
> > to be included in the pastebin , so pasted a sampler)
>
> I asked for the full log because that's what I need to look deeper.  A
> sampler won't be enough.  There are file sharing websites for sharing
> larger content, and if you compress the file before uploading it, you
> should be able to achieve a fairly impressive compression ratio.
> Dropbox is generally a good choice for sharing fairly large content.
> Dropbox also works for image data, like the "top" screenshot I asked for
> above.
>
> > Another thing is as we observed the CPU cycles yesterday in high load
> > condition we observed that the Highlighter component was taking
> > longest , is there anything in particular we forgot to include that
> > highlighting doesn't gives a performance hit .
> > Attached is the snapshot taken from jvisualvm.
>
> Attachments rarely make it through the mailing list.  Yours didn't, so I
> cannot see that snapshot.
>
> I do not know anything about highlighting, so I cannot comment on how
> much CPU it takes.  I've never used the feature.
>
> My best idea about why your CPU is so high is problems with garbage
> collection.  To look into that, I need to have the full GC log.  The
> rest of the information I've asked for will help focus my efforts.
>
> Thanks,
> Shawn
>
>

Re: High CPU utilization on Upgrading to Solr Version 6.3

[Erick Erickson <er...@gmail.com>]

Atita:

Thanks for that update. I've opened SOLR-11188 to track this, please
add any details to that JIRA that you can, in particular what bit of
code you've identified as the problem.

Also, do you have any document you can share that would cause this?
I'm wondering if it's sensitive to the particular data or the query
or.....

Thanks again!
Erick

On Wed, Aug 2, 2017 at 11:38 PM, Atita Arora <at...@gmail.com> wrote:
> Hi All ,
>
> Just thought of giving quick update on this.
> So we were able to *knock down this issue by using jvisualvm* which comes
> with java .
> So , we enabled monitoring  through jmx and the CPU profiling showed (as
> attached in one of my previous emails) *Highlighting taking maximum
> processing.*
> Mysteriously , this was happening in highlighting-> merge which was invoked
> through when we enabled *mergecontiguous=true* I'm still surprised as to
> turning this only property false, resolved the issue and we happily went
> live last week.
>
> Later , as I found the code for this particular property is causing endless
> recursions as I traced.
>
> Please guide / share if you may have any other thoughts.
>
> Thanks,
> Atita
>
>
>
> On Fri, Jul 28, 2017 at 7:18 PM, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 7/27/2017 1:30 AM, Atita Arora wrote:
>> > What OS is Solr running on?  I'm only asking because some additional
>> > information I'm after has different gathering methods depending on OS.
>> > Other questions:
>> >
>> > /*OpenJDK 64-Bit Server VM (25.141-b16) for linux-amd64 JRE
>> > (1.8.0_141-b16), built on Jul 20 2017 21:47:59 by "mockbuild" with gcc
>> > 4.4.7 20120313 (Red Hat 4.4.7-18)*/
>> > /*Memory: 4k page, physical 264477520k(92198808k free), swap 0k(0k
>> free)*/
>>
>> Linux is the easiest to get good information from.  Run the "top"
>> program in a commandline session.  Press shift-M to sort by memory size,
>> and grab a screenshot.  Share that screenshot with a file sharing site
>> and give us the URL.
>>
>> > Is there only one Solr process per machine, or more than one?
>> > /*On an average yes , one solr process per machine , however , we do
>> > have a machine (where this log is taken) has two solr processes
>> > (master and slave)*/
>>
>> Running a master and a slave on one machine does nothing for
>> redundancy.  They need to be on separate machines for that to really
>> help.  As for multiple processes per machine, tou can have many indexes
>> in one Solr instance -- you don't need more than one in most cases.
>>
>> > How many total documents are managed by one machine?
>> > */About 220945 per machine ( and double for this machine as it has
>> > instance of master as well as other slave)/*
>> >
>> > How big is all the index data managed by one machine?
>> > */The index is about 4G./*
>>
>> If less than a quarter of a million documents results in a 4GB index,
>> those documents must be ENORMOUS, or else there is something strange
>> going on.
>>
>> > What is the max heap on each Solr process?
>> > */Max heap is 25G for each Solr Process. (Xms 25g Xmx 25g)/*
>> > */
>> > /*
>> > The reason of choosing RAMDirectory was that it was used in the
>> > similar manner while the production Solr was on Version 4.3.2, so no
>> > particular reason but just replicated how it was working , never
>> > thought this may give troubles.
>>
>> Set up the slaves just like the masters, with
>> NRTCachingDirectoryFactory.  For a couple hundred thousand docs, you
>> probably only need a 2GB heap, possibly even less.
>>
>> > I had included a pastebin of GC snapshot (the complete log was too big
>> > to be included in the pastebin , so pasted a sampler)
>>
>> I asked for the full log because that's what I need to look deeper.  A
>> sampler won't be enough.  There are file sharing websites for sharing
>> larger content, and if you compress the file before uploading it, you
>> should be able to achieve a fairly impressive compression ratio.
>> Dropbox is generally a good choice for sharing fairly large content.
>> Dropbox also works for image data, like the "top" screenshot I asked for
>> above.
>>
>> > Another thing is as we observed the CPU cycles yesterday in high load
>> > condition we observed that the Highlighter component was taking
>> > longest , is there anything in particular we forgot to include that
>> > highlighting doesn't gives a performance hit .
>> > Attached is the snapshot taken from jvisualvm.
>>
>> Attachments rarely make it through the mailing list.  Yours didn't, so I
>> cannot see that snapshot.
>>
>> I do not know anything about highlighting, so I cannot comment on how
>> much CPU it takes.  I've never used the feature.
>>
>> My best idea about why your CPU is so high is problems with garbage
>> collection.  To look into that, I need to have the full GC log.  The
>> rest of the information I've asked for will help focus my efforts.
>>
>> Thanks,
>> Shawn
>>
>>