You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Surya R <su...@gmail.com> on 2023/10/23 17:47:20 UTC
Solr Heap usage
Hi. We have a solr index with around 2.5M documents , running solr 8.11 .
We have bunch of fields that we faceet/sort/highlight on. Heap is 8GB, out
of 12 GB RAM
These fields were not defined with docValues=true due to some legacy design
issues , but if we were to rewrite the app, we would definitely have the
docvalues enabled for those fields that participate in
facet/sort/highlight.
Problem:
I do see garbage collection ( G1GC) kicking in when the traffic is moderate
to high, with a "pause full" event that takes around 2seconds to complete.
This event happens more often when the traffic is high.
I see this GC event happening when the humongous regions go beyond 1000.
The humongous region size is 4M
I took a heap dump and analyzed with VisualVM, but I couldnt pinpoint what
objects are using the 'humougous region' .
My question is, using the VisualVM, is it possible to pinpoint which / what
objects are using the 'humongous region' ?
If for example I have an solr indexed field called "answerSnippet" , if
that is the culprit, Is it possible to see the field answerSnippet in the
VisualVM heap dump output?
All i see is byte[] with 55% heap occupancy, but dont know what objects
constitute that byte[]
Can anybody throw some light on this ? I really appreciate the help.
thanks
-- Surya
Re: Solr Heap usage
Posted by Surya R <su...@gmail.com>.
Thank you Rajani and Arrieta, I will read the link. Ishan also reached out
to me earlier on with some insights , and am analyzing them as well.
I did take a heap dump using VisualVM and analyzing the details, after
loading through the visualVM tool.
thanks
Surya
On Tue, Oct 24, 2023 at 9:18 PM Arrieta, Alejandro <
aarrieta@perrinsoftware.com> wrote:
> Hello,
>
> this link may help you, it has examples:
> https://risdenk.github.io/2017/12/18/ambari-infra-solr-ranger.html
>
> Kind Regards,
> Alejandro Arrieta
>
> On Tue, Oct 24, 2023 at 9:18 PM rajani m <ra...@gmail.com> wrote:
>
> > When you see memory filling up on the visualvm gui, request a heap dump.
> > The heap dump file gets stored on the node's /tmp location. Use the
> eclipse
> > memory analyzer tool (MAT) to load the downloaded ".dmp" file, the MAT
> > analyzes the heap and generates a report of objects on the heap.
> >
> > On Mon, Oct 23, 2023 at 1:48 PM Surya R <su...@gmail.com> wrote:
> >
> > > Hi. We have a solr index with around 2.5M documents , running solr
> 8.11 .
> > > We have bunch of fields that we faceet/sort/highlight on. Heap is 8GB,
> > out
> > > of 12 GB RAM
> > >
> > > These fields were not defined with docValues=true due to some legacy
> > design
> > > issues , but if we were to rewrite the app, we would definitely have
> the
> > > docvalues enabled for those fields that participate in
> > > facet/sort/highlight.
> > >
> > > Problem:
> > > I do see garbage collection ( G1GC) kicking in when the traffic is
> > moderate
> > > to high, with a "pause full" event that takes around 2seconds to
> > complete.
> > > This event happens more often when the traffic is high.
> > >
> > > I see this GC event happening when the humongous regions go beyond
> 1000.
> > > The humongous region size is 4M
> > >
> > > I took a heap dump and analyzed with VisualVM, but I couldnt pinpoint
> > what
> > > objects are using the 'humougous region' .
> > >
> > > My question is, using the VisualVM, is it possible to pinpoint which /
> > what
> > > objects are using the 'humongous region' ?
> > >
> > > If for example I have an solr indexed field called "answerSnippet" ,
> if
> > > that is the culprit, Is it possible to see the field answerSnippet in
> the
> > > VisualVM heap dump output?
> > >
> > > All i see is byte[] with 55% heap occupancy, but dont know what objects
> > > constitute that byte[]
> > >
> > > Can anybody throw some light on this ? I really appreciate the help.
> > >
> > >
> > > thanks
> > > -- Surya
> > >
> >
>
--
-- Surya
Re: Solr Heap usage
Posted by "Arrieta, Alejandro" <aa...@perrinsoftware.com>.
Hello,
this link may help you, it has examples:
https://risdenk.github.io/2017/12/18/ambari-infra-solr-ranger.html
Kind Regards,
Alejandro Arrieta
On Tue, Oct 24, 2023 at 9:18 PM rajani m <ra...@gmail.com> wrote:
> When you see memory filling up on the visualvm gui, request a heap dump.
> The heap dump file gets stored on the node's /tmp location. Use the eclipse
> memory analyzer tool (MAT) to load the downloaded ".dmp" file, the MAT
> analyzes the heap and generates a report of objects on the heap.
>
> On Mon, Oct 23, 2023 at 1:48 PM Surya R <su...@gmail.com> wrote:
>
> > Hi. We have a solr index with around 2.5M documents , running solr 8.11 .
> > We have bunch of fields that we faceet/sort/highlight on. Heap is 8GB,
> out
> > of 12 GB RAM
> >
> > These fields were not defined with docValues=true due to some legacy
> design
> > issues , but if we were to rewrite the app, we would definitely have the
> > docvalues enabled for those fields that participate in
> > facet/sort/highlight.
> >
> > Problem:
> > I do see garbage collection ( G1GC) kicking in when the traffic is
> moderate
> > to high, with a "pause full" event that takes around 2seconds to
> complete.
> > This event happens more often when the traffic is high.
> >
> > I see this GC event happening when the humongous regions go beyond 1000.
> > The humongous region size is 4M
> >
> > I took a heap dump and analyzed with VisualVM, but I couldnt pinpoint
> what
> > objects are using the 'humougous region' .
> >
> > My question is, using the VisualVM, is it possible to pinpoint which /
> what
> > objects are using the 'humongous region' ?
> >
> > If for example I have an solr indexed field called "answerSnippet" , if
> > that is the culprit, Is it possible to see the field answerSnippet in the
> > VisualVM heap dump output?
> >
> > All i see is byte[] with 55% heap occupancy, but dont know what objects
> > constitute that byte[]
> >
> > Can anybody throw some light on this ? I really appreciate the help.
> >
> >
> > thanks
> > -- Surya
> >
>
Re: Solr Heap usage
Posted by rajani m <ra...@gmail.com>.
When you see memory filling up on the visualvm gui, request a heap dump.
The heap dump file gets stored on the node's /tmp location. Use the eclipse
memory analyzer tool (MAT) to load the downloaded ".dmp" file, the MAT
analyzes the heap and generates a report of objects on the heap.
On Mon, Oct 23, 2023 at 1:48 PM Surya R <su...@gmail.com> wrote:
> Hi. We have a solr index with around 2.5M documents , running solr 8.11 .
> We have bunch of fields that we faceet/sort/highlight on. Heap is 8GB, out
> of 12 GB RAM
>
> These fields were not defined with docValues=true due to some legacy design
> issues , but if we were to rewrite the app, we would definitely have the
> docvalues enabled for those fields that participate in
> facet/sort/highlight.
>
> Problem:
> I do see garbage collection ( G1GC) kicking in when the traffic is moderate
> to high, with a "pause full" event that takes around 2seconds to complete.
> This event happens more often when the traffic is high.
>
> I see this GC event happening when the humongous regions go beyond 1000.
> The humongous region size is 4M
>
> I took a heap dump and analyzed with VisualVM, but I couldnt pinpoint what
> objects are using the 'humougous region' .
>
> My question is, using the VisualVM, is it possible to pinpoint which / what
> objects are using the 'humongous region' ?
>
> If for example I have an solr indexed field called "answerSnippet" , if
> that is the culprit, Is it possible to see the field answerSnippet in the
> VisualVM heap dump output?
>
> All i see is byte[] with 55% heap occupancy, but dont know what objects
> constitute that byte[]
>
> Can anybody throw some light on this ? I really appreciate the help.
>
>
> thanks
> -- Surya
>