You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Amrit Sarkar <sa...@gmail.com> on 2017/12/18 11:10:27 UTC
Identify Reference Leak in Custom Code related to Solr
Hi,
We incorporated *https://github.com/sematext/solr-researcher
<https://github.com/sematext/solr-researcher>* into our project and it is
responsible for memory leak / reference leak which is causing multiple
*SolrIndexSearcher
*objects in the heap dump.
37 instances of *"org.apache.solr.search.SolrIndexSearcher"*, loaded
by *"org.eclipse.jetty.webapp.WebAppClassLoader
@ 0x5e0020830"*occupy *744,482,384 (48.16%)* bytes.
Biggest instances:
- org.apache.solr.search.SolrIndexSearcher @ 0x5fcac64c0 - 108,168,104
(7.00%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x616b414b0 - 54,982,536
(3.56%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x60aaa5820 - 35,614,544
(2.30%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x5ed303418 - 26,742,472
(1.73%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x6c04d8948 - 26,413,728
(1.71%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x66d2f1ca8 - 26,230,600
(1.70%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x624904550 - 25,800,200
(1.67%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x6baa4c5f8 - 25,094,760
(1.62%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x676fefdd0 - 24,720,312
(1.60%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x6634d7a08 - 24,315,864
(1.57%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x652a82880 - 24,186,328
(1.56%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x6ad3ef080 - 24,078,800
(1.56%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x64bf747b0 - 24,073,736
(1.56%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x6a752cce0 - 23,937,584
(1.55%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x698fba4f8 - 23,339,000
(1.51%) bytes.
- org.apache.solr.search.SolrIndexSearcher @ 0x6a12724c0 - 23,066,512
(1.49%) bytes.
We would really appreciate if some can help us on how to pin-point:
1. *Reference leak* (since it is an independent third-party plugin).
This is taking almost 80% of the total heap memory allocated (16GB).
Looking forward to positive responses.
Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2
Medium: https://medium.com/@sarkaramrit2
Re: Identify Reference Leak in Custom Code related to Solr
Posted by Emir Arnautović <em...@sematext.com>.
Hi Amrit,
I suppose it is related to https://github.com/sematext/solr-researcher/issues/6 <https://github.com/sematext/solr-researcher/issues/6>
Can you please continue with conversation there, and one of us can update ML when issue is resolved in case there are others who could be interested in this component.
Thanks,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 18 Dec 2017, at 14:53, Amrit Sarkar <sa...@gmail.com> wrote:
>
> Emir,
>
> Solr version: 6.6, SolrCloud
>
> We followed the instructions on README.md on the github project.
>
> Amrit Sarkar
> Search Engineer
> Lucidworks, Inc.
> 415-589-9269
> www.lucidworks.com
> Twitter http://twitter.com/lucidworks
> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
> Medium: https://medium.com/@sarkaramrit2
>
> On Mon, Dec 18, 2017 at 5:13 PM, Emir Arnautović <
> emir.arnautovic@sematext.com> wrote:
>
>> Hi Amrit,
>> I’ll check with my colleague that worked on this. In the meantime, can you
>> provide more info about setup: Solr version, M-S or cloud and steps that we
>> can do to reproduce it.
>>
>> Thanks,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>>
>>> On 18 Dec 2017, at 12:10, Amrit Sarkar <sa...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> We incorporated *https://github.com/sematext/solr-researcher
>>> <https://github.com/sematext/solr-researcher>* into our project and it
>> is
>>> responsible for memory leak / reference leak which is causing multiple
>>> *SolrIndexSearcher
>>> *objects in the heap dump.
>>>
>>> 37 instances of *"org.apache.solr.search.SolrIndexSearcher"*, loaded
>>> by *"org.eclipse.jetty.webapp.WebAppClassLoader
>>> @ 0x5e0020830"*occupy *744,482,384 (48.16%)* bytes.
>>>
>>> Biggest instances:
>>>
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x5fcac64c0 - 108,168,104
>>> (7.00%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x616b414b0 - 54,982,536
>>> (3.56%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x60aaa5820 - 35,614,544
>>> (2.30%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x5ed303418 - 26,742,472
>>> (1.73%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x6c04d8948 - 26,413,728
>>> (1.71%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x66d2f1ca8 - 26,230,600
>>> (1.70%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x624904550 - 25,800,200
>>> (1.67%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x6baa4c5f8 - 25,094,760
>>> (1.62%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x676fefdd0 - 24,720,312
>>> (1.60%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x6634d7a08 - 24,315,864
>>> (1.57%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x652a82880 - 24,186,328
>>> (1.56%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x6ad3ef080 - 24,078,800
>>> (1.56%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x64bf747b0 - 24,073,736
>>> (1.56%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x6a752cce0 - 23,937,584
>>> (1.55%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x698fba4f8 - 23,339,000
>>> (1.51%) bytes.
>>> - org.apache.solr.search.SolrIndexSearcher @ 0x6a12724c0 - 23,066,512
>>> (1.49%) bytes.
>>>
>>>
>>> We would really appreciate if some can help us on how to pin-point:
>>>
>>> 1. *Reference leak* (since it is an independent third-party plugin).
>>>
>>> This is taking almost 80% of the total heap memory allocated (16GB).
>>> Looking forward to positive responses.
>>>
>>> Amrit Sarkar
>>> Search Engineer
>>> Lucidworks, Inc.
>>> 415-589-9269
>>> www.lucidworks.com
>>> Twitter http://twitter.com/lucidworks
>>> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>>> Medium: https://medium.com/@sarkaramrit2
>>
>>
Re: Identify Reference Leak in Custom Code related to Solr
Posted by Amrit Sarkar <sa...@gmail.com>.
Emir,
Solr version: 6.6, SolrCloud
We followed the instructions on README.md on the github project.
Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2
Medium: https://medium.com/@sarkaramrit2
On Mon, Dec 18, 2017 at 5:13 PM, Emir Arnautović <
emir.arnautovic@sematext.com> wrote:
> Hi Amrit,
> I’ll check with my colleague that worked on this. In the meantime, can you
> provide more info about setup: Solr version, M-S or cloud and steps that we
> can do to reproduce it.
>
> Thanks,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 18 Dec 2017, at 12:10, Amrit Sarkar <sa...@gmail.com> wrote:
> >
> > Hi,
> >
> > We incorporated *https://github.com/sematext/solr-researcher
> > <https://github.com/sematext/solr-researcher>* into our project and it
> is
> > responsible for memory leak / reference leak which is causing multiple
> > *SolrIndexSearcher
> > *objects in the heap dump.
> >
> > 37 instances of *"org.apache.solr.search.SolrIndexSearcher"*, loaded
> > by *"org.eclipse.jetty.webapp.WebAppClassLoader
> > @ 0x5e0020830"*occupy *744,482,384 (48.16%)* bytes.
> >
> > Biggest instances:
> >
> > - org.apache.solr.search.SolrIndexSearcher @ 0x5fcac64c0 - 108,168,104
> > (7.00%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x616b414b0 - 54,982,536
> > (3.56%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x60aaa5820 - 35,614,544
> > (2.30%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x5ed303418 - 26,742,472
> > (1.73%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x6c04d8948 - 26,413,728
> > (1.71%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x66d2f1ca8 - 26,230,600
> > (1.70%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x624904550 - 25,800,200
> > (1.67%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x6baa4c5f8 - 25,094,760
> > (1.62%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x676fefdd0 - 24,720,312
> > (1.60%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x6634d7a08 - 24,315,864
> > (1.57%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x652a82880 - 24,186,328
> > (1.56%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x6ad3ef080 - 24,078,800
> > (1.56%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x64bf747b0 - 24,073,736
> > (1.56%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x6a752cce0 - 23,937,584
> > (1.55%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x698fba4f8 - 23,339,000
> > (1.51%) bytes.
> > - org.apache.solr.search.SolrIndexSearcher @ 0x6a12724c0 - 23,066,512
> > (1.49%) bytes.
> >
> >
> > We would really appreciate if some can help us on how to pin-point:
> >
> > 1. *Reference leak* (since it is an independent third-party plugin).
> >
> > This is taking almost 80% of the total heap memory allocated (16GB).
> > Looking forward to positive responses.
> >
> > Amrit Sarkar
> > Search Engineer
> > Lucidworks, Inc.
> > 415-589-9269
> > www.lucidworks.com
> > Twitter http://twitter.com/lucidworks
> > LinkedIn: https://www.linkedin.com/in/sarkaramrit2
> > Medium: https://medium.com/@sarkaramrit2
>
>
Re: Identify Reference Leak in Custom Code related to Solr
Posted by Emir Arnautović <em...@sematext.com>.
Hi Amrit,
I’ll check with my colleague that worked on this. In the meantime, can you provide more info about setup: Solr version, M-S or cloud and steps that we can do to reproduce it.
Thanks,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 18 Dec 2017, at 12:10, Amrit Sarkar <sa...@gmail.com> wrote:
>
> Hi,
>
> We incorporated *https://github.com/sematext/solr-researcher
> <https://github.com/sematext/solr-researcher>* into our project and it is
> responsible for memory leak / reference leak which is causing multiple
> *SolrIndexSearcher
> *objects in the heap dump.
>
> 37 instances of *"org.apache.solr.search.SolrIndexSearcher"*, loaded
> by *"org.eclipse.jetty.webapp.WebAppClassLoader
> @ 0x5e0020830"*occupy *744,482,384 (48.16%)* bytes.
>
> Biggest instances:
>
> - org.apache.solr.search.SolrIndexSearcher @ 0x5fcac64c0 - 108,168,104
> (7.00%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x616b414b0 - 54,982,536
> (3.56%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x60aaa5820 - 35,614,544
> (2.30%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x5ed303418 - 26,742,472
> (1.73%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x6c04d8948 - 26,413,728
> (1.71%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x66d2f1ca8 - 26,230,600
> (1.70%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x624904550 - 25,800,200
> (1.67%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x6baa4c5f8 - 25,094,760
> (1.62%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x676fefdd0 - 24,720,312
> (1.60%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x6634d7a08 - 24,315,864
> (1.57%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x652a82880 - 24,186,328
> (1.56%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x6ad3ef080 - 24,078,800
> (1.56%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x64bf747b0 - 24,073,736
> (1.56%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x6a752cce0 - 23,937,584
> (1.55%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x698fba4f8 - 23,339,000
> (1.51%) bytes.
> - org.apache.solr.search.SolrIndexSearcher @ 0x6a12724c0 - 23,066,512
> (1.49%) bytes.
>
>
> We would really appreciate if some can help us on how to pin-point:
>
> 1. *Reference leak* (since it is an independent third-party plugin).
>
> This is taking almost 80% of the total heap memory allocated (16GB).
> Looking forward to positive responses.
>
> Amrit Sarkar
> Search Engineer
> Lucidworks, Inc.
> 415-589-9269
> www.lucidworks.com
> Twitter http://twitter.com/lucidworks
> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
> Medium: https://medium.com/@sarkaramrit2