You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Hendrik Haddorp <he...@gmx.net> on 2017/08/28 15:42:03 UTC
Solr memory leak
Hi,
we noticed that triggering collection reloads on many collections has a
good chance to result in an OOM-Error. To investigate that further I did
a simple test:
- Start solr with a 2GB heap and 1GB Metaspace
- create a trivial collection with a few documents (I used only 2
fields and 100 documents)
- trigger a collection reload in a loop (I used SolrJ for this)
Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
worked better but also failed after 1100 loops.
When looking at the memory usage on the Solr dashboard it looks like the
space left after GC cycles gets less and less. Then Solr gets very slow,
as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
my last run this was actually for the Metaspace. So it looks like more
and more heap and metaspace is being used by just constantly reloading a
trivial collection.
regards,
Hendrik
RE: Solr memory leak
Posted by Markus Jelsma <ma...@openindex.io>.
See https://issues.apache.org/jira/browse/SOLR-10506
Fixed for 7.0
Markus
-----Original message-----
> From:Hendrik Haddorp <he...@gmx.net>
> Sent: Monday 28th August 2017 17:42
> To: solr-user@lucene.apache.org
> Subject: Solr memory leak
>
> Hi,
>
> we noticed that triggering collection reloads on many collections has a
> good chance to result in an OOM-Error. To investigate that further I did
> a simple test:
> - Start solr with a 2GB heap and 1GB Metaspace
> - create a trivial collection with a few documents (I used only 2
> fields and 100 documents)
> - trigger a collection reload in a loop (I used SolrJ for this)
>
> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
> worked better but also failed after 1100 loops.
>
> When looking at the memory usage on the Solr dashboard it looks like the
> space left after GC cycles gets less and less. Then Solr gets very slow,
> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
> my last run this was actually for the Metaspace. So it looks like more
> and more heap and metaspace is being used by just constantly reloading a
> trivial collection.
>
> regards,
> Hendrik
>
Re: Solr memory leak
Posted by Hendrik Haddorp <he...@gmx.net>.
I didn't meant to say that the fix is not in 7.0. I just stated that I
do not see it listed in the release notes
(https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310230&version=12335718).
Thanks for explaining the release process.
regards,
Hendrik
On 10.09.2017 17:32, Erick Erickson wrote:
> There will be no 6.7. Once the X+1 version is released, all past fixes
> are applied to as minor releases to the last released version of the
> previous major release. So now that 7.0 has been cut, there might be a
> 6.6.2 (6.6.1 was just released) but no 6.7. Current un-released JIRAs
> are parked on the 6.x (as opposed to branch_6_6) for convenience. If
> anyone steps up to release 6.6.2, they can include ths.
>
> Why do you say this isn't in 7.0? The "Fix Versions" clearly states
> so, as does CHANGES.txt for 7.0. The new file is is in the 7.0 branch.
>
>
> If you need it in 6x you have a couple of options:
>
> 1> agitate fo ra 6.6.2 with this included
> 2> apply the patch yourself and compile it locally
>
> Best,
> Erick
>
> On Sun, Sep 10, 2017 at 6:04 AM, Hendrik Haddorp
> <he...@gmx.net> wrote:
>> Hi,
>>
>> looks like SOLR-10506 didn't make it into 6.6.1. I do however also not see
>> it listen in the current release notes for 6.7 nor 7.0:
>> https://issues.apache.org/jira/projects/SOLR/versions/12340568
>> https://issues.apache.org/jira/projects/SOLR/versions/12335718
>>
>> Is there any any rough idea already when 6.7 or 7.0 will be released?
>>
>> thanks,
>> Hendrik
>>
>>
>> On 28.08.2017 18:31, Erick Erickson wrote:
>>> Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including
>>> it.
>>>
>>> On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <wu...@wunderwood.org>
>>> wrote:
>>>> That would be a really good reason for a 6.7.
>>>>
>>>> wunder
>>>> Walter Underwood
>>>> wunder@wunderwood.org
>>>> http://observer.wunderwood.org/ (my blog)
>>>>
>>>>
>>>>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <ma...@openindex.io>
>>>>> wrote:
>>>>>
>>>>> It is, unfortunately, not committed for 6.7.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original message-----
>>>>>> From:Markus Jelsma <ma...@openindex.io>
>>>>>> Sent: Monday 28th August 2017 17:46
>>>>>> To: solr-user@lucene.apache.org
>>>>>> Subject: RE: Solr memory leak
>>>>>>
>>>>>> See https://issues.apache.org/jira/browse/SOLR-10506
>>>>>> Fixed for 7.0
>>>>>>
>>>>>> Markus
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original message-----
>>>>>>> From:Hendrik Haddorp <he...@gmx.net>
>>>>>>> Sent: Monday 28th August 2017 17:42
>>>>>>> To: solr-user@lucene.apache.org
>>>>>>> Subject: Solr memory leak
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> we noticed that triggering collection reloads on many collections has
>>>>>>> a
>>>>>>> good chance to result in an OOM-Error. To investigate that further I
>>>>>>> did
>>>>>>> a simple test:
>>>>>>> - Start solr with a 2GB heap and 1GB Metaspace
>>>>>>> - create a trivial collection with a few documents (I used only 2
>>>>>>> fields and 100 documents)
>>>>>>> - trigger a collection reload in a loop (I used SolrJ for this)
>>>>>>>
>>>>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr
>>>>>>> 6.6
>>>>>>> worked better but also failed after 1100 loops.
>>>>>>>
>>>>>>> When looking at the memory usage on the Solr dashboard it looks like
>>>>>>> the
>>>>>>> space left after GC cycles gets less and less. Then Solr gets very
>>>>>>> slow,
>>>>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>>>>>> my last run this was actually for the Metaspace. So it looks like more
>>>>>>> and more heap and metaspace is being used by just constantly reloading
>>>>>>> a
>>>>>>> trivial collection.
>>>>>>>
>>>>>>> regards,
>>>>>>> Hendrik
>>>>>>>
Re: Solr memory leak
Posted by Erick Erickson <er...@gmail.com>.
There will be no 6.7. Once the X+1 version is released, all past fixes
are applied to as minor releases to the last released version of the
previous major release. So now that 7.0 has been cut, there might be a
6.6.2 (6.6.1 was just released) but no 6.7. Current un-released JIRAs
are parked on the 6.x (as opposed to branch_6_6) for convenience. If
anyone steps up to release 6.6.2, they can include ths.
Why do you say this isn't in 7.0? The "Fix Versions" clearly states
so, as does CHANGES.txt for 7.0. The new file is is in the 7.0 branch.
If you need it in 6x you have a couple of options:
1> agitate fo ra 6.6.2 with this included
2> apply the patch yourself and compile it locally
Best,
Erick
On Sun, Sep 10, 2017 at 6:04 AM, Hendrik Haddorp
<he...@gmx.net> wrote:
> Hi,
>
> looks like SOLR-10506 didn't make it into 6.6.1. I do however also not see
> it listen in the current release notes for 6.7 nor 7.0:
> https://issues.apache.org/jira/projects/SOLR/versions/12340568
> https://issues.apache.org/jira/projects/SOLR/versions/12335718
>
> Is there any any rough idea already when 6.7 or 7.0 will be released?
>
> thanks,
> Hendrik
>
>
> On 28.08.2017 18:31, Erick Erickson wrote:
>>
>> Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including
>> it.
>>
>> On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <wu...@wunderwood.org>
>> wrote:
>>>
>>> That would be a really good reason for a 6.7.
>>>
>>> wunder
>>> Walter Underwood
>>> wunder@wunderwood.org
>>> http://observer.wunderwood.org/ (my blog)
>>>
>>>
>>>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <ma...@openindex.io>
>>>> wrote:
>>>>
>>>> It is, unfortunately, not committed for 6.7.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Original message-----
>>>>>
>>>>> From:Markus Jelsma <ma...@openindex.io>
>>>>> Sent: Monday 28th August 2017 17:46
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: RE: Solr memory leak
>>>>>
>>>>> See https://issues.apache.org/jira/browse/SOLR-10506
>>>>> Fixed for 7.0
>>>>>
>>>>> Markus
>>>>>
>>>>>
>>>>>
>>>>> -----Original message-----
>>>>>>
>>>>>> From:Hendrik Haddorp <he...@gmx.net>
>>>>>> Sent: Monday 28th August 2017 17:42
>>>>>> To: solr-user@lucene.apache.org
>>>>>> Subject: Solr memory leak
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> we noticed that triggering collection reloads on many collections has
>>>>>> a
>>>>>> good chance to result in an OOM-Error. To investigate that further I
>>>>>> did
>>>>>> a simple test:
>>>>>> - Start solr with a 2GB heap and 1GB Metaspace
>>>>>> - create a trivial collection with a few documents (I used only 2
>>>>>> fields and 100 documents)
>>>>>> - trigger a collection reload in a loop (I used SolrJ for this)
>>>>>>
>>>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr
>>>>>> 6.6
>>>>>> worked better but also failed after 1100 loops.
>>>>>>
>>>>>> When looking at the memory usage on the Solr dashboard it looks like
>>>>>> the
>>>>>> space left after GC cycles gets less and less. Then Solr gets very
>>>>>> slow,
>>>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>>>>> my last run this was actually for the Metaspace. So it looks like more
>>>>>> and more heap and metaspace is being used by just constantly reloading
>>>>>> a
>>>>>> trivial collection.
>>>>>>
>>>>>> regards,
>>>>>> Hendrik
>>>>>>
>
Re: Solr memory leak
Posted by Hendrik Haddorp <he...@gmx.net>.
Hi,
looks like SOLR-10506 didn't make it into 6.6.1. I do however also not
see it listen in the current release notes for 6.7 nor 7.0:
https://issues.apache.org/jira/projects/SOLR/versions/12340568
https://issues.apache.org/jira/projects/SOLR/versions/12335718
Is there any any rough idea already when 6.7 or 7.0 will be released?
thanks,
Hendrik
On 28.08.2017 18:31, Erick Erickson wrote:
> Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including it.
>
> On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <wu...@wunderwood.org> wrote:
>> That would be a really good reason for a 6.7.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>
>>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <ma...@openindex.io> wrote:
>>>
>>> It is, unfortunately, not committed for 6.7.
>>>
>>>
>>>
>>>
>>>
>>> -----Original message-----
>>>> From:Markus Jelsma <ma...@openindex.io>
>>>> Sent: Monday 28th August 2017 17:46
>>>> To: solr-user@lucene.apache.org
>>>> Subject: RE: Solr memory leak
>>>>
>>>> See https://issues.apache.org/jira/browse/SOLR-10506
>>>> Fixed for 7.0
>>>>
>>>> Markus
>>>>
>>>>
>>>>
>>>> -----Original message-----
>>>>> From:Hendrik Haddorp <he...@gmx.net>
>>>>> Sent: Monday 28th August 2017 17:42
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: Solr memory leak
>>>>>
>>>>> Hi,
>>>>>
>>>>> we noticed that triggering collection reloads on many collections has a
>>>>> good chance to result in an OOM-Error. To investigate that further I did
>>>>> a simple test:
>>>>> - Start solr with a 2GB heap and 1GB Metaspace
>>>>> - create a trivial collection with a few documents (I used only 2
>>>>> fields and 100 documents)
>>>>> - trigger a collection reload in a loop (I used SolrJ for this)
>>>>>
>>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
>>>>> worked better but also failed after 1100 loops.
>>>>>
>>>>> When looking at the memory usage on the Solr dashboard it looks like the
>>>>> space left after GC cycles gets less and less. Then Solr gets very slow,
>>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>>>> my last run this was actually for the Metaspace. So it looks like more
>>>>> and more heap and metaspace is being used by just constantly reloading a
>>>>> trivial collection.
>>>>>
>>>>> regards,
>>>>> Hendrik
>>>>>
Re: Solr memory leak
Posted by Hendrik Haddorp <he...@gmx.net>.
Did you get an answer? Would really be nice to have that in the next
release.
On 28.08.2017 18:31, Erick Erickson wrote:
> Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including it.
>
> On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <wu...@wunderwood.org> wrote:
>> That would be a really good reason for a 6.7.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>
>>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <ma...@openindex.io> wrote:
>>>
>>> It is, unfortunately, not committed for 6.7.
>>>
>>>
>>>
>>>
>>>
>>> -----Original message-----
>>>> From:Markus Jelsma <ma...@openindex.io>
>>>> Sent: Monday 28th August 2017 17:46
>>>> To: solr-user@lucene.apache.org
>>>> Subject: RE: Solr memory leak
>>>>
>>>> See https://issues.apache.org/jira/browse/SOLR-10506
>>>> Fixed for 7.0
>>>>
>>>> Markus
>>>>
>>>>
>>>>
>>>> -----Original message-----
>>>>> From:Hendrik Haddorp <he...@gmx.net>
>>>>> Sent: Monday 28th August 2017 17:42
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: Solr memory leak
>>>>>
>>>>> Hi,
>>>>>
>>>>> we noticed that triggering collection reloads on many collections has a
>>>>> good chance to result in an OOM-Error. To investigate that further I did
>>>>> a simple test:
>>>>> - Start solr with a 2GB heap and 1GB Metaspace
>>>>> - create a trivial collection with a few documents (I used only 2
>>>>> fields and 100 documents)
>>>>> - trigger a collection reload in a loop (I used SolrJ for this)
>>>>>
>>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
>>>>> worked better but also failed after 1100 loops.
>>>>>
>>>>> When looking at the memory usage on the Solr dashboard it looks like the
>>>>> space left after GC cycles gets less and less. Then Solr gets very slow,
>>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>>>> my last run this was actually for the Metaspace. So it looks like more
>>>>> and more heap and metaspace is being used by just constantly reloading a
>>>>> trivial collection.
>>>>>
>>>>> regards,
>>>>> Hendrik
>>>>>
Re: Solr memory leak
Posted by Erick Erickson <er...@gmail.com>.
Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including it.
On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <wu...@wunderwood.org> wrote:
> That would be a really good reason for a 6.7.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>
>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <ma...@openindex.io> wrote:
>>
>> It is, unfortunately, not committed for 6.7.
>>
>>
>>
>>
>>
>> -----Original message-----
>>> From:Markus Jelsma <ma...@openindex.io>
>>> Sent: Monday 28th August 2017 17:46
>>> To: solr-user@lucene.apache.org
>>> Subject: RE: Solr memory leak
>>>
>>> See https://issues.apache.org/jira/browse/SOLR-10506
>>> Fixed for 7.0
>>>
>>> Markus
>>>
>>>
>>>
>>> -----Original message-----
>>>> From:Hendrik Haddorp <he...@gmx.net>
>>>> Sent: Monday 28th August 2017 17:42
>>>> To: solr-user@lucene.apache.org
>>>> Subject: Solr memory leak
>>>>
>>>> Hi,
>>>>
>>>> we noticed that triggering collection reloads on many collections has a
>>>> good chance to result in an OOM-Error. To investigate that further I did
>>>> a simple test:
>>>> - Start solr with a 2GB heap and 1GB Metaspace
>>>> - create a trivial collection with a few documents (I used only 2
>>>> fields and 100 documents)
>>>> - trigger a collection reload in a loop (I used SolrJ for this)
>>>>
>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
>>>> worked better but also failed after 1100 loops.
>>>>
>>>> When looking at the memory usage on the Solr dashboard it looks like the
>>>> space left after GC cycles gets less and less. Then Solr gets very slow,
>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>>> my last run this was actually for the Metaspace. So it looks like more
>>>> and more heap and metaspace is being used by just constantly reloading a
>>>> trivial collection.
>>>>
>>>> regards,
>>>> Hendrik
>>>>
>>>
>
Re: Solr memory leak
Posted by Walter Underwood <wu...@wunderwood.org>.
That would be a really good reason for a 6.7.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <ma...@openindex.io> wrote:
>
> It is, unfortunately, not committed for 6.7.
>
>
>
>
>
> -----Original message-----
>> From:Markus Jelsma <ma...@openindex.io>
>> Sent: Monday 28th August 2017 17:46
>> To: solr-user@lucene.apache.org
>> Subject: RE: Solr memory leak
>>
>> See https://issues.apache.org/jira/browse/SOLR-10506
>> Fixed for 7.0
>>
>> Markus
>>
>>
>>
>> -----Original message-----
>>> From:Hendrik Haddorp <he...@gmx.net>
>>> Sent: Monday 28th August 2017 17:42
>>> To: solr-user@lucene.apache.org
>>> Subject: Solr memory leak
>>>
>>> Hi,
>>>
>>> we noticed that triggering collection reloads on many collections has a
>>> good chance to result in an OOM-Error. To investigate that further I did
>>> a simple test:
>>> - Start solr with a 2GB heap and 1GB Metaspace
>>> - create a trivial collection with a few documents (I used only 2
>>> fields and 100 documents)
>>> - trigger a collection reload in a loop (I used SolrJ for this)
>>>
>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
>>> worked better but also failed after 1100 loops.
>>>
>>> When looking at the memory usage on the Solr dashboard it looks like the
>>> space left after GC cycles gets less and less. Then Solr gets very slow,
>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>> my last run this was actually for the Metaspace. So it looks like more
>>> and more heap and metaspace is being used by just constantly reloading a
>>> trivial collection.
>>>
>>> regards,
>>> Hendrik
>>>
>>
RE: Solr memory leak
Posted by Markus Jelsma <ma...@openindex.io>.
It is, unfortunately, not committed for 6.7.
-----Original message-----
> From:Markus Jelsma <ma...@openindex.io>
> Sent: Monday 28th August 2017 17:46
> To: solr-user@lucene.apache.org
> Subject: RE: Solr memory leak
>
> See https://issues.apache.org/jira/browse/SOLR-10506
> Fixed for 7.0
>
> Markus
>
>
>
> -----Original message-----
> > From:Hendrik Haddorp <he...@gmx.net>
> > Sent: Monday 28th August 2017 17:42
> > To: solr-user@lucene.apache.org
> > Subject: Solr memory leak
> >
> > Hi,
> >
> > we noticed that triggering collection reloads on many collections has a
> > good chance to result in an OOM-Error. To investigate that further I did
> > a simple test:
> > - Start solr with a 2GB heap and 1GB Metaspace
> > - create a trivial collection with a few documents (I used only 2
> > fields and 100 documents)
> > - trigger a collection reload in a loop (I used SolrJ for this)
> >
> > Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
> > worked better but also failed after 1100 loops.
> >
> > When looking at the memory usage on the Solr dashboard it looks like the
> > space left after GC cycles gets less and less. Then Solr gets very slow,
> > as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
> > my last run this was actually for the Metaspace. So it looks like more
> > and more heap and metaspace is being used by just constantly reloading a
> > trivial collection.
> >
> > regards,
> > Hendrik
> >
>