Possible memory leak in recovery

[Markus Jelsma <ma...@openindex.io>]

Hi,

We wiped clean the data directories for one node. That node is never able to recover and regularly runs OOM. On another cluster (with an older build, september 10th) memory consumption on recovery is fairly low when recoverign and with only a 250MB heap allocated it's easy to recover two 4GB cores from scratch at the same time. On this new test cluster we see the following happening: 
- no index, start recovery
- recovery fails (see other thread, cannot read past EOF when reading index files)
- heap is not released
- recovery is retried, fails
- heap is not released
.. OOM

The distinct saw tooth pattern is not there, heap consumption only grows with siginifant steps when recovery is retried but fails. If i increase heap recovery simply fails a few more times.

I cannot find an existing issue but may have overlooked it. File bug or did i miss an Jira issue?

Thanks,
Markus

Re: Possible memory leak in recovery

[Mark Miller <ma...@gmail.com>]

Nothing I know of - file a bug please. Might be related to the EOF issue, so you might add the details to that JIRA.

- Mark 

On Nov 2, 2012, at 10:13 AM, Markus Jelsma <ma...@openindex.io> wrote:

> Hi,
> 
> We wiped clean the data directories for one node. That node is never able to recover and regularly runs OOM. On another cluster (with an older build, september 10th) memory consumption on recovery is fairly low when recoverign and with only a 250MB heap allocated it's easy to recover two 4GB cores from scratch at the same time. On this new test cluster we see the following happening: 
> - no index, start recovery
> - recovery fails (see other thread, cannot read past EOF when reading index files)
> - heap is not released
> - recovery is retried, fails
> - heap is not released
> .. OOM
> 
> The distinct saw tooth pattern is not there, heap consumption only grows with siginifant steps when recovery is retried but fails. If i increase heap recovery simply fails a few more times.
> 
> I cannot find an existing issue but may have overlooked it. File bug or did i miss an Jira issue?
> 
> Thanks,
> Markus
>