[jira] [Comment Edited] (OAK-6500) NRTIndex leaks file handles due to unclosed IndexReader

["Chetan Mehrotra (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104499#comment-16104499 ] 

Chetan Mehrotra edited comment on OAK-6500 at 7/28/17 6:16 AM:
---------------------------------------------------------------

Added ignored test with 1803248. 

The test uses [UnixOperatingSystemMXBean#maxFileDescriptorCount|https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html#getMaxFileDescriptorCount()] to check how many file handles are open. Hence would only work on unix setups. 

The test configures nrt logic to use SimpleFSDirectory which avoids memory mapped file access and hence any open file gets reflected in file descriptor count

{noformat}
Open file count - Post creation at /content/a is 301
Open file count - Post async run at /content/a is 304
Open file count - Post creation at /content/b is 421
Open file count - Post async run at /content/b is 424
Open file count - Post creation at /content/c is 541
Open file count - Post async run at /content/c is 544
Open file count - Post creation at /content/d is 667
Open file count - Post async run at /content/d is 668
Open file count - Post creation at /content/e is 785
Open file count - Post async run at /content/e is 788
{noformat}

From the testcase the count can be seen increasing over time


was (Author: chetanm):
Added ignored test with 1803248. 

The test uses [UnixOperatingSystemMXBean#maxFileDescriptorCount|https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html#getMaxFileDescriptorCount()] to check how many file handles are open. Hence would only work on unix setups. 

The test configures nrt logic to use SimpleFSDirectory which avoids memory mapped file access and hence any open file gets reflected in file descriptor count

> NRTIndex leaks file handles due to unclosed IndexReader
> -------------------------------------------------------
>
>                 Key: OAK-6500
>                 URL: https://issues.apache.org/jira/browse/OAK-6500
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: lucene
>    Affects Versions: 1.6.0
>            Reporter: Chetan Mehrotra
>            Assignee: Chetan Mehrotra
>            Priority: Critical
>             Fix For: 1.8, 1.7.5, 1.6.4
>
>         Attachments: OAK-6500-v1.patch
>
>
> On some setups under stress it has been seen that NRTIndex leaks file handles over time. 
> Checking with lsof indicates that more than 3 nrt folders per index are being used. However per design there can be max 3 and after system is not in use max 1 should be present.
> {noformat}
> $ lsof -p 9550 | grep '/nrt' | gawk 'match($0, /.*crx-quickstart\/repository\/index\/(.*?)\/\_.*$/, m) { print m[1]; }' | sort | uniq
> cqPageLucene-1501065263331/nrt1501065335930
> cqPageLucene-1501065263331/nrt1501065374667
> cqPageLucene-1501065263331/nrt1501065392492
> cqPageLucene-1501065263331/nrt1501065440955
> cqPageLucene-1501065263331/nrt1501065473286
> cqPageLucene-1501065263331/nrt1501065507345
> slingeventJob-1501065263330/nrt1501065356975
> slingeventJob-1501065263330/nrt1501065373229
> slingeventJob-1501065263330/nrt1501065394142
> slingeventJob-1501065263330/nrt1501065440953
> slingeventJob-1501065263330/nrt1501065473282
> slingeventJob-1501065263330/nrt1501065507342
> versionStoreIndex-1501065263332/nrt1501065335925
> versionStoreIndex-1501065263332/nrt1501065366781
> versionStoreIndex-1501065263332/nrt1501065392490
> versionStoreIndex-1501065263332/nrt1501065441232
> versionStoreIndex-1501065263332/nrt1501065473285
> {noformat}
> Further actually checking index folder indicates that those folder are actually deleted. So some where the file handle is still referring them.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)