[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).

["Dawid Weiss (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013590#comment-14013590 ] 

Dawid Weiss commented on LUCENE-5716:
-------------------------------------

I spoke with Robert at Buzzwords about it a bit. It'd be interesting to capture file handle leaks from tests (unclosed sockets, unclosed files). There are essentially no java built-in mechanisms to do it. One could use the operating system's tools such as lsof, but this would measure ALL file handles, including handles of open jar files, etc. It's also an interesting puzzle to see if one can do it from Java in a portable (?) way.

There are a few options.

1) Override bootclasspath for a particular release of the JVM and modify system classes to allow resource tracking. A similar "solution" kind of exists already in FileInputStream:
{code}
    public int read() throws IOException {
        Object traceContext = IoTrace.fileReadBegin(path);
        int b = 0;
        try {
            b = read0();
        } finally {
            IoTrace.fileReadEnd(traceContext, b == -1 ? 0 : 1);
        }
        return b;
    }
{code}
the obscure IoTrace class unfortunately only tracks file writes, not its open/close status. A similar hack could be applied to capture constructors and close calls though.

2) Use bytecode transformation and capture all places where interesting objects are created/ closed. AspectJ is closest to my heart since it allows fast prototyping and is essentially just a bytecode manipulation framework. I've written a simple aspect tracing FileInputStream usage, here's the code.

https://github.com/dweiss/fhandle-tracing

The aspect itself:
https://github.com/dweiss/fhandle-tracing/blob/master/src/main/aspect/com/carrotsearch/aspects/FileInputStreamTracker.aj

There is a JUnit test in there and if you run mvn test, you can see that it's actually working quite nice. Not everything can be easily addressed (for example, it's difficult to advise close in classes that inherit from FIS but don't override this method), but alternative solutions to the problem also exist (capture all close calls, capture all constructors of FileInputStream+.new). Doable and should work for 99% of the use cases I think.

3) use jvmti instrumentation agents to essentially provide the same instrumentation as above. I don't think it is functionally any different from (2) and I like aspect's syntax for fast hacking. The only exception might be (didn't check) if we could redefine and reload the definitions of core Java classes (FileInputStream, for example) so that we can instrument constructors and methods of these base classes directly. This would be quite nice because then we'd be able to make it portable across all JVMs and it should work for all code fragments, including standard library itself.

I'll keep experimenting with (2) for now as it's the low-hanging fruit (already works), but if somebody wants to inspect (3) it'd be quite interesting.


> Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
> -------------------------------------------------------------------------
>
>                 Key: LUCENE-5716
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5716
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org