[jira] [Commented] (MNG-6836) Too Many Open Files for large projects

["Cody Wayne Holmes (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MNG-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006898#comment-17006898 ] 

Cody Wayne Holmes commented on MNG-6836:
----------------------------------------

Sure, some of the numbers for one project are: (Note: these are approximate numbers, but are fairly close to the real world example)

 

Number of modules: 50 (3 are packaging type pom)

Relative total build time using 9 build threads: 22 mins

Total Project dependencies: 918

Total number of plugins used: 31

Size of fresh local m2 after a build" ~1GB (4 - 6k files)

Ulimit set when number of files exceeded: 1024s/4096h (It is hitting the hard limit of 4096)

> Too Many Open Files for large projects
> --------------------------------------
>
>                 Key: MNG-6836
>                 URL: https://issues.apache.org/jira/browse/MNG-6836
>             Project: Maven
>          Issue Type: Bug
>          Components: Plugins and Lifecycle
>    Affects Versions: 3.5.2, 3.5.3, 3.5.4, 3.6.0, 3.6.1, 3.6.2, 3.6.3
>            Reporter: Cody Wayne Holmes
>            Priority: Major
>
> This issue seems to be that all plugins classloaders are created and then cached, and plugins are internally creating classloaders that are not being closed and therefore leaving open file handles to jar files waiting to be finalized.
>  
> When a project gets big enough with enough modules and plugins than this becomes a problem as the project will reach a large number of open jar files by the time it completes and if the file system is not set up to allow the standard ulimit to be exceeded, then this will cause build failures.
>  
> One plugin example that opens and does not close a URLClassloader is the [maven-pmd-plugin|#L796]]. As you can see it calls the method to prepend the classpath for the plugin to run. This method then [creates|#L196]] a URLClassloader that is not closed and must be finalized.
>  
> With the deprecation of the finalizer and the knowledge that it is not a good mechanism for closing system resources reliably, this also causes issues for larger projects with a lot of modules and a slightly larger build time.
>  
> I am not sure what can be done about this issue in the maven lifecycle, but just wanted to ask the question and find out if anyone else has seen similar issues or knows of a way to resolve this open file handles problem. And possibly find a way to limit the number of open file handles to the downloaded artifacts.
>  
> Thanks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)