[jira] Commented: (MRM-1097) Error 500 "too many open files"

["Maria Odea Ching (JIRA)" <ji...@codehaus.org>]

    [ http://jira.codehaus.org/browse/MRM-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=242082#action_242082 ] 

Maria Odea Ching commented on MRM-1097:
---------------------------------------

After reviewing the code for the repo scanning and indexing, these are the different scenarios where an indexing task may be executed:
* When a repository scan is triggered, the {{beginScan(..)}} method of all the repository consumers is called. The indexing context is created at the {{beginScan(..)}} method of the indexer-consumer, where the index reader/writer is opened. For each artifact being processed by the indexer-consumer, an indexing task is created and is queued in the {{ArchivaIndexingTaskExecutor}}, which calls the Nexus indexer engine and indexes the artifact. Once the repository scan is finished, the {{completeScan(..)}} method of all the repository consumers is called. The indexing context created at the start by the indexer-consumer will then be queued as an indexing task in the {{ArchivaIndexingTaskExecutor}}, but this time the task is for closing the context.
* For each artifact deployment and for each artifact fetched from the proxies, the {{RepositoryContentConsumers.executeConsumers(..)}} method is called. This method does the same thing as with the repository scanner where it calls the {{beginScan(..)}} method of all repo consumers then processes the deployed/proxied artifact and then calls the {{completeScan(..)}} method. This means that an indexing context is created for every deployed/proxied artifact, which in turn results to an open index reader/writer while the indexing task for the artifact is still queued.
* It's also possible that a repo scanning may have already finished, but the indexing tasks for the artifacts in the repo are still queued. This means that the indexing context is still open because it would only be closed once all the indexing tasks for the artifacts in the repo have finished executing. Since the repo scanning has finished, a user would be able to trigger another repository scan or the repo scanning schedule would fire and execute again, thereby creating and opening another indexing context.

If these three scenarios happen at the same time, especially when the repository being scanned is a large one, the "Too many open files" is bound to happen.


> Error 500 "too many open files"
> -------------------------------
>
>                 Key: MRM-1097
>                 URL: http://jira.codehaus.org/browse/MRM-1097
>             Project: Archiva
>          Issue Type: Bug
>          Components: system
>    Affects Versions: 1.1.3
>         Environment: Linux RedHat, Tomcat 6 + Native Connector
>            Reporter: Marc Lustig
>             Fix For: 1.4
>
>         Attachments: MRM-1097.patch
>
>
> In our start-script we have configured "ulimit -n 10000"
> But still, occassionally we get http error 500 "java.io.FileNotFoundException ... (Too many open files)" on a Maven-artifact-request.
> I suppose the files are not properly closed after they are delivered.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira