You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Ben reeves (JIRA)" <ji...@apache.org> on 2006/03/11 16:02:02 UTC

[jira] Created: (SOLR-4) java.io.FileNotFoundException: (Too many open files)

java.io.FileNotFoundException: (Too many open files)
----------------------------------------------------

         Key: SOLR-4
         URL: http://issues.apache.org/jira/browse/SOLR-4
     Project: Solr
        Type: Bug
  Components: update  
 Environment: linux - ubuntu dapper latest jre 1.
    Reporter: Ben reeves
    Priority: Blocker


After adding a few thousand files I get the following error, when adding or deleting:

<result status="1">java.io.FileNotFoundException: /home/me/Desktop/www/ftpsearch/admin/lucene/database/file_index/data/index/_5o.tii (Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.&lt;init>(Unknown Source)
        at org.apache.lucene.store.FSIndexInput$Descriptor.&lt;init>(FSDirectory.java:425)
        at org.apache.lucene.store.FSIndexInput.&lt;init>(FSDirectory.java:434)
        at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:324)
        at org.apache.lucene.index.TermInfosReader.&lt;init>(TermInfosReader.java:52)
        at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:147)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:129)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:110)
        at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:154)
        at org.apache.lucene.store.Lock$With.run(Lock.java:109)
        at org.apache.lucene.index.IndexReader.open(IndexReader.java:143)
        at org.apache.lucene.index.IndexReader.open(IndexReader.java:127)
        at org.apache.solr.search.SolrIndexSearcher.&lt;init>(Unknown Source)
        at org.apache.solr.core.SolrCore.newSearcher(Unknown Source)
        at org.apache.solr.update.DirectUpdateHandler2.openSearcher(Unknown Source)
        at org.apache.solr.update.DirectUpdateHandler2.doDeletions(Unknown Source)
        at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(Unknown Source)
        at org.apache.solr.core.SolrCore.update(Unknown Source)
        at org.apache.solr.servlet.SolrServlet.doPost(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:767)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:408)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:350)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:195)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:164)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:536)
        at org.mortbay.jetty.Server.handle(Server.java:309)
        at org.mortbay.jetty.Server.handle(Server.java:285)
        at org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:363)
        at org.mortbay.jetty.HttpConnection.access$1600(HttpConnection.java:45)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:625)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:613)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:195)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:297)
        at org.mortbay.jetty.nio.SelectChannelConnector$HttpEndPoint.run(SelectChannelConnector.java:680)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:412)

I have rebuild evrything from scratch several times, but I always get the error. 


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

[jira] Closed: (SOLR-4) java.io.FileNotFoundException: (Too many open files)

Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/SOLR-4?page=all ]
     
Yonik Seeley closed SOLR-4:
---------------------------

    Resolution: Invalid

closing this, since it really isn't a bug.

> java.io.FileNotFoundException: (Too many open files)
> ----------------------------------------------------
>
>          Key: SOLR-4
>          URL: http://issues.apache.org/jira/browse/SOLR-4
>      Project: Solr
>         Type: Bug
>   Components: update
>  Environment: linux - ubuntu dapper latest jre 1.
>     Reporter: Ben reeves
>     Priority: Blocker

>
> After adding a few thousand files I get the following error, when adding or deleting:
> <result status="1">java.io.FileNotFoundException: /home/me/Desktop/www/ftpsearch/admin/lucene/database/file_index/data/index/_5o.tii (Too many open files)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.&lt;init>(Unknown Source)
>         at org.apache.lucene.store.FSIndexInput$Descriptor.&lt;init>(FSDirectory.java:425)
>         at org.apache.lucene.store.FSIndexInput.&lt;init>(FSDirectory.java:434)
>         at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:324)
>         at org.apache.lucene.index.TermInfosReader.&lt;init>(TermInfosReader.java:52)
>         at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:147)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:129)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:110)
>         at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:154)
>         at org.apache.lucene.store.Lock$With.run(Lock.java:109)
>         at org.apache.lucene.index.IndexReader.open(IndexReader.java:143)
>         at org.apache.lucene.index.IndexReader.open(IndexReader.java:127)
>         at org.apache.solr.search.SolrIndexSearcher.&lt;init>(Unknown Source)
>         at org.apache.solr.core.SolrCore.newSearcher(Unknown Source)
>         at org.apache.solr.update.DirectUpdateHandler2.openSearcher(Unknown Source)
>         at org.apache.solr.update.DirectUpdateHandler2.doDeletions(Unknown Source)
>         at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(Unknown Source)
>         at org.apache.solr.core.SolrCore.update(Unknown Source)
>         at org.apache.solr.servlet.SolrServlet.doPost(Unknown Source)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:767)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:408)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:350)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:195)
>         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:164)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:536)
>         at org.mortbay.jetty.Server.handle(Server.java:309)
>         at org.mortbay.jetty.Server.handle(Server.java:285)
>         at org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:363)
>         at org.mortbay.jetty.HttpConnection.access$1600(HttpConnection.java:45)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:625)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:613)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:195)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:297)
>         at org.mortbay.jetty.nio.SelectChannelConnector$HttpEndPoint.run(SelectChannelConnector.java:680)
>         at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:412)
> I have rebuild evrything from scratch several times, but I always get the error. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

[jira] Commented: (SOLR-4) java.io.FileNotFoundException: (Too many open files)

Posted by "Ben reeves (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/SOLR-4?page=comments#action_12369992 ] 

Ben reeves commented on SOLR-4:
-------------------------------

Ok i found the olution, it is not really a solr bug.

If you are using linux you need to run:
#su root
#ulmit -n 1000000

That should fix it. Is there a way to revent solr from having to open so many files at once? I think that would be a better solution

> java.io.FileNotFoundException: (Too many open files)
> ----------------------------------------------------
>
>          Key: SOLR-4
>          URL: http://issues.apache.org/jira/browse/SOLR-4
>      Project: Solr
>         Type: Bug
>   Components: update
>  Environment: linux - ubuntu dapper latest jre 1.
>     Reporter: Ben reeves
>     Priority: Blocker

>
> After adding a few thousand files I get the following error, when adding or deleting:
> <result status="1">java.io.FileNotFoundException: /home/me/Desktop/www/ftpsearch/admin/lucene/database/file_index/data/index/_5o.tii (Too many open files)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.&lt;init>(Unknown Source)
>         at org.apache.lucene.store.FSIndexInput$Descriptor.&lt;init>(FSDirectory.java:425)
>         at org.apache.lucene.store.FSIndexInput.&lt;init>(FSDirectory.java:434)
>         at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:324)
>         at org.apache.lucene.index.TermInfosReader.&lt;init>(TermInfosReader.java:52)
>         at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:147)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:129)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:110)
>         at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:154)
>         at org.apache.lucene.store.Lock$With.run(Lock.java:109)
>         at org.apache.lucene.index.IndexReader.open(IndexReader.java:143)
>         at org.apache.lucene.index.IndexReader.open(IndexReader.java:127)
>         at org.apache.solr.search.SolrIndexSearcher.&lt;init>(Unknown Source)
>         at org.apache.solr.core.SolrCore.newSearcher(Unknown Source)
>         at org.apache.solr.update.DirectUpdateHandler2.openSearcher(Unknown Source)
>         at org.apache.solr.update.DirectUpdateHandler2.doDeletions(Unknown Source)
>         at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(Unknown Source)
>         at org.apache.solr.core.SolrCore.update(Unknown Source)
>         at org.apache.solr.servlet.SolrServlet.doPost(Unknown Source)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:767)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:408)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:350)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:195)
>         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:164)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:536)
>         at org.mortbay.jetty.Server.handle(Server.java:309)
>         at org.mortbay.jetty.Server.handle(Server.java:285)
>         at org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:363)
>         at org.mortbay.jetty.HttpConnection.access$1600(HttpConnection.java:45)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:625)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:613)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:195)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:297)
>         at org.mortbay.jetty.nio.SelectChannelConnector$HttpEndPoint.run(SelectChannelConnector.java:680)
>         at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:412)
> I have rebuild evrything from scratch several times, but I always get the error. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

[jira] Commented: (SOLR-4) java.io.FileNotFoundException: (Too many open files)

Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/SOLR-4?page=comments#action_12370001 ] 

Yonik Seeley commented on SOLR-4:
---------------------------------

> That should fix it. Is there a way to revent solr from having to open so many files at once? I think that would be a better solution

Yes, there are many ways to control this.
Each index contains multiple segments, and each segment contains multiple parts, each stored as a separate file.
Details are here:http://lucene.apache.org/java/docs/fileformats.html

You can set useCompoundFile to true in solrconfig.xml to use Lucene's compound file format 
(fragment from the example solrconfig.xml below):

<config>
  <mainIndex>
    <!-- options specific to the main on-disk lucene index -->
    <useCompoundFile>false</useCompoundFile>
    <mergeFactor>10</mergeFactor>

The compound file format stuffs all of the segment parts into a single file.  It's not as efficient as the non-compound format though.

Another setting you might want to look at is mergeFactor (also directly from Lucene).
http://www.onjava.com/pub/a/onjava/2003/03/05/lucene.html?page=1

A lower mergeFactor (such as 2) will slow down indexing speed, but speed up searchers.  It will also result in fewer segments and hence fewer open files.






> java.io.FileNotFoundException: (Too many open files)
> ----------------------------------------------------
>
>          Key: SOLR-4
>          URL: http://issues.apache.org/jira/browse/SOLR-4
>      Project: Solr
>         Type: Bug
>   Components: update
>  Environment: linux - ubuntu dapper latest jre 1.
>     Reporter: Ben reeves
>     Priority: Blocker

>
> After adding a few thousand files I get the following error, when adding or deleting:
> <result status="1">java.io.FileNotFoundException: /home/me/Desktop/www/ftpsearch/admin/lucene/database/file_index/data/index/_5o.tii (Too many open files)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.&lt;init>(Unknown Source)
>         at org.apache.lucene.store.FSIndexInput$Descriptor.&lt;init>(FSDirectory.java:425)
>         at org.apache.lucene.store.FSIndexInput.&lt;init>(FSDirectory.java:434)
>         at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:324)
>         at org.apache.lucene.index.TermInfosReader.&lt;init>(TermInfosReader.java:52)
>         at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:147)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:129)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:110)
>         at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:154)
>         at org.apache.lucene.store.Lock$With.run(Lock.java:109)
>         at org.apache.lucene.index.IndexReader.open(IndexReader.java:143)
>         at org.apache.lucene.index.IndexReader.open(IndexReader.java:127)
>         at org.apache.solr.search.SolrIndexSearcher.&lt;init>(Unknown Source)
>         at org.apache.solr.core.SolrCore.newSearcher(Unknown Source)
>         at org.apache.solr.update.DirectUpdateHandler2.openSearcher(Unknown Source)
>         at org.apache.solr.update.DirectUpdateHandler2.doDeletions(Unknown Source)
>         at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(Unknown Source)
>         at org.apache.solr.core.SolrCore.update(Unknown Source)
>         at org.apache.solr.servlet.SolrServlet.doPost(Unknown Source)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:767)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:408)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:350)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:195)
>         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:164)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:536)
>         at org.mortbay.jetty.Server.handle(Server.java:309)
>         at org.mortbay.jetty.Server.handle(Server.java:285)
>         at org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:363)
>         at org.mortbay.jetty.HttpConnection.access$1600(HttpConnection.java:45)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:625)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:613)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:195)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:297)
>         at org.mortbay.jetty.nio.SelectChannelConnector$HttpEndPoint.run(SelectChannelConnector.java:680)
>         at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:412)
> I have rebuild evrything from scratch several times, but I always get the error. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira