You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-dev@lucene.apache.org by "Ryan McKinley (JIRA)" <ji...@apache.org> on 2007/02/24 22:20:05 UTC

[jira] Created: (SOLR-173) "too many open files" with posting to update handler

"too many open files" with posting to update handler
----------------------------------------------------

                 Key: SOLR-173
                 URL: https://issues.apache.org/jira/browse/SOLR-173
             Project: Solr
          Issue Type: Bug
          Components: update
    Affects Versions: 1.2
            Reporter: Ryan McKinley


>From brian:

1) Download trunk/nightly
2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
3) java -jar start.jar...
3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
4) After a few seconds/minutes jetty will crash with "too many open files"

- - - - -

all you've got to do is

apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done

with the request handler pointing to /update. Use

# lsof | grep solr | wc -l

to watch the fdescs fly.



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [jira] Updated: (SOLR-173) "too many open files" with posting toupdate handler

Posted by Ryan McKinley <ry...@gmail.com>.

This problem is different then the normal situation where you can
increase it with "ulimit -n XXX".

This is a real bug introduced by SOLR-104.  As Yonik pointed out, the
filter does not close its request and when the searcher is opened for
an update (unnecessary) it hangs on to closed files indefinitely.

I think the patch on SOLR-173 is good to go - if anyone has a chance
to look at it, that would be great.

thanks
ryan


On 2/26/07, Cook, Jeryl <JC...@innodata-isogen.com> wrote:
> I remember when using Lucene directly in a project, I noticed this error
> only comes up if the system is deployed on UNIX boxes..in our case
> SOLARIS.
>
> You can increase the number of allowed "open files" on unix ..i forget
> the command but the default is 256.   We increased it, and the problem
> went away.(but the patch looks to address it programmatically as well.)
>
> Jeryl Cook
>
> -----Original Message-----
> From: Ryan McKinley (JIRA) [mailto:jira@apache.org]
> Sent: Saturday, February 24, 2007 7:43 PM
> To: solr-dev@lucene.apache.org
> Subject: [jira] Updated: (SOLR-173) "too many open files" with posting
> toupdate handler
>
>
>      [
> https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.p
> lugin.system.issuetabpanels:all-tabpanel ]
>
> Ryan McKinley updated SOLR-173:
> -------------------------------
>
>     Attachment: SOLR-173-open-files-bug.patch
>
> modifying for problem #2 was easier then i expected, it only touches
>  JSONResponseWriter.java
>  TextResponseWriter.java
>  XMLWriter.java
>
> rather then initalize the searcher in the constructor, they load a local
> SolrIndexSearcher inside the writeDocList() function
>
> > "too many open files" with posting to update handler
> > ----------------------------------------------------
> >
> >                 Key: SOLR-173
> >                 URL: https://issues.apache.org/jira/browse/SOLR-173
> >             Project: Solr
> >          Issue Type: Bug
> >          Components: update
> >    Affects Versions: 1.2
> >            Reporter: Ryan McKinley
> >         Attachments: SOLR-173-open-files-bug.patch,
> SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
> >
> >
> > From brian:
> > 1) Download trunk/nightly
> > 2) Change line 347 of example/solr/conf/solrconfig.xml to
> <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> > 3) java -jar start.jar...
> > 3) Run post.sh a bunch of times on the same xml file... (in a shell
> script or whatever)
> > 4) After a few seconds/minutes jetty will crash with "too many open
> files"
> > - - - - -
> > all you've got to do is
> > apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do
> ./post.sh hd.xml; done
> > with the request handler pointing to /update. Use
> > # lsof | grep solr | wc -l
> > to watch the fdescs fly.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>

RE: [jira] Updated: (SOLR-173) "too many open files" with posting toupdate handler

Posted by "Cook, Jeryl" <JC...@innodata-isogen.com>.

I remember when using Lucene directly in a project, I noticed this error
only comes up if the system is deployed on UNIX boxes..in our case
SOLARIS.

You can increase the number of allowed "open files" on unix ..i forget
the command but the default is 256.   We increased it, and the problem
went away.(but the patch looks to address it programmatically as well.)

Jeryl Cook

-----Original Message-----
From: Ryan McKinley (JIRA) [mailto:jira@apache.org] 
Sent: Saturday, February 24, 2007 7:43 PM
To: solr-dev@lucene.apache.org
Subject: [jira] Updated: (SOLR-173) "too many open files" with posting
toupdate handler


     [
https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.p
lugin.system.issuetabpanels:all-tabpanel ]

Ryan McKinley updated SOLR-173:
-------------------------------

    Attachment: SOLR-173-open-files-bug.patch

modifying for problem #2 was easier then i expected, it only touches 
 JSONResponseWriter.java
 TextResponseWriter.java
 XMLWriter.java

rather then initalize the searcher in the constructor, they load a local
SolrIndexSearcher inside the writeDocList() function

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch,
SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell
script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open
files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do
./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (SOLR-173) "too many open files" with posting to update handler

Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475668 ] 

Yonik Seeley commented on SOLR-173:
-----------------------------------

The problem is the use of the response writers for something they weren't originally indended for.

Requests are pretty much always associated with an index searcher.  The searcher is grabbed on-demand and used for the remainder of the request's lifetime.  This is important (lucene docids can change, etc).  At the end of the lifetime of a request, close() *must* be called to decrement the reference count and close the searcher if necessary.  An open searcher holds the files of the index open (so it's view of the index never changes).

Problem #1: it doesn't look like doFilter() closes the request object.
Problem #2: update handlers should not request a searcher in the first place... the response writers need to be modified to call getSearcher() on-demand.

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (SOLR-173) "too many open files" with posting to update handler

Posted by "Hoss Man (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man reassigned SOLR-173:
-----------------------------

    Assignee: Hoss Man

this patch changes the public API of XMLWriter in a way that isn't backwards compatible if anyone was using it in their own plugin ... but i'm guessing the number of people doing that is fairly low (if any) and the added safety it provides seems worth while ... doing some sanity testing and then i'll commit.

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Assigned To: Hoss Man
>         Attachments: SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (SOLR-173) "too many open files" with posting to update handler

Posted by "Bertrand Delacretaz (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475664 ] 

Bertrand Delacretaz commented on SOLR-173:
------------------------------------------

After making the change shown above in solrconfig.xml, a commit is enough to make the number of open file handles grow:

  curl http://localhost:8983/solr/update --data-binary '<commit/>'  -H 'Content-type:text/xml; charset=utf-8'

lsof shows that many of these file handles point to files in data/index, which have been deleted during the commit: 

  lsof -p 9563 | grep data/index | wc -l           (where 9563 is my solr's process ID)

shows 398 file handles after a few commits, although my data/index dir contains only 47 files.

So it looks like something is keeping useless open handles to "old" index files after a commit.



> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (SOLR-173) "too many open files" with posting to update handler

Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ryan McKinley updated SOLR-173:
-------------------------------

    Attachment: SOLR-173-open-files-bug.patch

this patch takes care of Problem #1, it does not touch #2

closing the request object keeps the open files from stacking up.

thanks Yonik

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (SOLR-173) "too many open files" with posting to update handler

Posted by "Hoss Man (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man resolved SOLR-173.
---------------------------

    Resolution: Fixed

commited

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Assigned To: Hoss Man
>         Attachments: SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (SOLR-173) "too many open files" with posting to update handler

Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ryan McKinley updated SOLR-173:
-------------------------------

    Attachment: SOLR-173-open-files-bug.patch

This is not a real 'fix' but it does make the problem go away.  I don't understand how or why.  Essentially if you use the QueryResponseWriter it runs out of open files - if you write the output directly, it is fine.

It does not seem to make any difference what is in the response.

Anyone have any idea what could be going on?

- - - - 

to make post.sh work with the update handler stuff, you will need to add the content type to <commit/>

curl $URL --data-binary '<commit/>' -H 'Content-type:text/xml; charset=utf-8'


> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (SOLR-173) "too many open files" with posting to update handler

Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ryan McKinley updated SOLR-173:
-------------------------------

    Attachment: SOLR-173-open-files-bug.patch

modifying for problem #2 was easier then i expected, it only touches 
 JSONResponseWriter.java
 TextResponseWriter.java
 XMLWriter.java

rather then initalize the searcher in the constructor, they load a local SolrIndexSearcher inside the writeDocList() function

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.