You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Colin Howe <co...@conversocial.com> on 2012/03/14 11:13:55 UTC
Too many open files - lots of sockets
Hello,
We keep hitting the too many open files exception. Looking at lsof we have
a lot (several thousand) of entries like this:
java 19339 root 1619u sock 0,7 0t0
682291383 can't identify protocol
However, netstat -a doesn't show any of these.
Can anyone suggest a way to diagnose what these socket entries are? Happy
to post any more information as needed.
Cheers,
Colin
--
Colin Howe
@colinhowe
VP of Engineering
Conversocial Ltd
conversocial.com
Re: Too many open files - lots of sockets
Posted by Erick Erickson <er...@gmail.com>.
Colin:
FYI, you might consider just setting up the autocommit (or commitWithin if
you're using SolrJ) for some reasonable interval (I often use 10 minutes or so).
Even though you've figured it is a Tomcat issue, each
commit causes searcher re-opens, perhaps replication in a master/slave
setup, increased merges etc. It works, but it's also resource intensive...
FWIW
Erick
On Wed, Mar 14, 2012 at 6:40 AM, Michael Kuhlmann <ku...@solarier.de> wrote:
> Ah, good to know! Thank you!
>
> I already had Jetty under suspicion, but we had this failure quite often in
> October and November, when the bug was not yet reported.
>
> -Kuli
>
> Am 14.03.2012 12:08, schrieb Colin Howe:
>
>> After some more digging around I discovered that there was a bug reported
>> in jetty 6: https://jira.codehaus.org/browse/JETTY-1458
>>
>> This prompted me to upgrade to Jetty 7 and things look a bit more stable
>> now :)
Re: Too many open files - lots of sockets
Posted by Michael Kuhlmann <ku...@solarier.de>.
Ah, good to know! Thank you!
I already had Jetty under suspicion, but we had this failure quite often
in October and November, when the bug was not yet reported.
-Kuli
Am 14.03.2012 12:08, schrieb Colin Howe:
> After some more digging around I discovered that there was a bug reported
> in jetty 6: https://jira.codehaus.org/browse/JETTY-1458
>
> This prompted me to upgrade to Jetty 7 and things look a bit more stable
> now :)
Re: Too many open files - lots of sockets
Posted by Colin Howe <co...@conversocial.com>.
After some more digging around I discovered that there was a bug reported
in jetty 6: https://jira.codehaus.org/browse/JETTY-1458
This prompted me to upgrade to Jetty 7 and things look a bit more stable
now :)
On Wed, Mar 14, 2012 at 10:26 AM, Michael Kuhlmann <ku...@solarier.de> wrote:
> I had the same problem, without auto-commit.
>
> I never really found out what exactly the reason was, but I think it was
> because commits were triggered before a previous commit had the chance to
> finish.
>
> We now commit after every minute or 1000 (quite large) documents, whatever
> comes first. And we never optimize. We haven't had this exceptions for
> months now.
>
> Good luck!
> -Kuli
>
> Am 14.03.2012 11:22, schrieb Colin Howe:
>
>> Currently using 3.4.0. We have autocommit enabled but we manually do
>> commits every 100 documents anyway... I can turn it off if you think that
>> might help.
>>
>>
>> Cheers,
>> Colin
>>
>>
>> On Wed, Mar 14, 2012 at 10:24 AM, Markus Jelsma
>> <ma...@openindex.io>**wrote:
>>
>> Are you running trunk and have auto-commit enabled? Then disable
>>> auto-commit. Even if you increase ulimits it will continue to swallow all
>>> available file descriptors.
>>>
>>>
>>> On Wed, 14 Mar 2012 10:13:55 +0000, Colin Howe<co...@conversocial.com>
>>> wrote:
>>>
>>> Hello,
>>>>
>>>> We keep hitting the too many open files exception. Looking at lsof we
>>>> have
>>>> a lot (several thousand) of entries like this:
>>>>
>>>> java 19339 root 1619u sock 0,7
>>>> 0t0
>>>> 682291383 can't identify protocol
>>>>
>>>>
>>>> However, netstat -a doesn't show any of these.
>>>>
>>>> Can anyone suggest a way to diagnose what these socket entries are?
>>>> Happy
>>>> to post any more information as needed.
>>>>
>>>>
>>>> Cheers,
>>>> Colin
>>>>
>>>>
>>> --
>>> Markus Jelsma - CTO - Openindex
>>> http://www.linkedin.com/in/****markus17<http://www.linkedin.com/in/**markus17>
>>> <http://www.linkedin.**com/in/markus17<http://www.linkedin.com/in/markus17>
>>> >
>>> 050-8536600 / 06-50258350
>>>
>>>
>>
>>
>>
>
--
Colin Howe
@colinhowe
VP of Engineering
Conversocial Ltd
conversocial.com
Re: Too many open files - lots of sockets
Posted by Michael Kuhlmann <ku...@solarier.de>.
I had the same problem, without auto-commit.
I never really found out what exactly the reason was, but I think it was
because commits were triggered before a previous commit had the chance
to finish.
We now commit after every minute or 1000 (quite large) documents,
whatever comes first. And we never optimize. We haven't had this
exceptions for months now.
Good luck!
-Kuli
Am 14.03.2012 11:22, schrieb Colin Howe:
> Currently using 3.4.0. We have autocommit enabled but we manually do
> commits every 100 documents anyway... I can turn it off if you think that
> might help.
>
>
> Cheers,
> Colin
>
>
> On Wed, Mar 14, 2012 at 10:24 AM, Markus Jelsma
> <ma...@openindex.io>wrote:
>
>> Are you running trunk and have auto-commit enabled? Then disable
>> auto-commit. Even if you increase ulimits it will continue to swallow all
>> available file descriptors.
>>
>>
>> On Wed, 14 Mar 2012 10:13:55 +0000, Colin Howe<co...@conversocial.com>
>> wrote:
>>
>>> Hello,
>>>
>>> We keep hitting the too many open files exception. Looking at lsof we have
>>> a lot (several thousand) of entries like this:
>>>
>>> java 19339 root 1619u sock 0,7 0t0
>>> 682291383 can't identify protocol
>>>
>>>
>>> However, netstat -a doesn't show any of these.
>>>
>>> Can anyone suggest a way to diagnose what these socket entries are? Happy
>>> to post any more information as needed.
>>>
>>>
>>> Cheers,
>>> Colin
>>>
>>
>> --
>> Markus Jelsma - CTO - Openindex
>> http://www.linkedin.com/in/**markus17<http://www.linkedin.com/in/markus17>
>> 050-8536600 / 06-50258350
>>
>
>
>
Re: Too many open files - lots of sockets
Posted by Colin Howe <co...@conversocial.com>.
Currently using 3.4.0. We have autocommit enabled but we manually do
commits every 100 documents anyway... I can turn it off if you think that
might help.
Cheers,
Colin
On Wed, Mar 14, 2012 at 10:24 AM, Markus Jelsma
<ma...@openindex.io>wrote:
> Are you running trunk and have auto-commit enabled? Then disable
> auto-commit. Even if you increase ulimits it will continue to swallow all
> available file descriptors.
>
>
> On Wed, 14 Mar 2012 10:13:55 +0000, Colin Howe <co...@conversocial.com>
> wrote:
>
>> Hello,
>>
>> We keep hitting the too many open files exception. Looking at lsof we have
>> a lot (several thousand) of entries like this:
>>
>> java 19339 root 1619u sock 0,7 0t0
>> 682291383 can't identify protocol
>>
>>
>> However, netstat -a doesn't show any of these.
>>
>> Can anyone suggest a way to diagnose what these socket entries are? Happy
>> to post any more information as needed.
>>
>>
>> Cheers,
>> Colin
>>
>
> --
> Markus Jelsma - CTO - Openindex
> http://www.linkedin.com/in/**markus17<http://www.linkedin.com/in/markus17>
> 050-8536600 / 06-50258350
>
--
Colin Howe
@colinhowe
VP of Engineering
Conversocial Ltd
conversocial.com
Re: Too many open files - lots of sockets
Posted by Markus Jelsma <ma...@openindex.io>.
Are you running trunk and have auto-commit enabled? Then disable
auto-commit. Even if you increase ulimits it will continue to swallow
all available file descriptors.
On Wed, 14 Mar 2012 10:13:55 +0000, Colin Howe <co...@conversocial.com>
wrote:
> Hello,
>
> We keep hitting the too many open files exception. Looking at lsof we
> have
> a lot (several thousand) of entries like this:
>
> java 19339 root 1619u sock 0,7
> 0t0
> 682291383 can't identify protocol
>
>
> However, netstat -a doesn't show any of these.
>
> Can anyone suggest a way to diagnose what these socket entries are?
> Happy
> to post any more information as needed.
>
>
> Cheers,
> Colin
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536600 / 06-50258350