You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@roller.apache.org by Glenn Nielsen <gl...@earthdome.org> on 2005/12/18 13:14:47 UTC
JVM Fails with Too Many Files Open
Since upgrading to roller 1.2 and configuring several feeds in
the Planet Tomcat has failed twice due to running out of file
descriptors. Both times the failure occurred about 10 days
after Tomcat was started. I suspect the background threads
for the Planet may be leaking file descriptors for sockets
but haven't confirmed this yet.
FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
Has anyone had similar problems?
Regards,
Glenn Nielsen
Re: JVM Fails with Too Many Files Open
Posted by Glenn Nielsen <gl...@earthdome.org>.
I am using FreeBSD and just installed the lsof port.
Thanks,
Glenn
On Mon, Jan 02, 2006 at 07:30:18AM -0800, Anil Gangolli wrote:
> If you have the lsof utility on the system (most linuxes), try using
> that for diagnosis.
> As root run: lsof -p <the_tomcat_pid>.
>
> Glenn Nielsen wrote:
>
> >I have had a third failure. Once again after 9-10 days. There is
> >definitley some sort of file descriptor leak but I couldn't find
> >any more information. Neither netstat or fstat showed anything
> >unusual. Neither did a full thread stack dump of the JVM.
> >
> >Regards,
> >
> >Glenn
> >
> >On Thu, Dec 22, 2005 at 11:56:08PM -0500, Max Rudman wrote:
> >
> >
> >>We've definitely seen this problem of Tomcat running out of sockets.
> >>I our case it's caused by Lucene which seems to leak file descriptors
> >>to search index. Unix shows lots of "orphaned" file descriptors to
> >>various segment files inside the index.
> >>
> >>Max
> >>
> >>
> >>On Dec 22, 2005, at 11:45 PM, Anil Gangolli wrote:
> >>
> >>
> >>
> >>>I've filed ROL-946, which identifies one probable source of a
> >>>connection leak. I'll fix this when I return in a week or so if it
> >>>hasn't already been addressed.
> >>>
> >>>--a
> >>>
> >>>Anil Gangolli wrote:
> >>>
> >>>
> >>>
> >>>>Yeah. Any idea of where the leaks are will help. Please do file
> >>>>a bug because I'd like to have a thread specific to this there.
> >>>>
> >>>>Glenn Nielsen wrote:
> >>>>
> >>>>
> >>>>
> >>>>>Next time it happens I will do some forensics before I restart
> >>>>>Tomcat.
> >>>>>
> >>>>>Things such as a JVM full thread stack dump, netstat, fstat, etc.
> >>>>>
> >>>>>Then I may have a better idea where the file descriptor leak is.
> >>>>>
> >>>>>One thing I have found is that the way GC works may impact the
> >>>>>number of file descriptors. If you don't explicitly close some
> >>>>>IO streams the file descriptor associated with that stream will
> >>>>>still exist until the object is finalized during garbage collection.
> >>>>>
> >>>>>Regards,
> >>>>>
> >>>>>Glenn Nielsen
> >>>>>
> >>>>>On Sun, Dec 18, 2005 at 08:59:22AM -0800, Allen Gilliland wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Actually, you may also want to check the search stuff. i have
> >>>>>>seen that
> >>>>>>same thing happen with Lucene in other apps before.
> >>>>>>
> >>>>>>-- Allen
> >>>>>>
> >>>>>>
> >>>>>>On Sun, 2005-12-18 at 06:38, Anil Gangolli wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>Sounds like something we need to track and look into. Please
> >>>>>>>file a bug report on our issue tracker: http://
> >>>>>>>opensource2.atlassian.com/projects/roller/secure/Dashboard.jspa
> >>>>>>>
> >>>>>>>--a.
> >>>>>>>
> >>>>>>>Glenn Nielsen wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>Since upgrading to roller 1.2 and configuring several feeds in
> >>>>>>>>the Planet Tomcat has failed twice due to running out of file
> >>>>>>>>descriptors. Both times the failure occurred about 10 days
> >>>>>>>>after Tomcat was started. I suspect the background threads
> >>>>>>>>for the Planet may be leaking file descriptors for sockets
> >>>>>>>>but haven't confirmed this yet.
> >>>>>>>>
> >>>>>>>>FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
> >>>>>>>>
> >>>>>>>>Has anyone had similar problems?
> >>>>>>>>
> >>>>>>>>Regards,
> >>>>>>>>
> >>>>>>>>Glenn Nielsen
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >Glenn
> >
> >
> >
> >
>
Glenn
Re: JVM Fails with Too Many Files Open
Posted by Anil Gangolli <an...@busybuddha.org>.
If you have the lsof utility on the system (most linuxes), try using
that for diagnosis.
As root run: lsof -p <the_tomcat_pid>.
Glenn Nielsen wrote:
>I have had a third failure. Once again after 9-10 days. There is
>definitley some sort of file descriptor leak but I couldn't find
>any more information. Neither netstat or fstat showed anything
>unusual. Neither did a full thread stack dump of the JVM.
>
>Regards,
>
>Glenn
>
>On Thu, Dec 22, 2005 at 11:56:08PM -0500, Max Rudman wrote:
>
>
>>We've definitely seen this problem of Tomcat running out of sockets.
>>I our case it's caused by Lucene which seems to leak file descriptors
>>to search index. Unix shows lots of "orphaned" file descriptors to
>>various segment files inside the index.
>>
>>Max
>>
>>
>>On Dec 22, 2005, at 11:45 PM, Anil Gangolli wrote:
>>
>>
>>
>>>I've filed ROL-946, which identifies one probable source of a
>>>connection leak. I'll fix this when I return in a week or so if it
>>>hasn't already been addressed.
>>>
>>>--a
>>>
>>>Anil Gangolli wrote:
>>>
>>>
>>>
>>>>Yeah. Any idea of where the leaks are will help. Please do file
>>>>a bug because I'd like to have a thread specific to this there.
>>>>
>>>>Glenn Nielsen wrote:
>>>>
>>>>
>>>>
>>>>>Next time it happens I will do some forensics before I restart
>>>>>Tomcat.
>>>>>
>>>>>Things such as a JVM full thread stack dump, netstat, fstat, etc.
>>>>>
>>>>>Then I may have a better idea where the file descriptor leak is.
>>>>>
>>>>>One thing I have found is that the way GC works may impact the
>>>>>number of file descriptors. If you don't explicitly close some
>>>>>IO streams the file descriptor associated with that stream will
>>>>>still exist until the object is finalized during garbage collection.
>>>>>
>>>>>Regards,
>>>>>
>>>>>Glenn Nielsen
>>>>>
>>>>>On Sun, Dec 18, 2005 at 08:59:22AM -0800, Allen Gilliland wrote:
>>>>>
>>>>>
>>>>>
>>>>>>Actually, you may also want to check the search stuff. i have
>>>>>>seen that
>>>>>>same thing happen with Lucene in other apps before.
>>>>>>
>>>>>>-- Allen
>>>>>>
>>>>>>
>>>>>>On Sun, 2005-12-18 at 06:38, Anil Gangolli wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Sounds like something we need to track and look into. Please
>>>>>>>file a bug report on our issue tracker: http://
>>>>>>>opensource2.atlassian.com/projects/roller/secure/Dashboard.jspa
>>>>>>>
>>>>>>>--a.
>>>>>>>
>>>>>>>Glenn Nielsen wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>Since upgrading to roller 1.2 and configuring several feeds in
>>>>>>>>the Planet Tomcat has failed twice due to running out of file
>>>>>>>>descriptors. Both times the failure occurred about 10 days
>>>>>>>>after Tomcat was started. I suspect the background threads
>>>>>>>>for the Planet may be leaking file descriptors for sockets
>>>>>>>>but haven't confirmed this yet.
>>>>>>>>
>>>>>>>>FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
>>>>>>>>
>>>>>>>>Has anyone had similar problems?
>>>>>>>>
>>>>>>>>Regards,
>>>>>>>>
>>>>>>>>Glenn Nielsen
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>Glenn
>
>
>
>
Re: JVM Fails with Too Many Files Open
Posted by Glenn Nielsen <gl...@earthdome.org>.
I have had a third failure. Once again after 9-10 days. There is
definitley some sort of file descriptor leak but I couldn't find
any more information. Neither netstat or fstat showed anything
unusual. Neither did a full thread stack dump of the JVM.
Regards,
Glenn
On Thu, Dec 22, 2005 at 11:56:08PM -0500, Max Rudman wrote:
> We've definitely seen this problem of Tomcat running out of sockets.
> I our case it's caused by Lucene which seems to leak file descriptors
> to search index. Unix shows lots of "orphaned" file descriptors to
> various segment files inside the index.
>
> Max
>
>
> On Dec 22, 2005, at 11:45 PM, Anil Gangolli wrote:
>
> >I've filed ROL-946, which identifies one probable source of a
> >connection leak. I'll fix this when I return in a week or so if it
> >hasn't already been addressed.
> >
> >--a
> >
> >Anil Gangolli wrote:
> >
> >>Yeah. Any idea of where the leaks are will help. Please do file
> >>a bug because I'd like to have a thread specific to this there.
> >>
> >>Glenn Nielsen wrote:
> >>
> >>>Next time it happens I will do some forensics before I restart
> >>>Tomcat.
> >>>
> >>>Things such as a JVM full thread stack dump, netstat, fstat, etc.
> >>>
> >>>Then I may have a better idea where the file descriptor leak is.
> >>>
> >>>One thing I have found is that the way GC works may impact the
> >>>number of file descriptors. If you don't explicitly close some
> >>>IO streams the file descriptor associated with that stream will
> >>>still exist until the object is finalized during garbage collection.
> >>>
> >>>Regards,
> >>>
> >>>Glenn Nielsen
> >>>
> >>>On Sun, Dec 18, 2005 at 08:59:22AM -0800, Allen Gilliland wrote:
> >>>
> >>>>Actually, you may also want to check the search stuff. i have
> >>>>seen that
> >>>>same thing happen with Lucene in other apps before.
> >>>>
> >>>>-- Allen
> >>>>
> >>>>
> >>>>On Sun, 2005-12-18 at 06:38, Anil Gangolli wrote:
> >>>>
> >>>>>Sounds like something we need to track and look into. Please
> >>>>>file a bug report on our issue tracker: http://
> >>>>>opensource2.atlassian.com/projects/roller/secure/Dashboard.jspa
> >>>>>
> >>>>>--a.
> >>>>>
> >>>>>Glenn Nielsen wrote:
> >>>>>
> >>>>>
> >>>>>>Since upgrading to roller 1.2 and configuring several feeds in
> >>>>>>the Planet Tomcat has failed twice due to running out of file
> >>>>>>descriptors. Both times the failure occurred about 10 days
> >>>>>>after Tomcat was started. I suspect the background threads
> >>>>>>for the Planet may be leaking file descriptors for sockets
> >>>>>>but haven't confirmed this yet.
> >>>>>>
> >>>>>>FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
> >>>>>>
> >>>>>>Has anyone had similar problems?
> >>>>>>
> >>>>>>Regards,
> >>>>>>
> >>>>>>Glenn Nielsen
> >>>>>>
> >>>>>
> >>>
> >>>
> >>>
> >>
> >>
Glenn
Re: JVM Fails with Too Many Files Open
Posted by Max Rudman <mr...@steelbrick.com>.
We've definitely seen this problem of Tomcat running out of sockets.
I our case it's caused by Lucene which seems to leak file descriptors
to search index. Unix shows lots of "orphaned" file descriptors to
various segment files inside the index.
Max
On Dec 22, 2005, at 11:45 PM, Anil Gangolli wrote:
> I've filed ROL-946, which identifies one probable source of a
> connection leak. I'll fix this when I return in a week or so if it
> hasn't already been addressed.
>
> --a
>
> Anil Gangolli wrote:
>
>> Yeah. Any idea of where the leaks are will help. Please do file
>> a bug because I'd like to have a thread specific to this there.
>>
>> Glenn Nielsen wrote:
>>
>>> Next time it happens I will do some forensics before I restart
>>> Tomcat.
>>>
>>> Things such as a JVM full thread stack dump, netstat, fstat, etc.
>>>
>>> Then I may have a better idea where the file descriptor leak is.
>>>
>>> One thing I have found is that the way GC works may impact the
>>> number of file descriptors. If you don't explicitly close some
>>> IO streams the file descriptor associated with that stream will
>>> still exist until the object is finalized during garbage collection.
>>>
>>> Regards,
>>>
>>> Glenn Nielsen
>>>
>>> On Sun, Dec 18, 2005 at 08:59:22AM -0800, Allen Gilliland wrote:
>>>
>>>> Actually, you may also want to check the search stuff. i have
>>>> seen that
>>>> same thing happen with Lucene in other apps before.
>>>>
>>>> -- Allen
>>>>
>>>>
>>>> On Sun, 2005-12-18 at 06:38, Anil Gangolli wrote:
>>>>
>>>>> Sounds like something we need to track and look into. Please
>>>>> file a bug report on our issue tracker: http://
>>>>> opensource2.atlassian.com/projects/roller/secure/Dashboard.jspa
>>>>>
>>>>> --a.
>>>>>
>>>>> Glenn Nielsen wrote:
>>>>>
>>>>>
>>>>>> Since upgrading to roller 1.2 and configuring several feeds in
>>>>>> the Planet Tomcat has failed twice due to running out of file
>>>>>> descriptors. Both times the failure occurred about 10 days
>>>>>> after Tomcat was started. I suspect the background threads
>>>>>> for the Planet may be leaking file descriptors for sockets
>>>>>> but haven't confirmed this yet.
>>>>>>
>>>>>> FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
>>>>>>
>>>>>> Has anyone had similar problems?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Glenn Nielsen
>>>>>>
>>>>>
>>>
>>>
>>>
>>
>>
Re: JVM Fails with Too Many Files Open
Posted by Anil Gangolli <an...@busybuddha.org>.
I've filed ROL-946, which identifies one probable source of a connection
leak. I'll fix this when I return in a week or so if it hasn't already
been addressed.
--a
Anil Gangolli wrote:
> Yeah. Any idea of where the leaks are will help. Please do file a
> bug because I'd like to have a thread specific to this there.
>
> Glenn Nielsen wrote:
>
>> Next time it happens I will do some forensics before I restart Tomcat.
>>
>> Things such as a JVM full thread stack dump, netstat, fstat, etc.
>>
>> Then I may have a better idea where the file descriptor leak is.
>>
>> One thing I have found is that the way GC works may impact the
>> number of file descriptors. If you don't explicitly close some
>> IO streams the file descriptor associated with that stream will
>> still exist until the object is finalized during garbage collection.
>>
>> Regards,
>>
>> Glenn Nielsen
>>
>> On Sun, Dec 18, 2005 at 08:59:22AM -0800, Allen Gilliland wrote:
>>
>>
>>> Actually, you may also want to check the search stuff. i have seen
>>> that
>>> same thing happen with Lucene in other apps before.
>>>
>>> -- Allen
>>>
>>>
>>> On Sun, 2005-12-18 at 06:38, Anil Gangolli wrote:
>>>
>>>
>>>> Sounds like something we need to track and look into. Please file
>>>> a bug report on our issue tracker:
>>>> http://opensource2.atlassian.com/projects/roller/secure/Dashboard.jspa
>>>>
>>>> --a.
>>>>
>>>> Glenn Nielsen wrote:
>>>>
>>>>
>>>>
>>>>> Since upgrading to roller 1.2 and configuring several feeds in
>>>>> the Planet Tomcat has failed twice due to running out of file
>>>>> descriptors. Both times the failure occurred about 10 days
>>>>> after Tomcat was started. I suspect the background threads
>>>>> for the Planet may be leaking file descriptors for sockets
>>>>> but haven't confirmed this yet.
>>>>>
>>>>> FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
>>>>>
>>>>> Has anyone had similar problems?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Glenn Nielsen
>>>>>
>>>>
>>
>>
>>
>>
>
>
Re: JVM Fails with Too Many Files Open
Posted by Anil Gangolli <an...@busybuddha.org>.
Yeah. Any idea of where the leaks are will help. Please do file a bug
because I'd like to have a thread specific to this there.
Glenn Nielsen wrote:
>Next time it happens I will do some forensics before I restart Tomcat.
>
>Things such as a JVM full thread stack dump, netstat, fstat, etc.
>
>Then I may have a better idea where the file descriptor leak is.
>
>One thing I have found is that the way GC works may impact the
>number of file descriptors. If you don't explicitly close some
>IO streams the file descriptor associated with that stream will
>still exist until the object is finalized during garbage collection.
>
>Regards,
>
>Glenn Nielsen
>
>On Sun, Dec 18, 2005 at 08:59:22AM -0800, Allen Gilliland wrote:
>
>
>>Actually, you may also want to check the search stuff. i have seen that
>>same thing happen with Lucene in other apps before.
>>
>>-- Allen
>>
>>
>>On Sun, 2005-12-18 at 06:38, Anil Gangolli wrote:
>>
>>
>>>Sounds like something we need to track and look into. Please file a bug
>>>report on our issue tracker:
>>>http://opensource2.atlassian.com/projects/roller/secure/Dashboard.jspa
>>>
>>>--a.
>>>
>>>Glenn Nielsen wrote:
>>>
>>>
>>>
>>>>Since upgrading to roller 1.2 and configuring several feeds in
>>>>the Planet Tomcat has failed twice due to running out of file
>>>>descriptors. Both times the failure occurred about 10 days
>>>>after Tomcat was started. I suspect the background threads
>>>>for the Planet may be leaking file descriptors for sockets
>>>>but haven't confirmed this yet.
>>>>
>>>>FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
>>>>
>>>>Has anyone had similar problems?
>>>>
>>>>Regards,
>>>>
>>>>Glenn Nielsen
>>>>
>>>>
>
>
>
>
Re: JVM Fails with Too Many Files Open
Posted by Glenn Nielsen <gl...@earthdome.org>.
Next time it happens I will do some forensics before I restart Tomcat.
Things such as a JVM full thread stack dump, netstat, fstat, etc.
Then I may have a better idea where the file descriptor leak is.
One thing I have found is that the way GC works may impact the
number of file descriptors. If you don't explicitly close some
IO streams the file descriptor associated with that stream will
still exist until the object is finalized during garbage collection.
Regards,
Glenn Nielsen
On Sun, Dec 18, 2005 at 08:59:22AM -0800, Allen Gilliland wrote:
> Actually, you may also want to check the search stuff. i have seen that
> same thing happen with Lucene in other apps before.
>
> -- Allen
>
>
> On Sun, 2005-12-18 at 06:38, Anil Gangolli wrote:
> > Sounds like something we need to track and look into. Please file a bug
> > report on our issue tracker:
> > http://opensource2.atlassian.com/projects/roller/secure/Dashboard.jspa
> >
> > --a.
> >
> > Glenn Nielsen wrote:
> >
> > >Since upgrading to roller 1.2 and configuring several feeds in
> > >the Planet Tomcat has failed twice due to running out of file
> > >descriptors. Both times the failure occurred about 10 days
> > >after Tomcat was started. I suspect the background threads
> > >for the Planet may be leaking file descriptors for sockets
> > >but haven't confirmed this yet.
> > >
> > >FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
> > >
> > >Has anyone had similar problems?
> > >
> > >Regards,
> > >
> > >Glenn Nielsen
Re: JVM Fails with Too Many Files Open
Posted by Allen Gilliland <Al...@Sun.COM>.
Actually, you may also want to check the search stuff. i have seen that
same thing happen with Lucene in other apps before.
-- Allen
On Sun, 2005-12-18 at 06:38, Anil Gangolli wrote:
> Sounds like something we need to track and look into. Please file a bug
> report on our issue tracker:
> http://opensource2.atlassian.com/projects/roller/secure/Dashboard.jspa
>
> --a.
>
> Glenn Nielsen wrote:
>
> >Since upgrading to roller 1.2 and configuring several feeds in
> >the Planet Tomcat has failed twice due to running out of file
> >descriptors. Both times the failure occurred about 10 days
> >after Tomcat was started. I suspect the background threads
> >for the Planet may be leaking file descriptors for sockets
> >but haven't confirmed this yet.
> >
> >FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
> >
> >Has anyone had similar problems?
> >
> >Regards,
> >
> >Glenn Nielsen
> >
> >
> >
> >
>
Re: JVM Fails with Too Many Files Open
Posted by Anil Gangolli <an...@busybuddha.org>.
Sounds like something we need to track and look into. Please file a bug
report on our issue tracker:
http://opensource2.atlassian.com/projects/roller/secure/Dashboard.jspa
--a.
Glenn Nielsen wrote:
>Since upgrading to roller 1.2 and configuring several feeds in
>the Planet Tomcat has failed twice due to running out of file
>descriptors. Both times the failure occurred about 10 days
>after Tomcat was started. I suspect the background threads
>for the Planet may be leaking file descriptors for sockets
>but haven't confirmed this yet.
>
>FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
>
>Has anyone had similar problems?
>
>Regards,
>
>Glenn Nielsen
>
>
>
>
Re: JVM Fails with Too Many Files Open
Posted by Glenn Nielsen <gl...@earthdome.org>.
After a third failure I ran lsof and found several thousand
descriptors for KQUEUE. Googling on this I ran across the
following:
http://lists.freebsd.org/pipermail/freebsd-java/2004-November/003037.html
Looks like I have run across a FreeBSD bug unrelated to Roller itself.
Roller just triggered the bug once I started using Planet Roller.
Regards,
Glenn
On Sun, Dec 18, 2005 at 06:14:47AM -0600, Glenn Nielsen wrote:
> Since upgrading to roller 1.2 and configuring several feeds in
> the Planet Tomcat has failed twice due to running out of file
> descriptors. Both times the failure occurred about 10 days
> after Tomcat was started. I suspect the background threads
> for the Planet may be leaking file descriptors for sockets
> but haven't confirmed this yet.
>
> FreeBSD 5.3, Java 1.4, Tomcat 4.1, Roller 1.2.
>
> Has anyone had similar problems?
>
> Regards,
>
> Glenn Nielsen