You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Michael Douglass <mi...@staff.texas.net> on 1998/04/30 15:49:10 UTC
immortal httpd processes.
Okay, has anyone enjoyed had any problems under Solaris 2.6 where you would
get unkillable (even with -9) web processes? It's annoying the crap out of
me.
Thanks for any feedback,
--
Michael Douglass
Texas Networking, Inc.
<msmith> it's raining...it's pouring...the old man...
*** Describe: msmith shuts up now.
Re: archnotes (was Re: immortal httpd processes.)
Posted by Brian Behlendorf <br...@hyperreal.org>.
At 11:09 AM 5/10/98 -0700, you wrote:
>I'm thinking of putting it in the doc tree at htdocs/manual/archnotes.
Given that we have (limited) per-platform tuning documents, this makes good
sense.
Brian
--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
pure chewing satisfaction brian@apache.org
brian@hyperreal.org
archnotes (was Re: immortal httpd processes.)
Posted by Dean Gaudet <dg...@arctic.org>.
It's messages like this which I wish we could make more readily available
to our users. I've been keeping them in my new-httpd mailbox for
months... and I just flushed them all out. Visit
<http://www.arctic.org/~dgaudet/apache/archnotes/>.
I want this to remain trivial to add new notes to, otherwise we won't add
them. So I *don't* want this stuff turned into html, and I don't want it
made all pretty. I want to be able to just save the message from my
mailer to the files.
I'm thinking of putting it in the doc tree at htdocs/manual/archnotes.
Dean
On Sun, 10 May 1998, Michael Douglass wrote:
> On Fri, May 01, 1998 at 10:47:50PM -0500, Michael Douglass said:
>
> FYI, for all of you who involved yourselves in attempting to assist
> me in my delima, let me tell you the fix. The fix was quite simple
> actually; the problem an annoying one. We have run into this immortal
> process before; but it was on a hugely threaded process (typhoon) on
> another 2.6 box. The kernel patch 105181-05 (still in test-patch
> status, not released to public yet) was given to us for that box. It
> was my understanding that it only affected threaded programs; well,
> little did I know that it also seems to affect non-threaded-but-very-
> busy programs like apache (well, on a large web server).
>
> So, if anyone else has a problem with a solaris 2.6 box with imortal
> processes (won't show up until server gets really busy) contact sun
> to see if you can get T-patch 105181-05. It will save you alot of
> annoying hassles.
Re: immortal httpd processes.
Posted by Michael Douglass <mi...@staff.texas.net>.
On Fri, May 01, 1998 at 10:47:50PM -0500, Michael Douglass said:
FYI, for all of you who involved yourselves in attempting to assist
me in my delima, let me tell you the fix. The fix was quite simple
actually; the problem an annoying one. We have run into this immortal
process before; but it was on a hugely threaded process (typhoon) on
another 2.6 box. The kernel patch 105181-05 (still in test-patch
status, not released to public yet) was given to us for that box. It
was my understanding that it only affected threaded programs; well,
little did I know that it also seems to affect non-threaded-but-very-
busy programs like apache (well, on a large web server).
So, if anyone else has a problem with a solaris 2.6 box with imortal
processes (won't show up until server gets really busy) contact sun
to see if you can get T-patch 105181-05. It will save you alot of
annoying hassles.
> On Fri, May 01, 1998 at 09:34:57PM -0600, Marc Slemko said:
>
> > Could the connectivity between your NFS client and server be dying?
> >
> > Are your NFS mounts interruptable and/or soft? (note that using either
> > makes NFS unreliable but it is already unreliable)
>
> Actually, I just realized that the admins that setup this box didn't
> put any explicit options in there for the NFS mounts; so looking at
> the defaults in `man mount_nfs` on that box: hard and interruptable.
> Also ver3 and tcp are the obvious defaults; we tend to revert most NFS
> mounts back to UDP since we've had a fair number of issues with some
> TCP mounts that go away once moved to UDP.
>
> > Are you reading content or anything else from NFS disks?
>
> Yes, the same box is where the content is stored.
>
> > What sort of box is serving the NFS?
>
> Netapp
>
> > What does ps -elf show the pcoess to be at when it can't be killed? I
> > wish Solaris had nice state info to indicate what things are blocked on
> > like FreeBSD does.
>
> I haven't tried a ps -elf.
>
> > Does lsof show anything of interest with the process?
>
> Won't return
>
> > If it can't be killed, it must be a kernel issue. Apache could trigger
> > it, but no user process should ever be able to do anything to make it
> > unkillable.
>
> Right, I was asking here for two reasons. One, I know you guys to be
> highly intelligent in these types of issues; and two, I was hoping that
> someone else had experienced and worked around this issue.
>
> Our other problem is that we can't seem to get the machine to generate
> a core dump (of the entire system) to send to sun. Is there a command
> under solaris to do this? (ie. intentionally panic the system)
>
>
> --
> Michael Douglass
> Texas Networking, Inc.
>
> <msmith> it's raining...it's pouring...the old man...
> *** Describe: msmith shuts up now.
--
Michael Douglass
Texas Networking, Inc.
<msmith> it's raining...it's pouring...the old man...
*** Describe: msmith shuts up now.
Re: immortal httpd processes.
Posted by Michael Douglass <mi...@staff.texas.net>.
On Fri, May 01, 1998 at 09:34:57PM -0600, Marc Slemko said:
> Could the connectivity between your NFS client and server be dying?
>
> Are your NFS mounts interruptable and/or soft? (note that using either
> makes NFS unreliable but it is already unreliable)
Actually, I just realized that the admins that setup this box didn't
put any explicit options in there for the NFS mounts; so looking at
the defaults in `man mount_nfs` on that box: hard and interruptable.
Also ver3 and tcp are the obvious defaults; we tend to revert most NFS
mounts back to UDP since we've had a fair number of issues with some
TCP mounts that go away once moved to UDP.
> Are you reading content or anything else from NFS disks?
Yes, the same box is where the content is stored.
> What sort of box is serving the NFS?
Netapp
> What does ps -elf show the pcoess to be at when it can't be killed? I
> wish Solaris had nice state info to indicate what things are blocked on
> like FreeBSD does.
I haven't tried a ps -elf.
> Does lsof show anything of interest with the process?
Won't return
> If it can't be killed, it must be a kernel issue. Apache could trigger
> it, but no user process should ever be able to do anything to make it
> unkillable.
Right, I was asking here for two reasons. One, I know you guys to be
highly intelligent in these types of issues; and two, I was hoping that
someone else had experienced and worked around this issue.
Our other problem is that we can't seem to get the machine to generate
a core dump (of the entire system) to send to sun. Is there a command
under solaris to do this? (ie. intentionally panic the system)
--
Michael Douglass
Texas Networking, Inc.
<msmith> it's raining...it's pouring...the old man...
*** Describe: msmith shuts up now.
Re: immortal httpd processes.
Posted by Marc Slemko <ma...@worldgate.com>.
Could the connectivity between your NFS client and server be dying?
Are your NFS mounts interruptable and/or soft? (note that using either
makes NFS unreliable but it is already unreliable)
Are you reading content or anything else from NFS disks?
What sort of box is serving the NFS?
What does ps -elf show the pcoess to be at when it can't be killed? I
wish Solaris had nice state info to indicate what things are blocked on
like FreeBSD does.
Does lsof show anything of interest with the process?
If it can't be killed, it must be a kernel issue. Apache could trigger
it, but no user process should ever be able to do anything to make it
unkillable.
On Fri, 1 May 1998, Michael Douglass wrote:
> On Fri, May 01, 1998 at 10:18:49PM -0500, Michael Douglass said:
> > On Fri, May 01, 1998 at 08:16:44PM -0700, Dean Gaudet said:
> > > Yeah, as marc would say, "nfs locking isn't". It can easily cause all
> > > sorts of nasties.
> > >
> > > What about mod_rewrite, do you have a rewrite log on an NFS filesystem?
> >
> > No, I don't do rewrite logging unless I'm debugging a rule... But...
> > I do have a fairly fast growing access log file on an NFS file system...
> > I was just boiling it down to a possible NFS problem since NFS problems
> > have their ways of making things most interesting for "normalcy" in general.
>
> Oh, an idea of how fast it grows is basically in 22 hours 92MB. So it's
> logging alot of information.
>
> --
> Michael Douglass
> Texas Networking, Inc.
>
> <msmith> it's raining...it's pouring...the old man...
> *** Describe: msmith shuts up now.
>
Re: immortal httpd processes.
Posted by Michael Douglass <mi...@staff.texas.net>.
On Fri, May 01, 1998 at 10:18:49PM -0500, Michael Douglass said:
> On Fri, May 01, 1998 at 08:16:44PM -0700, Dean Gaudet said:
> > Yeah, as marc would say, "nfs locking isn't". It can easily cause all
> > sorts of nasties.
> >
> > What about mod_rewrite, do you have a rewrite log on an NFS filesystem?
>
> No, I don't do rewrite logging unless I'm debugging a rule... But...
> I do have a fairly fast growing access log file on an NFS file system...
> I was just boiling it down to a possible NFS problem since NFS problems
> have their ways of making things most interesting for "normalcy" in general.
Oh, an idea of how fast it grows is basically in 22 hours 92MB. So it's
logging alot of information.
--
Michael Douglass
Texas Networking, Inc.
<msmith> it's raining...it's pouring...the old man...
*** Describe: msmith shuts up now.
Re: immortal httpd processes.
Posted by Michael Douglass <mi...@staff.texas.net>.
On Fri, May 01, 1998 at 08:16:44PM -0700, Dean Gaudet said:
> Yeah, as marc would say, "nfs locking isn't". It can easily cause all
> sorts of nasties.
>
> What about mod_rewrite, do you have a rewrite log on an NFS filesystem?
No, I don't do rewrite logging unless I'm debugging a rule... But...
I do have a fairly fast growing access log file on an NFS file system...
I was just boiling it down to a possible NFS problem since NFS problems
have their ways of making things most interesting for "normalcy" in general.
--
Michael Douglass
Texas Networking, Inc.
<msmith> it's raining...it's pouring...the old man...
*** Describe: msmith shuts up now.
Re: immortal httpd processes.
Posted by Dean Gaudet <dg...@arctic.org>.
Yeah, as marc would say, "nfs locking isn't". It can easily cause all
sorts of nasties.
What about mod_rewrite, do you have a rewrite log on an NFS filesystem?
Dean
On Fri, 1 May 1998, Michael Douglass wrote:
> On Thu, Apr 30, 1998 at 10:32:56AM -0700, Dean Gaudet said:
> > LockFile on an NFS partition?
>
> No; LockFile is on a local file system. Would that cause it to become
> immortal though? (ie. kill -9 won't kill the process. You have to
> literally reboot the box to get it to come back alive again.)
>
> --
> Michael Douglass
> Texas Networking, Inc.
>
> <msmith> it's raining...it's pouring...the old man...
> *** Describe: msmith shuts up now.
>
Re: immortal httpd processes.
Posted by Michael Douglass <mi...@staff.texas.net>.
On Thu, Apr 30, 1998 at 10:32:56AM -0700, Dean Gaudet said:
> LockFile on an NFS partition?
No; LockFile is on a local file system. Would that cause it to become
immortal though? (ie. kill -9 won't kill the process. You have to
literally reboot the box to get it to come back alive again.)
--
Michael Douglass
Texas Networking, Inc.
<msmith> it's raining...it's pouring...the old man...
*** Describe: msmith shuts up now.
Re: immortal httpd processes.
Posted by Dean Gaudet <dg...@arctic.org>.
LockFile on an NFS partition?
Dean
On Thu, 30 Apr 1998, Michael Douglass wrote:
> Okay, has anyone enjoyed had any problems under Solaris 2.6 where you would
> get unkillable (even with -9) web processes? It's annoying the crap out of
> me.
>
> Thanks for any feedback,
> --
> Michael Douglass
> Texas Networking, Inc.
>
> <msmith> it's raining...it's pouring...the old man...
> *** Describe: msmith shuts up now.
>