You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Gabriele Bulfon <gb...@sonicle.com.INVALID> on 2020/11/02 10:13:12 UTC
[users@httpd] Self built httpd 2.4.43 problems
Hi,
as the mantainers of the XStreamOS/illumos distro, we also have specific server packages where we build Apache httpd together with other server components.
We have systems with apache 2.4.27 running for years without particular problems.
Recently we built version 2.4.43 and installed on a test machine.
Here, we are experiencing a problem where almost once a week we have to restart apache, which is no more responding.
Threads are there, but none is answering on port 80, waiting forever.
Here is the output of httpd -V:
Server version: Apache/2.4.43 (Unix)
Server built: Jul 2 2020 02:26:34
Server's Module Magic Number: 20120211:92
Server loaded: APR 1.5.2, APR-UTIL 1.5.4
Compiled using: APR 1.5.2, APR-UTIL 1.5.4
Architecture: 32-bit
Server MPM: event
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_PROC_PTHREAD_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/sonicle"
-D SUEXEC_BIN="/sonicle/bin/suexec"
-D DEFAULT_PIDLOG="var/apache/logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="etc/apache/mime.types"
-D SERVER_CONFIG_FILE="etc/apache/httpd.conf"
What may be the issue?
Thanks
Gabriele
Sonicle S.r.l.
:
http://www.sonicle.com
Music:
http://www.gabrielebulfon.com
eXoplanets :
https://gabrielebulfon.bandcamp.com/album/exoplanets
Re: [users@httpd] Self built httpd 2.4.43 problems
Posted by Eric Covener <co...@gmail.com>.
On Mon, Nov 30, 2020 at 9:08 AM Gabriele Bulfon
<gb...@sonicle.com.invalid> wrote:
>
> Is there any way I can provoke a core dump from the httpd threads when the freeze happens?
> I tried using truss (strace solaris like) but nothing was moving on the httpd processes.
> Analyzing server-status I see nothing strange, and then it sill happens every 7-10 days...
That is usually OS specific. For example gcore, gencore, pstack, or
attaching with gdb.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Posted by Gabriele Bulfon <gb...@sonicle.com.INVALID>.
Is there any way I can provoke a core dump from the httpd threads when the freeze happens?
I tried using truss (strace solaris like) but nothing was moving on the httpd processes.
Analyzing server-status I see nothing strange, and then it sill happens every 7-10 days...
Gabriele
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
----------------------------------------------------------------------------------
Da: Rainer Canavan <ra...@avenga.com>
A: users@httpd.apache.org
Data: 2 novembre 2020 17.19.27 CET
Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems
On Mon, Nov 2, 2020 at 4:17 PM Gabriele Bulfon
<gb...@sonicle.com.invalid> wrote:
>
> Thanks, I configured and ran server-status after stopping/starting apache.
> Top output is:
>
[...]
> What should I check?
> Also, when system blocks I won't be able to see server-status, as it will be not responding.
> Should I check it daily and look for a specific info that grows?
"requests currently being processed" would probably increase if
threads are permanently blocked. I would recommend logging this every
few seconds, so that you can at least check after the fact how quickly
the system filled up.
You should have ExtendedStatus enabled, which should give you a
complete list of all threads and their states. Any that are active
(probably "W", definitively not "." or "_") processing a single
request for extended periods are suspicious, especially if multiple of
the same kind strat piling up.
If httpd does not respond to requests anymore, and you have multiple
worker childs, you can sometimes get away with killing one and try to
squeeze a status request in there before it gets overrun again.
Anyway, serverstatus will only provide rough hints of what's going on.
If it is indeed httpd, you'll probably need gdb backtraces.
rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Posted by Gabriele Bulfon <gb...@sonicle.com.INVALID>.
Wow, that's it, actually we moved from mpm worker to mpm event on this test machine, and here are the results!
I'll check for a more recent APR ;)
Thanks a lot!
Gabriele
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
----------------------------------------------------------------------------------
Da: Yann Ylavic <yl...@gmail.com>
A: users@httpd.apache.org
Data: 10 febbraio 2021 0.25.58 CET
Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems
Hi Gabriele,
There have been some fixes to the APR library since 1.5.2, notably on
the Solaris apr_pollset_poll() implementation, though I can only think
of a bug [1] pertaining to MPM event (not MPM worker which you seem to
be using).
It could be worth upgrading to a more recent APR-1.6.5 or APR-1.7.0 still.
Regards;
Yann.
[1] https://bz.apache.org/bugzilla/show_bug.cgi?id=61786
On Tue, Feb 9, 2021 at 11:07 AM Gabriele Bulfon
<gb...@sonicle.com.invalid> wrote:
>
> Hi, I finally could produce the httpd blocked problem and ran a script to dump stack of all running processes/threads.
> They strangely look all quite the same! I attach here all the httpd threads dumps, maybe you can help us see why it was blocked?
>
> Thanks!
> Gabriele
>
>
> Sonicle S.r.l. : http://www.sonicle.com
> Music: http://www.gabrielebulfon.com
> eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
>
>
>
>
> ----------------------------------------------------------------------------------
>
> Da: Rainer Canavan <ra...@avenga.com>
> A: users@httpd.apache.org
> Data: 2 novembre 2020 17.19.27 CET
> Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems
>
> On Mon, Nov 2, 2020 at 4:17 PM Gabriele Bulfon
> <gb...@sonicle.com.invalid> wrote:
> >
> > Thanks, I configured and ran server-status after stopping/starting apache.
> > Top output is:
> >
> [...]
>
> > What should I check?
> > Also, when system blocks I won't be able to see server-status, as it will be not responding.
> > Should I check it daily and look for a specific info that grows?
>
> "requests currently being processed" would probably increase if
> threads are permanently blocked. I would recommend logging this every
> few seconds, so that you can at least check after the fact how quickly
> the system filled up.
>
> You should have ExtendedStatus enabled, which should give you a
> complete list of all threads and their states. Any that are active
> (probably "W", definitively not "." or "_") processing a single
> request for extended periods are suspicious, especially if multiple of
> the same kind strat piling up.
>
> If httpd does not respond to requests anymore, and you have multiple
> worker childs, you can sometimes get away with killing one and try to
> squeeze a status request in there before it gets overrun again.
>
> Anyway, serverstatus will only provide rough hints of what's going on.
> If it is indeed httpd, you'll probably need gdb backtraces.
>
> rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Gabriele,
There have been some fixes to the APR library since 1.5.2, notably on
the Solaris apr_pollset_poll() implementation, though I can only think
of a bug [1] pertaining to MPM event (not MPM worker which you seem to
be using).
It could be worth upgrading to a more recent APR-1.6.5 or APR-1.7.0 still.
Regards;
Yann.
[1] https://bz.apache.org/bugzilla/show_bug.cgi?id=61786
On Tue, Feb 9, 2021 at 11:07 AM Gabriele Bulfon
<gb...@sonicle.com.invalid> wrote:
>
> Hi, I finally could produce the httpd blocked problem and ran a script to dump stack of all running processes/threads.
> They strangely look all quite the same! I attach here all the httpd threads dumps, maybe you can help us see why it was blocked?
>
> Thanks!
> Gabriele
>
>
> Sonicle S.r.l. : http://www.sonicle.com
> Music: http://www.gabrielebulfon.com
> eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
>
>
>
>
> ----------------------------------------------------------------------------------
>
> Da: Rainer Canavan <ra...@avenga.com>
> A: users@httpd.apache.org
> Data: 2 novembre 2020 17.19.27 CET
> Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems
>
> On Mon, Nov 2, 2020 at 4:17 PM Gabriele Bulfon
> <gb...@sonicle.com.invalid> wrote:
> >
> > Thanks, I configured and ran server-status after stopping/starting apache.
> > Top output is:
> >
> [...]
>
> > What should I check?
> > Also, when system blocks I won't be able to see server-status, as it will be not responding.
> > Should I check it daily and look for a specific info that grows?
>
> "requests currently being processed" would probably increase if
> threads are permanently blocked. I would recommend logging this every
> few seconds, so that you can at least check after the fact how quickly
> the system filled up.
>
> You should have ExtendedStatus enabled, which should give you a
> complete list of all threads and their states. Any that are active
> (probably "W", definitively not "." or "_") processing a single
> request for extended periods are suspicious, especially if multiple of
> the same kind strat piling up.
>
> If httpd does not respond to requests anymore, and you have multiple
> worker childs, you can sometimes get away with killing one and try to
> squeeze a status request in there before it gets overrun again.
>
> Anyway, serverstatus will only provide rough hints of what's going on.
> If it is indeed httpd, you'll probably need gdb backtraces.
>
> rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Posted by Gabriele Bulfon <gb...@sonicle.com.INVALID>.
Hi, I finally could produce the httpd blocked problem and ran a script to dump stack of all running processes/threads.
They strangely look all quite the same! I attach here all the httpd threads dumps, maybe you can help us see why it was blocked?
Thanks!
Gabriele
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
----------------------------------------------------------------------------------
Da: Rainer Canavan <ra...@avenga.com>
A: users@httpd.apache.org
Data: 2 novembre 2020 17.19.27 CET
Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems
On Mon, Nov 2, 2020 at 4:17 PM Gabriele Bulfon
<gb...@sonicle.com.invalid> wrote:
>
> Thanks, I configured and ran server-status after stopping/starting apache.
> Top output is:
>
[...]
> What should I check?
> Also, when system blocks I won't be able to see server-status, as it will be not responding.
> Should I check it daily and look for a specific info that grows?
"requests currently being processed" would probably increase if
threads are permanently blocked. I would recommend logging this every
few seconds, so that you can at least check after the fact how quickly
the system filled up.
You should have ExtendedStatus enabled, which should give you a
complete list of all threads and their states. Any that are active
(probably "W", definitively not "." or "_") processing a single
request for extended periods are suspicious, especially if multiple of
the same kind strat piling up.
If httpd does not respond to requests anymore, and you have multiple
worker childs, you can sometimes get away with killing one and try to
squeeze a status request in there before it gets overrun again.
Anyway, serverstatus will only provide rough hints of what's going on.
If it is indeed httpd, you'll probably need gdb backtraces.
rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Posted by Rainer Canavan <ra...@avenga.com>.
On Mon, Nov 2, 2020 at 4:17 PM Gabriele Bulfon
<gb...@sonicle.com.invalid> wrote:
>
> Thanks, I configured and ran server-status after stopping/starting apache.
> Top output is:
>
[...]
> What should I check?
> Also, when system blocks I won't be able to see server-status, as it will be not responding.
> Should I check it daily and look for a specific info that grows?
"requests currently being processed" would probably increase if
threads are permanently blocked. I would recommend logging this every
few seconds, so that you can at least check after the fact how quickly
the system filled up.
You should have ExtendedStatus enabled, which should give you a
complete list of all threads and their states. Any that are active
(probably "W", definitively not "." or "_") processing a single
request for extended periods are suspicious, especially if multiple of
the same kind strat piling up.
If httpd does not respond to requests anymore, and you have multiple
worker childs, you can sometimes get away with killing one and try to
squeeze a status request in there before it gets overrun again.
Anyway, serverstatus will only provide rough hints of what's going on.
If it is indeed httpd, you'll probably need gdb backtraces.
rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Posted by Gabriele Bulfon <gb...@sonicle.com.INVALID>.
Thanks, I configured and ran server-status after stopping/starting apache.
Top output is:
Current Time: Monday, 02-Nov-2020 16:16:14 CETRestart Time: Monday, 02-Nov-2020 16:04:54 CETParent Server Config. Generation: 1Parent Server MPM Generation: 0Server uptime: 11 minutes 20 secondsServer load: 0.16 0.23 0.35Total accesses: 553 - Total Traffic: 10.6 MB - Total Duration: 1637098CPU Usage: u4.22 s.41 cu0 cs0 - .681% CPU load.813 requests/sec - 15.9 kB/second - 19.6 kB/request - 2960.39 ms/request5 requests currently being processed, 95 idle workers
What should I check?
Also, when system blocks I won't be able to see server-status, as it will be not responding.
Should I check it daily and look for a specific info that grows?
Gabriele
Sonicle S.r.l.
:
http://www.sonicle.com
Music:
http://www.gabrielebulfon.com
eXoplanets :
https://gabrielebulfon.bandcamp.com/album/exoplanets
----------------------------------------------------------------------------------
Da: Rainer Canavan
A: users@httpd.apache.org
Data: 2 novembre 2020 14.24.06 CET
Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems
On Mon, Nov 2, 2020 at 11:13 AM Gabriele Bulfon
wrote:
[...]
Recently we built version 2.4.43 and installed on a test machine.
Here, we are experiencing a problem where almost once a week we have to restart apache, which is no more responding.
Threads are there, but none is answering on port 80, waiting forever.
[...]
What may be the issue?
Lots of options, more information needed. If it's an issue that slowly
eats up your worker threads, monitoring the server-status page with
ExtendedStatus On may provide good hints. Otherwise, wait until it
stops responding and try to check with strace if anything suspicious
is going on. Finally, check
"thread apply all bt" in a gdb session attached to some of the blocked
processes, possibly a few times with cont and Ctrl+C inbetween to see
where it's really waiting. You may need to re-build with debug info,
and/or install the debug info for all libraries your httpd is linked
against.
rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Posted by Rainer Canavan <ra...@avenga.com>.
On Mon, Nov 2, 2020 at 11:13 AM Gabriele Bulfon
<gb...@sonicle.com.invalid> wrote:
>[...]
> Recently we built version 2.4.43 and installed on a test machine.
> Here, we are experiencing a problem where almost once a week we have to restart apache, which is no more responding.
> Threads are there, but none is answering on port 80, waiting forever.
[...]
> What may be the issue?
Lots of options, more information needed. If it's an issue that slowly
eats up your worker threads, monitoring the server-status page with
ExtendedStatus On may provide good hints. Otherwise, wait until it
stops responding and try to check with strace if anything suspicious
is going on. Finally, check
"thread apply all bt" in a gdb session attached to some of the blocked
processes, possibly a few times with cont and Ctrl+C inbetween to see
where it's really waiting. You may need to re-build with debug info,
and/or install the debug info for all libraries your httpd is linked
against.
rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org