You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2017/10/25 15:03:24 UTC
[Bug 61666] New: regression on httpd reload (could not bind to
address 0.0.0.0:80)
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
Bug ID: 61666
Summary: regression on httpd reload (could not bind to address
0.0.0.0:80)
Product: Apache httpd-2
Version: 2.4.28
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mpm_event
Assignee: bugs@httpd.apache.org
Reporter: thomas.jarosch@intra2net.com
Target Milestone: ---
We have an autotest based automated test system that automatically tests
various httpd and mod_proxy configurations. A few tests failed 100% of the time
after updating from 2.4.25 to 2.4.29.
As soon as the script issues
As soon as a script issues a "httpd reload", this appears in the logs:
[mpm_event:notice] [pid 2452:tid 3072501504] AH00494: SIGHUP received.
Attempting to restart
(98)Address already in use: AH00072: make_sock: could not bind to address
0.0.0.0:80
[mpm_event:alert] [pid 2452:tid 3072501504] no listening sockets available,
shutting down
I've tracked it down to this change in httpd 2.4.26:
*) Don't set SO_REUSEPORT unless ListenCoresBucketsRatio is greater
than zero. [Eric Covener]
Commit in questions is this one:
https://github.com/apache/httpd/commit/d3e0ad6e2ba88f465765e0a8c315189c311583a5
Once I revert it, "httpd reload" works just fine.
Looks like graceful shutdowns now cause trouble when doing httpd reloads?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
--- Comment #2 from Eric Covener <co...@gmail.com> ---
Do you set ListenCoresBucketsRatio? Were you by any chance running tests work
prior to 2.4.17 (prior to SO_REUSEPORT)?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
--- Comment #4 from Eric Covener <co...@gmail.com> ---
> Listen 192.168.1.254:80
> Listen 192.168.1.254:443
> Listen 127.0.0.1:80
> Listen 127.0.0.1:443
> Listen 447
> -------------------
Interesting, did you redact? This is not consistent with the failure msg
(0.0.0.0:80 / 80)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
Thomas Jarosch <th...@intra2net.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |WORKSFORME
Status|NEW |RESOLVED
--- Comment #10 from Thomas Jarosch <th...@intra2net.com> ---
Finally tracked it down:
The automated system changes the "Listen" directives for a few tests and issues
a "httpd reload". While this worked fine in httpd 2.4.25, it stopped working
once the SO_REUSEPORT handling was changed in 2.4.26+.
I've adapted the tests to issue a "httpd restart" if the Listen directives
change and a quick test finished without errors. I'll run the full testsuite
over the weekend.
Here's a diff of the config file before the "httpd reload":
-Listen 172.18.0.1:80
-Listen 172.18.0.1:443
-Listen 192.168.2.1:80
-Listen 192.168.2.1:443
-Listen 127.0.0.1:80
-Listen 127.0.0.1:443
+Listen 80
Listen 447
So Joe was on the right track in comment #6. Changing from "Listen
127.0.0.1:80" to just "Listen 80" is a pretty uncommon (stupid?) thing.
For me the matter is resolved, I've adapted the code and all should be fine.
If someone else ever stumbles upon this issue, we could think about a warning
message in the error log that changing the Listen directives in this way is
unsupported during a SIGHUP.
My thanks to everyone involved!
Have a nice weekend,
Thomas
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
--- Comment #7 from Thomas Jarosch <th...@intra2net.com> ---
(In reply to Joe Orton from comment #6)
> For a config like:
>
> Listen 127.0.0.1:80
> Listen 80
I don't think we are having a mix of "Listen XX" with an explicit IP address
and one without on the same port number. That's a different thing.
In the first config I've posted, it's port 443 and *447*.
(we use 447 to serve a different SSL cert)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
--- Comment #5 from Thomas Jarosch <th...@intra2net.com> ---
(In reply to Eric Covener from comment #4)
> > Listen 192.168.1.254:80
> > Listen 192.168.1.254:443
> > Listen 127.0.0.1:80
> > Listen 127.0.0.1:443
> > Listen 447
> > -------------------
>
> Interesting, did you redact? This is not consistent with the failure msg
> (0.0.0.0:80 / 80)
sorry, I've grabbed the config from another VM while the original VM was
executing a different test.
This is how the config looks like:
----
Listen 80
Listen 447
----
The thread limits etc. are identical.
Regarding our upgrade history: We went straight from 2.0.64 to 2.4.7 in January
2014. The switch from the prefork mpm to the event mpm was done already with
version 2.4.18 in June 2016.
But it looks like I hit the issue last night again while doing a stress test on
mod_proxy with the reverted patch. It just doesn't appear every time.
I'll now double check the patch was in there. If so, I 'll take version 2.4.27
instead of 2.4.29 to make sure we are not searching for a different issue.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
--- Comment #6 from Joe Orton <jo...@redhat.com> ---
For a config like:
Listen 127.0.0.1:80
Listen 80
httpd should fail to start by default. This was true up to 2.4.16, but the
behaviour accidentally regressed in 2.4.17 with the listener bucket code. From
2.4.26 up it will fail again.
It's possible you had a config working with (2.4.17..25) which now fails, but
this is not a regression.
Alternatively, it's possible you have some unrelated issue. Please provide
output from:
strace -ebind ./bin/httpd
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
Thomas Jarosch <th...@intra2net.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |thomas.jarosch@intra2net.co
| |m
--- Comment #1 from Thomas Jarosch <th...@intra2net.com> ---
PS: There's no 2.4.29 version in bugzilla yet, 2.4.28 is the highest as of
writing.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
Joe Orton <jo...@redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|WORKSFORME |FIXED
Status|RESOLVED |CLOSED
--- Comment #11 from Joe Orton <jo...@redhat.com> ---
(In reply to Thomas Jarosch from comment #10)
> So Joe was on the right track in comment #6. Changing from "Listen
> 127.0.0.1:80" to just "Listen 80" is a pretty uncommon (stupid?) thing.
That's a pretty interesting special case, at least I wasn't aware of it, so
thanks a lot for reporting back!
I documented this in r1816112
https://httpd.apache.org/docs/trunk/bind.html#reload
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
--- Comment #8 from Joe Orton <jo...@redhat.com> ---
All you've shown us is a conflict error message from :80. If there is some
issue here other than a config error you need to show full Listen config &
strace output I indicated above.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
Thomas Jarosch <th...@intra2net.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|2.4.28 |2.4.29
--- Comment #3 from Thomas Jarosch <th...@intra2net.com> ---
ListenCoresBucketsRatio is not explicitly set as far as I can tell.
Relevant part of httpd.conf:
-------------------
Timeout 300
KeepAliveTimeout 300
GracefulShutdownTimeout 1800
<IfModule mpm_event_module>
# MinSpareThreads 10
# MaxSpareThreads 250
#
# Number of concurrent connections is: ServerLimit * ThreadsPerChild
# Result: 160 * 5 -> 800
#
StartServers 1
ServerLimit 160
ThreadLimit 5
ThreadsPerChild 5
MaxConnectionsPerChild 5000
</IfModule>
Listen 192.168.1.254:80
Listen 192.168.1.254:443
Listen 127.0.0.1:80
Listen 127.0.0.1:443
Listen 447
-------------------
I'll dig through our git history tomorrow at which pointed we switched from
httpd 2.2.x to 2.4.x. There was also another switch later on from prefork mpm
to the event mpm. 2.4.17 was released around September 2015, we'll see
tomorrow.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61666] regression on httpd reload (could not bind to address
0.0.0.0:80)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61666
--- Comment #9 from Thomas Jarosch <th...@intra2net.com> ---
(In reply to Joe Orton from comment #8)
> All you've shown us is a conflict error message from :80. If there is some
> issue here other than a config error you need to show full Listen config &
> strace output I indicated above.
the "Listen" lines have been provided in comment #5, here they are again:
Listen 80
Listen 447
# strace -ebind httpd -f /etc/httpd/conf/httpd.conf
bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
bind(4, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")},
16) = 0
bind(4, {sa_family=AF_INET, sin_port=htons(447),
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
The output is from httpd 2.4.29 + revert of the commit mentioned in comment #1.
I've been running the mod_proxy tests on two machines in a loop all afternoon.
So far no single failure. If I stick to vanilla httpd 2.4.29 or 2.4.27, it
fails on "httpd reload". Note: This is no *startup* failure, httpd starts fine.
The graceful restart during "httpd reload" -> SIGHUP dies.
What's the normal control flow once a SIGHUP is received in the event mpm?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org