You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2020/06/18 05:48:42 UTC
[Bug 64533] New: Http crashes observed during fuzzing testing
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
Bug ID: 64533
Summary: Http crashes observed during fuzzing testing
Product: Apache httpd-2
Version: 2.4.41
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: All
Assignee: bugs@httpd.apache.org
Reporter: wei-mark.zheng@nokia-sbell.com
Target Milestone: ---
http crashes was observed during the fuzzing testing. The http version is
Apache/2.4.41 (Unix). See logs attached. Please check it what is the cause of
this problem.
Note: Fuzzing testing is to send malformed packets targets to the service to
verify the http service robustness under this situation.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #16 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Hi Ruediger Pluem,
The linux distro we are using is linux MIPS64, kernel version 4.4.227.
[root@CFPU-0(RNC-1009) /root]
# uname -ar
Linux CFPU-0 4.4.227-octeon-distro.git-v2.105-2-rc-wnd #1 SMP Fri Jul 31
11:39:15 UTC 2020 mips64 mips64 mips64 GNU/Linux
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #37 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
It was proxy issue, now the link is accessible.
I will share this patch to team and come back later on.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #25 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Created attachment 37477
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37477&action=edit
backstraces_2909
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #22 from Ruediger Pluem <rp...@apache.org> ---
(In reply to wei-mark.zheng@nokia-sbell.com from comment #21)
> Hi Ruediger Pluem,
> I understood your suggestion. Here the situation is that our product is
> based on MIPS hardware and linux distribution for quite long time. And we
> are using apache component from the start of our product when it works fine
> without issue.
> The problem occurs starting from version 2.4.41 and remains still now. So it
> is likely the problem was introduced in version 2.4.41 by new software code.
> Some questions:
> 1. Do you receive similar issue from other products using same apache
> version ?
No.
> 2. Is there any other possible way to help debug this issue ?
> Please kindly suggest, thanks !
I am out of ideas for your platform. Hence the idea of you doing your fuzzing
against a more common platform where we could possibly get the data we need if
the issue is reproducable there.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #36 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Hi,
I am not able to access this link due to permission issue. how can this
permission granted ?
http://svn.apache.org/viewvc/httpd/httpd/trunk/server/mpm/event/event.c?r1=1882370&r2=1882369&pathrev=1882370&view=patch
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #1 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Created attachment 37314
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37314&action=edit
coredump
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #20 from Ruediger Pluem <rp...@apache.org> ---
As said, you should try to reproduce the issue on a more common Linux
distribution (Debian or RedHat based) and probably on non MIPS hardware. It
looks like your current setup is not capable of producing the debug information
needed for further investigations.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #26 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
The backstraces are collected, please check.Thanks.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #13 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Latest backtraces attached. This is following the guidelines
http://httpd.apache.org/dev/debugging.html#crashes. If this is no completed
traces, then can we have a virtual-meeting or mail discussion to discuss how to
proceeded for us ? Thanks.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #6 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
The backtrace and configuration files attached for your further checking.
Please let us know if anything needed.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #7 from Ruediger Pluem <rp...@apache.org> ---
Unfortunately the stacktraces do not help as they are not complete. Please try
to install debugging symbols for APR / APR-UTIL and httpd as well.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #31 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Attached here backtarces with “thread apply all bt”also
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #32 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Created attachment 37489
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37489&action=edit
backtraces_0910
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #11 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
BTW: could you share your mailaddress, then it would be more effective to
discuss directly in the mail.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #37314|0 |1
is obsolete| |
Attachment #37315|0 |1
is obsolete| |
Attachment #37323|0 |1
is obsolete| |
Attachment #37324|0 |1
is obsolete| |
Attachment #37355|0 |1
is obsolete| |
Attachment #37358|0 |1
is obsolete| |
--- Comment #27 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Created attachment 37478
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37478&action=edit
backtraces_0930
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #18 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Yes, we have observed the crash starting from the Apache/2.4.41 which is based
on another old kernel version.
But this issue was not observed from old Apache version until Apache/2.4.39.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #21 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Hi Ruediger Pluem,
I understood your suggestion. Here the situation is that our product is
based on MIPS hardware and linux distribution for quite long time. And we are
using apache component from the start of our product when it works fine without
issue.
The problem occurs starting from version 2.4.41 and remains still now. So it is
likely the problem was introduced in version 2.4.41 by new software code.
Some questions:
1. Do you receive similar issue from other products using same apache version ?
2. Is there any other possible way to help debug this issue ?
Please kindly suggest, thanks !
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #30 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Hi,
Checked from SW team, in our codes we don’t have anything related to lzma.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|2.4.41 |2.4.43
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|2.4.43 |2.4.46
--- Comment #23 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
We have observed the crash for new 2.4.46 version as well.
Server version: Apache/2.4.46 (Fedora) Server
In addition, the crash is seen when https [Port 80] and TLS [port 443]
codenomicon suites are run together which sending malformed packets
continously.
But no crash observed when both the suites are run separately.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #33 from Ruediger Pluem <rp...@apache.org> ---
Which version of openssl do you use? Is it taken from a distribution package or
do you compile it on your own?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #12 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Created attachment 37358
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37358&action=edit
backtraces_0710
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #3 from Ruediger Pluem <rp...@apache.org> ---
We need:
1. Stacktraces from gdb (see
http://httpd.apache.org/dev/debugging.html#crashes). The coredumps need to be
analyzed on the system where they got created.
2. We need the error and access logs that were recorded during the crash. The
more verbose the error logs the better.
3. We need the configuration used during the test.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #38 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Hi Ruediger Pluem,
Any instruction on how to patch it ?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #35 from Ruediger Pluem <rp...@apache.org> ---
Can you please try the patch from r1882370 :
http://svn.apache.org/viewvc/httpd/httpd/trunk/server/mpm/event/event.c?r1=1882370&r2=1882369&pathrev=1882370&view=patch
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #10 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Thanks for the feedback.
We are not sure what step is missing during the stacktraces capture.
Could you please kindly give some guidelines on the procedure to capture a full
stacktraces for your analysis ? Thanks.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |wei-mark.zheng@nokia-sbell.
| |com
--- Comment #8 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Created attachment 37355
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37355&action=edit
debug logs with 2.4.43
we have reproduced the issue and collected the logs again with http 2.4.43.
Attached the tar file which has details. Please check.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #5 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Created attachment 37324
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37324&action=edit
configuration files
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #9 from Ruediger Pluem <rp...@apache.org> ---
Unfortunately the stacktraces are still incomplete.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #2 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Created attachment 37315
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37315&action=edit
coredump_2
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #19 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Hi Ruediger Pluem,
Can you share any debug options which can be used for compiling httpd which
can further be used to get more information ?
Also could it possible for you to have remote debugging session where you can
login to our environment directly ? Thanks.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #34 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Hi,
We are currently using openssl-1.1.1g in platform. We will take the same from
the distribution and compile it locally for FPLD purpose.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
Yann Ylavic <yl...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|NEW |RESOLVED
--- Comment #40 from Yann Ylavic <yl...@gmail.com> ---
Backported to upcoming 2.4 (r1888917).
Will be in the next release.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #28 from Joe Orton <jo...@redhat.com> ---
It is quite hard to find actual backtraces in that information.
Please attach JUST the backtraces from running "thread apply all bt" from with
gdb.
The one I can find in there looks like:
Stack trace of thread 3411716:
#0 0x00007fcf5a0ae744 __pthread_rwlock_wrlock (libpthread.so.0)
#1 0x00007fcf59be1a49 CRYPTO_THREAD_write_lock (libcrypto.so.1.1)
#2 0x00007fcf59ba4b07 RAND_get_rand_method (libcrypto.so.1.1)
#3 0x00007fcf59ba4d82 RAND_seed (libcrypto.so.1.1)
#4 0x00007fcf59da25ee ssl_rand_seed (mod_ssl.so)
#5 0x00007fcf59d8d831 ssl_init_ssl_connection (mod_ssl.so)
#6 0x0000563ebc834eed n/a (/usr/sbin/httpd)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #14 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
any updates on this case ?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #39 from Ruediger Pluem <rp...@apache.org> ---
(In reply to wei-mark.zheng@nokia-sbell.com from comment #38)
> Hi Ruediger Pluem,
> Any instruction on how to patch it ?
Like any patch it is applied to the source code with the patch command (or
something similar like svn patch or git apply). For the patch command -p3 seems
to be a sensible option when you running this command from the root of the
Apache source. Afterwards you follow just your further build steps.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #24 from Ruediger Pluem <rp...@apache.org> ---
(In reply to wei-mark.zheng@nokia-sbell.com from comment #23)
> We have observed the crash for new 2.4.46 version as well.
> Server version: Apache/2.4.46 (Fedora) Server
Looks like you are now running your fuzzing tests against a Fedora build and
not your Linux MIPS distro. If this is the case can you please provide
stacktraces again. Hopefully they are usable then.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
Re: [Bug 64533] Http crashes observed during fuzzing testing
Posted by Joe Orton <jo...@redhat.com>.
On Fri, Oct 09, 2020 at 11:57:38AM +0200, Ruediger Pluem wrote:
> Would the following patch makes sense (against trunk)?
>
> Index: server/mpm/event/event.c
> ===================================================================
> --- server/mpm/event/event.c (revision 1882251)
> +++ server/mpm/event/event.c (working copy)
> @@ -2897,8 +2897,8 @@
> * the other threads in the process needs to take us down
> * (e.g., for MaxConnectionsPerChild) it will send us SIGTERM
> */
> + apr_signal(SIGTERM, dummy_signal_handler);
> unblock_signal(SIGTERM);
> - apr_signal(SIGTERM, dummy_signal_handler);
> /* Watch for any messages from the parent over the POD */
> while (1) {
> rv = ap_mpm_podx_check(my_bucket->pod);
>
>
> It looks like a queued SIG_TERM is delivered to the current SIG_TERM handler immediately after we unblocked it before we could
> change the handler.
Oh, very nice catch. Yes that looks exactly right, +1.
Regards, Joe
Re: [Bug 64533] Http crashes observed during fuzzing testing
Posted by Ruediger Pluem <rp...@apache.org>.
Looking at the latest backtraces, I see the following:
#7 0x00007fcf5921d716 in clean_child_exit (code=code@entry=0) at event.c:738
#8 0x00007fcf5921d73d in just_die (sig=<optimized out>) at event.c:743
#9 <signal handler called>
#10 pthread_sigmask (how=how@entry=1, newmask=<optimized out>, newmask@entry=0x7fff02b26230, oldmask=oldmask@entry=0x0) at
../sysdeps/unix/sysv/linux/pthread_sigmask.c:48
#11 0x00007fcf5921ccd5 in unblock_signal (sig=sig@entry=15) at event.c:1264
#12 0x00007fcf5921e5d4 in child_main (child_num_arg=child_num_arg@entry=7, child_bucket=child_bucket@entry=0) at event.c:2586
#13 0x00007fcf5921e914 in make_child (s=0x563ebcececf0, slot=slot@entry=7, bucket=bucket@entry=0) at event.c:2691
#14 0x00007fcf5921f290 in perform_idle_server_maintenance (num_buckets=<optimized out>, child_bucket=<optimized out>) at event.c:2886
#15 server_main_loop (num_buckets=1, remaining_children_to_start=0) at event.c:3015
#16 event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at event.c:3092
#17 0x0000563ebc837ce0 in ap_run_mpm (pconf=0x563ebcea5a48, plog=0x563ebced2c68, s=0x563ebcececf0) at mpm_common.c:94
#18 0x0000563ebc821eb3 in main (argc=<optimized out>, argv=<optimized out>) at main.c:819
Would the following patch makes sense (against trunk)?
Index: server/mpm/event/event.c
===================================================================
--- server/mpm/event/event.c (revision 1882251)
+++ server/mpm/event/event.c (working copy)
@@ -2897,8 +2897,8 @@
* the other threads in the process needs to take us down
* (e.g., for MaxConnectionsPerChild) it will send us SIGTERM
*/
+ apr_signal(SIGTERM, dummy_signal_handler);
unblock_signal(SIGTERM);
- apr_signal(SIGTERM, dummy_signal_handler);
/* Watch for any messages from the parent over the POD */
while (1) {
rv = ap_mpm_podx_check(my_bucket->pod);
It looks like a queued SIG_TERM is delivered to the current SIG_TERM handler immediately after we unblocked it before we could
change the handler.
Regards
Rüdiger
On 9/30/20 3:59 PM, Yann Ylavic wrote:
> On Wed, Sep 30, 2020 at 1:40 PM <bu...@apache.org> wrote:
>>
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
>>
>> --- Comment #29 from Ruediger Pluem <rp...@apache.org> ---
>> There seem to be further ones:
>>
>> #0 0x00007fcf591f2664 in __do_global_dtors_aux () from
>> /lib64/libnss_files.so.2
>> [Current thread is 1 (Thread 0x7fcf59dc0900 (LWP 3563021))]
>> (gdb) bt
>> #0 0x00007fcf591f2664 in __do_global_dtors_aux () from
>> /lib64/libnss_files.so.2
>> #1 0x00007fcf5a30d2eb in _dl_fini () at dl-fini.c:138
>> #2 0x00007fcf59f0ee87 in __run_exit_handlers (status=status@entry=0,
>> listp=0x7fcf5a092578 <__exit_funcs>,
>> run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
>> at exit.c:108
>> #3 0x00007fcf59f0f040 in __GI_exit (status=status@entry=0) at exit.c:139
>> #4 0x00007fcf5921d716 in clean_child_exit (code=code@entry=0) at event.c:738
>> #5 0x00007fcf5921d73d in just_die (sig=<optimized out>) at event.c:743
>> #6 <signal handler called>
>
> I think this trace may be caused by OpenSSL shutdown in child
> processes, registered with atexit or alike (either cleanup code is
> called after the DOS is unloaded, or it points to data on pchild).
> We discussed this already in [1], I don't think we should let atexit
> code run for children, and just call _exit() in *nix MPMs.
>
> But if some threads are still "using" pchild in between
> apr_pool_destroy(pchild) and _exit() we might crash at child shutdown
> still (like in Joe's trace I think), so possibly we don't want to
> destroy pchild either for ungraceful shutdowns (graceful ones are OK
> because we wait for workers).
>
> [1] https://lists.apache.org/thread.html/16ae4b2ff5a52b1af320b081bde4dfb02e0c28dd1253572beb84dd72%40%3Cdev.httpd.apache.org%3E
>
>>
>> and
>>
>> #0 0x00007fcf59e0b5f0 in __do_global_dtors_aux () from /lib64/liblzma.so.5
>> #1 0x00007fcf5a30d2eb in _dl_fini () from /lib64/ld-linux-x86-64.so.2
>> #2 0x00007fcf59f0ee87 in __run_exit_handlers () from /lib64/libc.so.6
>> #3 0x00007fcf59f0f040 in exit () from /lib64/libc.so.6
>> #4 0x00007fcf5921d716 in clean_child_exit () from
>> /usr/lib64/httpd/modules/mod_mpm_event.so
>> #5 0x00007fcf5921d73d in just_die () from
>> /usr/lib64/httpd/modules/mod_mpm_event.so
>> #6 <signal handler called>
>
> Same case here possibly.
>
>>
>> Do we ever use liblzma in vanialla httpd?
>
> I don't think so,
>
>
> Regards;
> Yann.
>
Re: [Bug 64533] Http crashes observed during fuzzing testing
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Sep 30, 2020 at 1:40 PM <bu...@apache.org> wrote:
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
>
> --- Comment #29 from Ruediger Pluem <rp...@apache.org> ---
> There seem to be further ones:
>
> #0 0x00007fcf591f2664 in __do_global_dtors_aux () from
> /lib64/libnss_files.so.2
> [Current thread is 1 (Thread 0x7fcf59dc0900 (LWP 3563021))]
> (gdb) bt
> #0 0x00007fcf591f2664 in __do_global_dtors_aux () from
> /lib64/libnss_files.so.2
> #1 0x00007fcf5a30d2eb in _dl_fini () at dl-fini.c:138
> #2 0x00007fcf59f0ee87 in __run_exit_handlers (status=status@entry=0,
> listp=0x7fcf5a092578 <__exit_funcs>,
> run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
> at exit.c:108
> #3 0x00007fcf59f0f040 in __GI_exit (status=status@entry=0) at exit.c:139
> #4 0x00007fcf5921d716 in clean_child_exit (code=code@entry=0) at event.c:738
> #5 0x00007fcf5921d73d in just_die (sig=<optimized out>) at event.c:743
> #6 <signal handler called>
I think this trace may be caused by OpenSSL shutdown in child
processes, registered with atexit or alike (either cleanup code is
called after the DOS is unloaded, or it points to data on pchild).
We discussed this already in [1], I don't think we should let atexit
code run for children, and just call _exit() in *nix MPMs.
But if some threads are still "using" pchild in between
apr_pool_destroy(pchild) and _exit() we might crash at child shutdown
still (like in Joe's trace I think), so possibly we don't want to
destroy pchild either for ungraceful shutdowns (graceful ones are OK
because we wait for workers).
[1] https://lists.apache.org/thread.html/16ae4b2ff5a52b1af320b081bde4dfb02e0c28dd1253572beb84dd72%40%3Cdev.httpd.apache.org%3E
>
> and
>
> #0 0x00007fcf59e0b5f0 in __do_global_dtors_aux () from /lib64/liblzma.so.5
> #1 0x00007fcf5a30d2eb in _dl_fini () from /lib64/ld-linux-x86-64.so.2
> #2 0x00007fcf59f0ee87 in __run_exit_handlers () from /lib64/libc.so.6
> #3 0x00007fcf59f0f040 in exit () from /lib64/libc.so.6
> #4 0x00007fcf5921d716 in clean_child_exit () from
> /usr/lib64/httpd/modules/mod_mpm_event.so
> #5 0x00007fcf5921d73d in just_die () from
> /usr/lib64/httpd/modules/mod_mpm_event.so
> #6 <signal handler called>
Same case here possibly.
>
> Do we ever use liblzma in vanialla httpd?
I don't think so,
Regards;
Yann.
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #29 from Ruediger Pluem <rp...@apache.org> ---
There seem to be further ones:
#0 0x00007fcf591f2664 in __do_global_dtors_aux () from
/lib64/libnss_files.so.2
[Current thread is 1 (Thread 0x7fcf59dc0900 (LWP 3563021))]
(gdb) bt
#0 0x00007fcf591f2664 in __do_global_dtors_aux () from
/lib64/libnss_files.so.2
#1 0x00007fcf5a30d2eb in _dl_fini () at dl-fini.c:138
#2 0x00007fcf59f0ee87 in __run_exit_handlers (status=status@entry=0,
listp=0x7fcf5a092578 <__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
at exit.c:108
#3 0x00007fcf59f0f040 in __GI_exit (status=status@entry=0) at exit.c:139
#4 0x00007fcf5921d716 in clean_child_exit (code=code@entry=0) at event.c:738
#5 0x00007fcf5921d73d in just_die (sig=<optimized out>) at event.c:743
#6 <signal handler called>
#7 pthread_sigmask (how=how@entry=1, newmask=<optimized out>,
newmask@entry=0x7fff02b26230, oldmask=oldmask@entry=0x0)
at ../sysdeps/unix/sysv/linux/pthread_sigmask.c:48
#8 0x00007fcf5921ccd5 in unblock_signal (sig=sig@entry=15) at event.c:1264
#9 0x00007fcf5921e5d4 in child_main (child_num_arg=child_num_arg@entry=14,
child_bucket=child_bucket@entry=0) at event.c:2586
#10 0x00007fcf5921e914 in make_child (s=0x563ebcececf0, slot=slot@entry=14,
bucket=bucket@entry=0) at event.c:2691
#11 0x00007fcf5921f290 in perform_idle_server_maintenance
(num_buckets=<optimized out>, child_bucket=<optimized out>) at event.c:2886
#12 server_main_loop (num_buckets=1, remaining_children_to_start=0) at
event.c:3015
#13 event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>)
at event.c:3092
#14 0x0000563ebc837ce0 in ap_run_mpm (pconf=0x563ebcea5a48,
plog=0x563ebced2c68, s=0x563ebcececf0) at mpm_common.c:94
#15 0x0000563ebc821eb3 in main (argc=<optimized out>, argv=<optimized out>) at
main.c:819
and
#0 0x00007fcf59e0b5f0 in __do_global_dtors_aux () from /lib64/liblzma.so.5
#1 0x00007fcf5a30d2eb in _dl_fini () from /lib64/ld-linux-x86-64.so.2
#2 0x00007fcf59f0ee87 in __run_exit_handlers () from /lib64/libc.so.6
#3 0x00007fcf59f0f040 in exit () from /lib64/libc.so.6
#4 0x00007fcf5921d716 in clean_child_exit () from
/usr/lib64/httpd/modules/mod_mpm_event.so
#5 0x00007fcf5921d73d in just_die () from
/usr/lib64/httpd/modules/mod_mpm_event.so
#6 <signal handler called>
#7 0x00007fcf5a0b13cb in pthread_sigmask () from /lib64/libpthread.so.0
#8 0x00007fcf5921ccd5 in unblock_signal () from
/usr/lib64/httpd/modules/mod_mpm_event.so
#9 0x00007fcf5921e5d4 in child_main () from
/usr/lib64/httpd/modules/mod_mpm_event.so
#10 0x00007fcf5921e914 in make_child () from
/usr/lib64/httpd/modules/mod_mpm_event.so
#11 0x00007fcf5921f290 in event_run () from
/usr/lib64/httpd/modules/mod_mpm_event.so
#12 0x0000563ebc837ce0 in ap_run_mpm ()
#13 0x0000563ebc821eb3 in main ()
Looks like crashes in library shutdown handlers when httpd is stopped.
Do we ever use liblzma in vanialla httpd?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #17 from Ruediger Pluem <rp...@apache.org> ---
Are you able to reproduce the same issue with a different Linux distribution?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #15 from Ruediger Pluem <rp...@apache.org> ---
Unfortunateley the backtraces are inclomplete and do not tell us where the
crash happens. I don't know why the instructions on
http://httpd.apache.org/dev/debugging.html#crashes do not deliver correct
backtraces. It is likely related to your Linux setup. Which Linux distro are
you using?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64533] Http crashes observed during fuzzing testing
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64533
--- Comment #4 from wei-mark.zheng@nokia-sbell.com <we...@nokia-sbell.com> ---
Created attachment 37323
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37323&action=edit
backtrace
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org