You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2019/01/22 12:29:06 UTC
[Bug 63098] New: Use after free errors when fuzzing against Apache
httpd using an APR pool debug build
https://bz.apache.org/bugzilla/show_bug.cgi?id=63098
Bug ID: 63098
Summary: Use after free errors when fuzzing against Apache
httpd using an APR pool debug build
Product: Apache httpd-2
Version: 2.4.37
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P2
Component: All
Assignee: bugs@httpd.apache.org
Reporter: rainer.jung@kippdata.de
Target Milestone: ---
A reporter uncovered use after free issues in the Apache HTTP Server triggered
by a fuzzing tool when APR pool debugging is in use. Pool debugging is not
enabled by default, and should not be used in production environments. The bugs
do not get triggered in a default build and we would not classify them as a
"zero day vulnerability".
The reporter has provided additional info publicly under:
https://github.com/hannob/apache-uaf/.
Daniel Gruno from the web server PMC has looked at the issue extensively, and
with tens of millions of tests against httpd versions 2.4.29 through 2.4.38 (he
reworked Hanno's code into a threaded python test) with no pool debugging.
Those tests did not show any problems. We therefore believe the problem to not
occur without pool debugging.
With pool debugging most of the occurrences happen when HTTP/2 is used via
mod_http2. Those problems are, at least partially, understood. There are
discussion between the web server and the APR project on how to best fix them.
As already noted, pool debugging is not meant for production use.
Another type of error also contained in the same original report, an assert in
__pthread_tpp_change_priority pointing at a pthread mutex problem, was analyzed
and fixed. We are not aware of a successful reproduction of that issue with
2.4.37 (or later).
Unfortunately the OpenBSD platform uses APR pool debugging by default. Due to
another bug, these builds would crash when using HTTP/2, so we do currently not
expect them to experience the majority of the use after free issues. We are in
contact with OpenBSD maintainers on how to best resolve the situation.
The Apache HTTP Server is a volunteer project, where people work on the project
as time and ability allows, and as such, we do triage problems based on some
simple rules;
- Is the issue reproducible in standard (production) systems, or is it
solely pertaining to debug/testing scenarios?
- Which version are affected? (old versus new, stable versions)
- Does the potential impact merit an immediate fix as CVE, or is it a bug
added to the list of things to do at a slower pace?
We examined the report closely, and determined that we needed more information
from the reporter on a reproducible vector here, as we were unable to, in any
way, reliably reproduce the issue on standard systems. As a response was not
received for several months, and as we deemed this did not impact stability or
security to a degree that merited concern, we have been in a standard holding
pattern, waiting for further feedback from the reporter, while simultaneously
looking closer at the issue, seeing if we ourselves could find a potential
exploit. We have of course reached out to the APR project to see if they can
help address the pool issue, and are working with them to further investigate
possible solutions.
Furthermore, we would like to point out that the 11 year old Bugzilla report
referenced deals with mod_php in the mix and did not have the requested stack
trace needed for further investigation, and as such, cannot be reliably
compared to the current issues. In addition the part of the new report that
shows the same symptom as the old Bugzilla report, has already been analyzed
and fixed.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 63098] Use after free errors when fuzzing against Apache httpd
using an APR pool debug build
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63098
--- Comment #1 from Rainer Jung <ra...@kippdata.de> ---
Additional current info from Stefan Sperling about the OpenBSD situation:
FYI, I have disabled pool debugging in OpenBSD's port of APR. We are now using
Yann's patch to force the default allocator to call free(3) when APR pools are
cleared: https://marc.info/?l=openbsd-ports-cvs&m=154815812713288&w=2
This change only affects OpenBSD -current. I do not plan to backport a patch to
the OpenBSD 6.4 release. We have had no reports indicating that http2 was
crashing on OpenBSD. The likely reason is that nobody is actually running such
a setup. If people intend to run such a setup, they should use -current for
now, or wait until OpenBSD 6.5 is released.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 63098] Use after free errors when fuzzing against Apache httpd
using an APR pool debug build
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63098
hanno@hboeck.de changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |hanno@hboeck.de
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 63098] Use after free errors when fuzzing against Apache httpd
using an APR pool debug build
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63098
--- Comment #5 from hanno@hboeck.de ---
(In reply to hanno from comment #4)
> * With this "MaxMemFree 1" option set and *without* the pool debugger I now
> get ASAN uaf warnings at a much faster and more reliable rate than ever
> before.
I have to retract that statement.
This was a fluke, because I had apr built without the pool debugger, but
apr-util was built while the apr pool debugger was still on. This caused the
errors to pop up quickly.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 63098] Use after free errors when fuzzing against Apache httpd
using an APR pool debug build
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63098
--- Comment #4 from hanno@hboeck.de ---
(In reply to Ruediger Pluem from comment #3)
> Try running with
>
> MaxMemFree 1
>
> in httpd.conf instead of using pool debugging. This should give you the same
> malloc / free behaviour as with pool debugging as needed for ASAN, but uses
> the non debugging code.
Okay, so this gives some interesting new insights:
* With this "MaxMemFree 1" option set and *without* the pool debugger I now get
ASAN uaf warnings at a much faster and more reliable rate than ever before.
* I however don't get the pthread Assertion warning.
As for whether the pthread assertion warning can be reproduced without the pool
debugger: Craig confirmed to me that he was able to get that, however with a
previous version of apache. We mailed you that in a mail in June.
So for the pthread assertion it may very well be that we're facing two separate
issues here: One that is fixed in the latest version and shows up without pool
debugging and one that is not fixed, but only happens with pool debugging.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 63098] Use after free errors when fuzzing against Apache httpd
using an APR pool debug build
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63098
Craig <ca...@integralblue.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |candrews@integralblue.com
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 63098] Use after free errors when fuzzing against Apache httpd
using an APR pool debug build
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63098
--- Comment #2 from hanno@hboeck.de ---
As "the reporter" I'd like to add two notes:
* The __pthread_tpp_change_priority definitely shows up for me in apache
2.4.37.
* My worries with the pool debugging are not that people may run it in
production. (Even though it seems OpenBSD does that.) My worries are primarily
that without pool debugging it's very difficult to detect memory safety bugs,
as the pool allocator will hide bugs from tools like ASAN.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 63098] Use after free errors when fuzzing against Apache httpd
using an APR pool debug build
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63098
--- Comment #3 from Ruediger Pluem <rp...@apache.org> ---
(In reply to hanno from comment #2)
> As "the reporter" I'd like to add two notes:
>
> * The __pthread_tpp_change_priority definitely shows up for me in apache
> 2.4.37.
>
> * My worries with the pool debugging are not that people may run it in
> production. (Even though it seems OpenBSD does that.) My worries are
> primarily that without pool debugging it's very difficult to detect memory
> safety bugs, as the pool allocator will hide bugs from tools like ASAN.
Try running with
MaxMemFree 1
in httpd.conf instead of using pool debugging. This should give you the same
malloc / free behaviour as with pool debugging as needed for ASAN, but uses the
non debugging code.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org