You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Arthur Ramsey <ar...@mediture.com> on 2016/07/25 15:53:52 UTC
[users@httpd] Apache 2.4.12+ on Windows x64 stops responding to requests
Hello,
I'm using Apache as a proxy to AJP, HTTP and HTTPS services. I have an
issue with Apache 2.4.12+ on Windows Server 2008 R2 where it
occasionally stops responding to requests (can't open socket) unless
they originate from the loopback adapter, which is resolved by
restarting the Apache service. I can see it still listening on
0.0.0.0:443 and I've got the Windows firewall disabled. I searched the
bug tracker and mailing list, but didn't find anyone with a similar
issue. I've got a decent amount of experience with Apache 2.2 on
Windows and never saw an issue like this. I see nothing in the error
log when this happens. I cannot provide a core dump because it may
contain sensitive data. If I knew how I would provide a thread dump.
I've tried reproducing the issue with various load tests and cannot.
I'm using the Apache Lounge VC14 build. Would I be better off reporting
this to them? I needed some 2.4 features, so I used 2.4.12, which was
the latest at the time. I tried upgrading to 2.4.20 and I'm seeing the
same problem. I also enabled HTTP/2 when I upgraded to 2.4.20 because I
hoped it was an APR issue and because HTTP/2 is nice to have.
Thanks,
Arthur
This e-mail and any attachments may contain CONFIDENTIAL information, including PROTECTED HEALTH INFORMATION. If you are not the intended recipient, any use or disclosure of this information is STRICTLY PROHIBITED; you are requested to delete this e-mail and any attachments, notify the sender immediately, and notify the Mediture Privacy Officer at privacyofficer@mediture.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.4.12+ on Windows x64 stops responding to
requests
Posted by Paul Spangler <pa...@ni.com>.
On 8/9/2016 12:07 PM, Jacob Champion wrote:
> At this point, my primary suspect is our use of recycled OVERLAPPED
> structs without reinitializing them to zero. To make matters worse,
> we're setting the OVERLAPPED's internal .Pointer field in the
> AcceptFilter 'data' case -- which we're not supposed to be doing to
> begin with [1]. We don't do that in the 'connect' filter.
>
> This is all just theorycrafting, though. I'll try to reproduce on my end
> too.
>
> --Jacob
>
> [1]
> https://msdn.microsoft.com/en-us/library/windows/desktop/ms684342(v=vs.85).aspx
> (the Members > Pointer section)
>
I think I've finally had some success finding a reproduction of this
issue, though it's somewhat involved. I set up an instance of Apache
2.4.16 64-bit (built from source) on a Windows 7 machine and spun up an
instance of WANem (http://wanem.sourceforge.net/) in a VirtualBox VM
hosted on my client machine (also Windows 7).
WANem configuration (Advanced Mode):
Bandwidth - 100Mbps
Random Disconnect Type - tcp-reset
Random Disconnect MTTF Low - 1
Random Disconnect MTTF High - 3
Random Disconnect MTTR Low - 0
Random Disconnect MTTR High - 0
This instructs WANem to inject a TCP RST into connections that pass
through it every 1 to 3 seconds (then recover after 0 seconds).
Then on my client machine, I added a route to the server that passes
through the WANem gateway (cmd prompt: ROUTE ADD <server-ip> <WANem-ip>).
Finally, I ran a program on the client that makes 10 cURL requests in
parallel repeatedly, performing a GET on a simple index.html page (well,
a 28 KB HTML page). Eventually, even requests made to localhost on the
server machine stop responding (they hang until the client times out).
Nothing shows in the error logs (I tried up to debug verbosity), and
once it reproduces, no more entries appear in the access logs. I have to
restart the server, though I haven't tried letting it sit for a period
of time to see if it recovers on its own.
When I do the whole process again with "AcceptFilter http connect", it
does not reproduce, and requests continue to work (when not being reset
by WANem).
Not easy to set up, but at least it doesn't involve a browser or
specific content on the server. I've seen it reproduce almost
immediately, but it usually does so within 10 seconds or so.
I'll see if Wireshark shows anything interesting going on around the RSTs.
--
Paul Spangler
LabVIEW R&D
National Instruments
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.4.12+ on Windows x64 stops responding to
requests
Posted by Jacob Champion <ch...@gmail.com>.
On 08/09/2016 07:22 AM, Paul Spangler wrote:
> Though in our case, we only needed to use
>
> AcceptFilter http connect
> AcceptFilter https connect
>
> rather than turning it off completely using "none". Setting it to
> connect allows the server to recycle sockets.
>
> I'll see if I can't look back into it and try to find a reproduction
> case again that I can narrow down.
Thanks Paul!
At this point, my primary suspect is our use of recycled OVERLAPPED
structs without reinitializing them to zero. To make matters worse,
we're setting the OVERLAPPED's internal .Pointer field in the
AcceptFilter 'data' case -- which we're not supposed to be doing to
begin with [1]. We don't do that in the 'connect' filter.
This is all just theorycrafting, though. I'll try to reproduce on my end
too.
--Jacob
[1]
https://msdn.microsoft.com/en-us/library/windows/desktop/ms684342(v=vs.85).aspx
(the Members > Pointer section)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.4.12+ on Windows x64 stops responding to
requests
Posted by Paul Spangler <pa...@ni.com>.
On 8/8/2016 6:14 PM, Jacob Champion wrote:
> On 07/25/2016 11:13 AM, Arthur Ramsey wrote:
>> I think I will try the following settings first, but failing that I'll
>> give the x86 build a try.
>>
>> AcceptFilter https none
>
> Any follow-up on this? I've been digging into the AcceptEx()
> implementation, since it looks like there have been intermittent
> problems with it for a while now. If you have a reproduction case, I'd
> be happy to run with it.
>
> --Jacob
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
We've also seen this in our 64-bit Apache 2.4.16 build on Windows.
Specifically, accessing/refreshing a particular page in Chrome would
lead to a 10-20 second hang for all clients, Firefox would maybe lead to
a 2-5 second hang sometimes, but IE had no effect. The page had a
mixture of static files (mainly JavaScript) and dynamic content loaded
via Ajax. I was able to determine from the access log that the actual
request processing took a normal amount of time, so the extra time must
have occurred before the request start time was recorded (i.e. during
accept/reading the request line).
Though in our case, we only needed to use
AcceptFilter http connect
AcceptFilter https connect
rather than turning it off completely using "none". Setting it to
connect allows the server to recycle sockets.
I'll see if I can't look back into it and try to find a reproduction
case again that I can narrow down.
--
Paul Spangler
LabVIEW R&D
National Instruments
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.4.12+ on Windows x64 stops responding to
requests
Posted by Jacob Champion <ch...@gmail.com>.
On 07/25/2016 11:13 AM, Arthur Ramsey wrote:
> I think I will try the following settings first, but failing that I'll
> give the x86 build a try.
>
> AcceptFilter https none
Any follow-up on this? I've been digging into the AcceptEx()
implementation, since it looks like there have been intermittent
problems with it for a while now. If you have a reproduction case, I'd
be happy to run with it.
--Jacob
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.4.12+ on Windows x64 stops responding to
requests
Posted by Arthur Ramsey <ar...@mediture.com>.
I think I will try the following settings first, but failing that I'll
give the x86 build a try.|
AcceptFilter https none|
On 07/25/2016 12:50 PM, Yehuda Katz wrote:
> Can you try the x86 build?
>
> - Y
>
> On Mon, Jul 25, 2016 at 12:34 PM, Arthur Ramsey
> <arthur_ramsey@mediture.com <ma...@mediture.com>> wrote:
>
> I'm using the x64 build.
>
>
> On 07/25/2016 11:33 AM, Yehuda Katz wrote:
>>
>> Are you using a 32-bit or 64-bit build?
>> I had a similar problem with the 64-bit build and when I switch
>> to the 32-bit build, it worked fine.
>>
>> - Y
>>
>>
>> On Jul 25, 2016 11:54 AM, "Arthur Ramsey"
>> <arthur_ramsey@mediture.com <ma...@mediture.com>>
>> wrote:
>>
>> Hello,
>>
>> I'm using Apache as a proxy to AJP, HTTP and HTTPS services.
>> I have an issue with Apache 2.4.12+ on Windows Server 2008 R2
>> where it occasionally stops responding to requests (can't
>> open socket) unless they originate from the loopback adapter,
>> which is resolved by restarting the Apache service. I can
>> see it still listening on 0.0.0.0:443 <http://0.0.0.0:443>
>> and I've got the Windows firewall disabled. I searched the
>> bug tracker and mailing list, but didn't find anyone with a
>> similar issue. I've got a decent amount of experience with
>> Apache 2.2 on Windows and never saw an issue like this. I
>> see nothing in the error log when this happens. I cannot
>> provide a core dump because it may contain sensitive data.
>> If I knew how I would provide a thread dump.
>>
>> I've tried reproducing the issue with various load tests and
>> cannot.
>>
>> I'm using the Apache Lounge VC14 build. Would I be better
>> off reporting this to them? I needed some 2.4 features, so I
>> used 2.4.12, which was the latest at the time. I tried
>> upgrading to 2.4.20 and I'm seeing the same problem. I also
>> enabled HTTP/2 when I upgraded to 2.4.20 because I hoped it
>> was an APR issue and because HTTP/2 is nice to have.
>>
>> Thanks,
>> Arthur
>>
>> This e-mail and any attachments may contain CONFIDENTIAL
>> information, including PROTECTED HEALTH INFORMATION. If you
>> are not the intended recipient, any use or disclosure of this
>> information is STRICTLY PROHIBITED; you are requested to
>> delete this e-mail and any attachments, notify the sender
>> immediately, and notify the Mediture Privacy Officer at
>> privacyofficer@mediture.com <ma...@mediture.com>.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> <ma...@httpd.apache.org>
>> For additional commands, e-mail: users-help@httpd.apache.org
>> <ma...@httpd.apache.org>
>>
>
> --
> Arthur Ramsey
> System Administrator
> Mediture
> arthur_ramsey@mediture.com <ma...@mediture.com>
> 952.400.0323 <tel:952.400.0323>
>
>
> This e-mail and any attachments may contain CONFIDENTIAL
> information, including PROTECTED HEALTH INFORMATION. If you are
> not the intended recipient, any use or disclosure of this
> information is STRICTLY PROHIBITED; you are requested to delete
> this e-mail and any attachments, notify the sender immediately,
> and notify the Mediture Privacy Officer at
> privacyofficer@mediture.com <ma...@mediture.com>.
>
>
>
--
Arthur Ramsey
System Administrator
Mediture
arthur_ramsey@mediture.com
952.400.0323
This e-mail and any attachments may contain CONFIDENTIAL information, including PROTECTED HEALTH INFORMATION. If you are not the intended recipient, any use or disclosure of this information is STRICTLY PROHIBITED; you are requested to delete this e-mail and any attachments, notify the sender immediately, and notify the Mediture Privacy Officer at privacyofficer@mediture.com.
Re: [users@httpd] Apache 2.4.12+ on Windows x64 stops responding to requests
Posted by Yehuda Katz <ye...@ymkatz.net>.
Can you try the x86 build?
- Y
On Mon, Jul 25, 2016 at 12:34 PM, Arthur Ramsey <ar...@mediture.com>
wrote:
> I'm using the x64 build.
>
>
> On 07/25/2016 11:33 AM, Yehuda Katz wrote:
>
> Are you using a 32-bit or 64-bit build?
> I had a similar problem with the 64-bit build and when I switch to the
> 32-bit build, it worked fine.
>
> - Y
>
> On Jul 25, 2016 11:54 AM, "Arthur Ramsey" <ar...@mediture.com>
> wrote:
>
>> Hello,
>>
>> I'm using Apache as a proxy to AJP, HTTP and HTTPS services. I have an
>> issue with Apache 2.4.12+ on Windows Server 2008 R2 where it occasionally
>> stops responding to requests (can't open socket) unless they originate from
>> the loopback adapter, which is resolved by restarting the Apache service.
>> I can see it still listening on 0.0.0.0:443 and I've got the Windows
>> firewall disabled. I searched the bug tracker and mailing list, but didn't
>> find anyone with a similar issue. I've got a decent amount of experience
>> with Apache 2.2 on Windows and never saw an issue like this. I see nothing
>> in the error log when this happens. I cannot provide a core dump because
>> it may contain sensitive data. If I knew how I would provide a thread dump.
>>
>> I've tried reproducing the issue with various load tests and cannot.
>>
>> I'm using the Apache Lounge VC14 build. Would I be better off reporting
>> this to them? I needed some 2.4 features, so I used 2.4.12, which was the
>> latest at the time. I tried upgrading to 2.4.20 and I'm seeing the same
>> problem. I also enabled HTTP/2 when I upgraded to 2.4.20 because I hoped
>> it was an APR issue and because HTTP/2 is nice to have.
>>
>> Thanks,
>> Arthur
>>
>> This e-mail and any attachments may contain CONFIDENTIAL information,
>> including PROTECTED HEALTH INFORMATION. If you are not the intended
>> recipient, any use or disclosure of this information is STRICTLY
>> PROHIBITED; you are requested to delete this e-mail and any attachments,
>> notify the sender immediately, and notify the Mediture Privacy Officer at
>> <pr...@mediture.com>privacyofficer@mediture.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
> --
> Arthur Ramsey
> System Administrator
> Mediturearthur_ramsey@mediture.com952.400.0323
>
>
> This e-mail and any attachments may contain CONFIDENTIAL information,
> including PROTECTED HEALTH INFORMATION. If you are not the intended
> recipient, any use or disclosure of this information is STRICTLY
> PROHIBITED; you are requested to delete this e-mail and any attachments,
> notify the sender immediately, and notify the Mediture Privacy Officer at
> privacyofficer@mediture.com.
>
>
Re: [users@httpd] Apache 2.4.12+ on Windows x64 stops responding to
requests
Posted by Kev <s7...@yahoo.co.uk.INVALID>.
Hi Arthur
We have experienced similar problems in the past.
Check this site as the solution might work for you:http://superuser.com/questions/516030/apache-2-4-on-windows-responds-slowly-hangs-when-serving-some-dynamic-pages
RegardsKev
On Monday, 25 July 2016, 17:34, Arthur Ramsey <ar...@mediture.com> wrote:
I'm using the x64 build.
On 07/25/2016 11:33 AM, Yehuda Katz wrote:
Are you using a 32-bit or 64-bit build?
I had a similar problem with the 64-bit build and when I switch to the 32-bit build, it worked fine. - Y
On Jul 25, 2016 11:54 AM, "Arthur Ramsey" <ar...@mediture.com> wrote:
Hello,
I'm using Apache as a proxy to AJP, HTTP and HTTPS services. I have an issue with Apache 2.4.12+ on Windows Server 2008 R2 where it occasionally stops responding to requests (can't open socket) unless they originate from the loopback adapter, which is resolved by restarting the Apache service. I can see it still listening on 0.0.0.0:443 and I've got the Windows firewall disabled. I searched the bug tracker and mailing list, but didn't find anyone with a similar issue. I've got a decent amount of experience with Apache 2.2 on Windows and never saw an issue like this. I see nothing in the error log when this happens. I cannot provide a core dump because it may contain sensitive data. If I knew how I would provide a thread dump.
I've tried reproducing the issue with various load tests and cannot.
I'm using the Apache Lounge VC14 build. Would I be better off reporting this to them? I needed some 2.4 features, so I used 2.4.12, which was the latest at the time. I tried upgrading to 2.4.20 and I'm seeing the same problem. I also enabled HTTP/2 when I upgraded to 2.4.20 because I hoped it was an APR issue and because HTTP/2 is nice to have.
Thanks,
Arthur
This e-mail and any attachments may contain CONFIDENTIAL information, including PROTECTED HEALTH INFORMATION. If you are not the intended recipient, any use or disclosure of this information is STRICTLY PROHIBITED; you are requested to delete this e-mail and any attachments, notify the sender immediately, and notify the Mediture Privacy Officer at privacyofficer@mediture.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
--
Arthur Ramsey
System Administrator
Mediture
arthur_ramsey@mediture.com
952.400.0323
This e-mail and any attachments may contain CONFIDENTIAL information, including PROTECTED HEALTH INFORMATION. If you are not the intended recipient, any use or disclosure of this information is STRICTLY PROHIBITED; you are requested to delete this e-mail and any attachments, notify the sender immediately, and notify the Mediture Privacy Officer at privacyofficer@mediture.com.
Re: [users@httpd] Apache 2.4.12+ on Windows x64 stops responding to
requests
Posted by Arthur Ramsey <ar...@mediture.com>.
I'm using the x64 build.
On 07/25/2016 11:33 AM, Yehuda Katz wrote:
>
> Are you using a 32-bit or 64-bit build?
> I had a similar problem with the 64-bit build and when I switch to the
> 32-bit build, it worked fine.
>
> - Y
>
>
> On Jul 25, 2016 11:54 AM, "Arthur Ramsey" <arthur_ramsey@mediture.com
> <ma...@mediture.com>> wrote:
>
> Hello,
>
> I'm using Apache as a proxy to AJP, HTTP and HTTPS services. I
> have an issue with Apache 2.4.12+ on Windows Server 2008 R2 where
> it occasionally stops responding to requests (can't open socket)
> unless they originate from the loopback adapter, which is resolved
> by restarting the Apache service. I can see it still listening on
> 0.0.0.0:443 <http://0.0.0.0:443> and I've got the Windows firewall
> disabled. I searched the bug tracker and mailing list, but didn't
> find anyone with a similar issue. I've got a decent amount of
> experience with Apache 2.2 on Windows and never saw an issue like
> this. I see nothing in the error log when this happens. I cannot
> provide a core dump because it may contain sensitive data. If I
> knew how I would provide a thread dump.
>
> I've tried reproducing the issue with various load tests and cannot.
>
> I'm using the Apache Lounge VC14 build. Would I be better off
> reporting this to them? I needed some 2.4 features, so I used
> 2.4.12, which was the latest at the time. I tried upgrading to
> 2.4.20 and I'm seeing the same problem. I also enabled HTTP/2
> when I upgraded to 2.4.20 because I hoped it was an APR issue and
> because HTTP/2 is nice to have.
>
> Thanks,
> Arthur
>
> This e-mail and any attachments may contain CONFIDENTIAL
> information, including PROTECTED HEALTH INFORMATION. If you are
> not the intended recipient, any use or disclosure of this
> information is STRICTLY PROHIBITED; you are requested to delete
> this e-mail and any attachments, notify the sender immediately,
> and notify the Mediture Privacy Officer at
> privacyofficer@mediture.com <ma...@mediture.com>.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> <ma...@httpd.apache.org>
> For additional commands, e-mail: users-help@httpd.apache.org
> <ma...@httpd.apache.org>
>
--
Arthur Ramsey
System Administrator
Mediture
arthur_ramsey@mediture.com
952.400.0323
This e-mail and any attachments may contain CONFIDENTIAL information, including PROTECTED HEALTH INFORMATION. If you are not the intended recipient, any use or disclosure of this information is STRICTLY PROHIBITED; you are requested to delete this e-mail and any attachments, notify the sender immediately, and notify the Mediture Privacy Officer at privacyofficer@mediture.com.
Re: [users@httpd] Apache 2.4.12+ on Windows x64 stops responding to requests
Posted by Yehuda Katz <ye...@ymkatz.net>.
Are you using a 32-bit or 64-bit build?
I had a similar problem with the 64-bit build and when I switch to the
32-bit build, it worked fine.
- Y
On Jul 25, 2016 11:54 AM, "Arthur Ramsey" <ar...@mediture.com>
wrote:
> Hello,
>
> I'm using Apache as a proxy to AJP, HTTP and HTTPS services. I have an
> issue with Apache 2.4.12+ on Windows Server 2008 R2 where it occasionally
> stops responding to requests (can't open socket) unless they originate from
> the loopback adapter, which is resolved by restarting the Apache service.
> I can see it still listening on 0.0.0.0:443 and I've got the Windows
> firewall disabled. I searched the bug tracker and mailing list, but didn't
> find anyone with a similar issue. I've got a decent amount of experience
> with Apache 2.2 on Windows and never saw an issue like this. I see nothing
> in the error log when this happens. I cannot provide a core dump because
> it may contain sensitive data. If I knew how I would provide a thread dump.
>
> I've tried reproducing the issue with various load tests and cannot.
>
> I'm using the Apache Lounge VC14 build. Would I be better off reporting
> this to them? I needed some 2.4 features, so I used 2.4.12, which was the
> latest at the time. I tried upgrading to 2.4.20 and I'm seeing the same
> problem. I also enabled HTTP/2 when I upgraded to 2.4.20 because I hoped
> it was an APR issue and because HTTP/2 is nice to have.
>
> Thanks,
> Arthur
>
> This e-mail and any attachments may contain CONFIDENTIAL information,
> including PROTECTED HEALTH INFORMATION. If you are not the intended
> recipient, any use or disclosure of this information is STRICTLY
> PROHIBITED; you are requested to delete this e-mail and any attachments,
> notify the sender immediately, and notify the Mediture Privacy Officer at
> privacyofficer@mediture.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>