You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by "Feldhacker, Chris" <Fe...@principal.com> on 2011/03/09 22:15:28 UTC
[users@httpd] Slow performance of Apache on Windows
Summary:
Apache HTTP Server 2.2.17 running on Windows Server 2003 & 2008 has slight HTTP transfer performance problems with Windows XP clients, and *extreme* performance problems with Windows 7 clients.
Apache HTTP Server running on RedHat Linux 5 performs fine with both XP and W7 clients.
IIS and Small HTTP Server on Windows 2003/2008 perform fine with both XP and W7 clients.
The purpose of this email is simply to report our findings for the benefit of anyone else that may have encountered similar problems. (And if anyone has the ability or desire to dive in and test other clients or scenarios, all the better. Ideally, it would be great if others could reproduce and confirm the findings so that this ultimately gets investigated and resolved with the Apache HTTP server for Windows.)
I know, I know, Apache server can't possibly have problems running on the Windows platform that causes different clients to perform differently -- it must be something else. Depending on your perspective and the tests run, the evidence will point in different directions and send you in circles. We're just reporting our findings.
Background:
We initially noticed problems using IBM Installation Manager which downloads several GBs of files from our internal HTTP server (using Apache on Windows Server 2003) in order to install/update various IBM products. With XP clients, transfers were fairly fast at 4000-5000KB/s. With Win7 clients, transfers were pretty slow at 350-1000KB/s. HTTP downloads from the internet seem normal, it seems the problem only occurred when using HTTP against an internal server. We were able to reproduce the problem using Curl as the HTTP client, so HTTP client didn't seem to be the problem.
Obviously the problem is with the client OS. Yes, there are widespread reports of network performance problems with Vista and Windows 7. Microsoft docs indicate the most common cause is problematic network devices that don't handle TCP window scaling or other RFC 1323 options very well, so there are various workarounds to adjust the TCP settings, most noteably the auto-tuning level. Network traces confirmed that window scaling and other 1323 flags were indeed present and being preserved between the client and the server. Nonetheless, we fully explored all the Windows 7 TCP global parameters: rss, chimney offloading, dma, dca, auto-tuning, congestion control provider, ecn, timestamps. Nothing resolved the issue.
Network traces taken in an isolated testing lab with desktop clients running on identical hardware on a 100mbit network also seemed to point to problems with the client OS: after the server sends a window of packets (44) and the client dutifully acknowledges every other one, when the last packet is sent with the push flag, the Windows 7 client takes between 100-300ms longer to send the acknowledgement compared to the Windows XP client. (Hmm...200ms sounds like an issue with selective acknowledgements/nagle's algorithm/TCP_NO_DELAY -- nope, explored that too.)
Again, obviously a problem with the client. Maybe it's a 64/32-bit issue (nope), maybe it's a virusscanner or firewall issue (nope and nope), maybe results would be different if we installed W7 SP1 (nope), removed the QoS provider (nope), or disabled IPv6 (nope). Thinking it could be a network driver issue we made sure ALL drivers were up-to-date, and verified the driver settings were the same between the XP and W7 machines (all the same, W7 has more options but nothing we tried changing improved performance). Monitoring the port on the network switch, we could confirm that duplexing mismatch was not an issue. Oh, we also disabled some RDC feature that's present in Windows 7, no luck.
It was still odd that the problem only seemed to occur from our internal server, HTTP downloads from the internet seemed fine. Out of curiosity we setup a Windows Server 2008 running Apache, and a RedHat Linux server running Apache. The performance of the 2008 server was the same as the 2003 server, but the performance of the Linux server was double that of the Windows machines (10-11MB/s)! In fact, not only did the Windows 7 client maintain 10-11MB/s, but the Windows XP client had the same 10-11MB/s performance as well!
Obviously it's a problem with the server. Since Apache was being used across all platforms, we initially looked at the OS. Again, we tried changing various registry changes to tweak the TCP settings (TcpWindowSize both above and below 64K, Tcp1323Opts (0,1,3), SackOpts, RSS, ICMPRedirect, etc.) -- pretty much everything we could find to no avail. When we setup a file share and copied a large file from the Windows server to the clients directly using Explorer, both the XP and W7 clients performed about the same (roughly 8MB/s). We installed the Test TCP (TTCP) tool on the Windows servers to test the raw TCP traffic and both the XP and W7 clients obtained 10-11MB/s.
We were about to open a ticket with Microsoft and see what they had to say, but we wanted to get to a state where only Microsoft products were involved -- so Microsoft could only blame themselves. So, we already experienced this problem with different HTTP clients (IBM IM, curl), but we went ahead and reproduced the problem with Wget and Firefox and IE8. (Now, performance does vary slightly between clients, but no HTTP client climbs above 1MB/s or so -- certainly far short of the 10-11MB/s that we know is possible!) But, at least the problem still occurs with Microsoft IE as the HTTP client.
Next, we installed IIS on the server. HTTP performance with both XP and W7 clients was now 10-11MB/s, but if we hit against Apache we were still getting 4MB/s with XP and at most 1MB/s with W7! Surely our Apache HTTP server must be setup wrong. But, we just had a default install of Apache HTTP 2.2.17 with absolutely no changes at all. It's quite odd that the default install should perform so poorly, but we started stripping down the config file to make it was as bare-bones as possible -- removed as many modules and directives as we could; after all, we were just providing a direct URL to a large file and just wanted it to transfer, nothing more. We've tried the platform specific options (EnableSendfile Off, EnableMMAP Off, Win32DisableAcceptEx), and tweaking the settings for the number of clients, threads, or age doesn't matter since we're just testing one client and one request. No luck.
Of course IIS performs better on Windows than Apache, Microsoft must have tweaked it or optimized it to perform better, right? The number of alternative HTTP servers for Windows appears to be quite small, and the only one that I found that I could install and setup quickly was "Small HTTP Server". Using Small HTTP Server, HTTP performance with both XP and W7 clients was still 10-11MB/s! I guess the deck really isn't stacked -- if Small HTTP Server can achieve 10-11MB/s, then shouldn't Apache be able to achieve this as well?
End result: We have a Windows 2003 Server with Apache HTTP Server, IIS, and Small HTTP Server all installed and running. If a Windows XP client hits IIS or SHTTPS they get 10-11MB/s, but if they hit Apache it gets 4MB/s. If a Windows 7 client hits IIS or SHTTPS it also gets 10-11MB/s, but if it hits Apache it gets AT MOST 1MB/s.
Conclusion: Apache HTTP Server on Windows exhibits performance issues with certain client OSes -- performance is 2 times as slow for XP clients, and 10-30 times slower for W7 clients. I have no explanation as to why. (And I completely acknowledge that the behavior observed doesn't even make sense, but it is what it is!) It seems our Apache/Windows server has had performance issues this whole time we've been using it and didn't realize it with our XP clients, it wasn't until we started testing with Windows 7 clients that the performance problems seem to be amplified.
In our mind we've resolved our issue -- use IIS on Windows Server instead of Apache, or switch to Apache on Linux. Exactly why Apache/Windows has performance issues is unresolved and a lot more investigation could be done, namely testing with additional clients besides just Windows, more detailed network tracing and analysis, or OS/Apache debugging and troubleshooting.
I apologize for the novel, but depending on where you start, what results you focus on, and what biases you carry, the evidence will lead you in different directions. There are definitely "known" performance issues in the network, client and server OSes that had to be eliminated as possible causes first. You really have to get the complete story or spin in circles a few times on your own before the conclusion becomes clear.
Hopefully the "end result" above can serve as simple "steps to reproduce" and allow other users to reproduce and confirm the problem quickly and easily.
I hope somebody else will someday gain some benefit from these findings.
-----Message Disclaimer-----
This e-mail message is intended only for the use of the individual or
entity to which it is addressed, and may contain information that is
privileged, confidential and exempt from disclosure under applicable law.
If you are not the intended recipient, any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify us immediately by
reply email to Connect@principal.com and delete or destroy all copies of
the original message and attachments thereto. Email sent to or from the
Principal Financial Group or any of its member companies may be retained
as required by law or regulation.
Nothing in this message is intended to constitute an Electronic signature
for purposes of the Uniform Electronic Transactions Act (UETA) or the
Electronic Signatures in Global and National Commerce Act ("E-Sign")
unless a specific statement to the contrary is included in this message.
While this communication may be used to promote or market a transaction
or an idea that is discussed in the publication, it is intended to provide
general information about the subject matter covered and is provided with
the understanding that The Principal is not rendering legal, accounting,
or tax advice. It is not a marketed opinion and may not be used to avoid
penalties under the Internal Revenue Code. You should consult with
appropriate counsel or other advisors on all matters pertaining to legal,
tax, or accounting obligations and requirements.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Slow performance of Apache on Windows
Posted by "Feldhacker, Chris" <Fe...@principal.com>.
-----Original Message-----
From: William A. Rowe Jr. [mailto:wrowe@rowe-clan.net]
Sent: Wednesday, March 09, 2011 4:46 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Slow performance of Apache on Windows
> On 3/9/2011 4:44 PM, William A. Rowe Jr. wrote:
>> On 3/9/2011 3:15 PM, Feldhacker, Chris wrote:
>>>
>>> It was still odd that the problem only seemed to occur from our internal server, HTTP downloads from the internet seemed fine. Out of curiosity we setup a Windows Server 2008 running Apache, and a RedHat Linux server running Apache. The performance of the 2008 server was the same as the 2003 server, but the performance of the Linux server was double that of the Windows machines (10-11MB/s)! In fact, not only did the Windows 7 client maintain 10-11MB/s, but the Windows XP client had the same 10-11MB/s performance as well!
>>
>> On this one issue, did you try 'Enable Sendfile off' on the windows box?
> Sorry, asked and answered in the original summary, I just missed it the first time through.
We did get this problem resolved -- you were right.
If you'd like the tail-between-our-legs explanation and hopefully to benefit anyone else that might encounter this problem:
Ultimately, we did not discover the solution sooner due to inconsistent testing and shortcuts. (Isn't that how it usually goes?) Most testing was done in an isolated lab (especially network captures), but sometimes we'd play with settings on the production network out of convenience. And since the problem occurred with both XP and W7, we would try different things with different clients. And, once we made a change and it didn't work, we would set things back to "default" and go on.
We did try setting the three Apache options for windows: EnableSendfile Off, EnableMMAP Off, Win32DisableAcceptEx. But we must have done this on the production network and tested with the XP client, which only got the expected slow(er) result. However, apparently the EnableSendfile directive really did make a difference, we just didn't realize it (we can only assume the production network must have been under some load at the time). So, we removed the settings back to default and kept searching...
Per your suggestion, we focused on this setting again in the isolated lab -- the XP client achieved the 10MB/s, as did the W7 client. Nice.
Before finding this solution we were giving additional focus to the network traces (after all, the answer must be there, right?). In case anyone has interest, focusing first on traces from the Windows 7 client:
The TCP window size are slightly different, the client was always consistent at 65700, but Apache=64099, IIS=64094, SHTTPS=64094. (Don't know why Apache is 5 bytes higher, but anyway...) The window size really doesn't seem to matter. Looking at the traffic patterns, the server sends 2 packets, and then the client ACKs the last packet sent. But sometimes the server must be sending packets fast enough that 3 packets get through and the client ACKs the last packet sent. So, sometimes the client is ACKing every other packet, sometimes it's every 3rd packet. Which means sometimes the last packet from the server with the Push flag can be 1st, 2nd, or 3rd in the final "set". When the packet is 2nd or 3rd, the client has no problems and an ACK is sent right away with very little delay. When the packet is 1st, then (with Apache) the server waits for an ACK from the client before continuing. Of course, this is where Nagel's algorithm comes into play, which explains the client delay (looking at the traces, it varies between 35-210ms).
I say I don't think the window size matters because the same traffic pattern appears with IIS and SHTTPS -- sometimes the W7 client ACKs every 3rd packet instead of every other packet, so it can still end up on an "odd" boundary. The difference being, with IIS and SHTTPS, when the last packet with the Push flag comes through and if it happens to be 1st, *the server starts on the next sequence and keeps sending packets without stopping to wait for the client ACK*. So, IIS and SHTTPS are continuously streaming without waiting for ACKs of the Push packets, but Apache stops and waits.
(You would think using the TCP_NODELAY option on the client should workaround this problem. We definitely tested this previously with no luck, so we double-checked to make sure and still no luck. We tested this using curl with the "--tcp-nodelay" option, so unless there are issues with curl, I'm not sure why this doesn't seem to result in any improvement...)
So, what does the traffic look like with a W7 client against Apache with the EnableSendfile Off directive? Client ACKs are still sent after every 2 to 3 packets, but when the packet with the Push flag is sent Apache no longer stops and waits for an ACK but continues to stream packets -- just like IIS and SHTTPS did. No only that, but Push packets appear a LOT more frequently, which must mean Apache is streaming content in smaller chunks. There are 5 regular packets and the 6th contains the Push (8192 bytes total). With EnableSendfile On, there are 44 regular packets and the 45th contains the Push (65536 bytes total).
I don't know what Apache is doing differently with the EnableSendfile Off, but it seems to resolve the performance issue by continuously streaming data despite apparently using a smaller buffer size.
(Could probably squeeze an additional 6% performance improvement if the buffer size was larger so that every 6th packet wasn't only 60% full, but at this point I'm not complaining!)
It's hard to draw any conclusions from the XP traces -- time precision of the XP machine must be pretty poor as almost all of the time deltas display as zero; there's occasionally a 15ms blip with no apparent pattern, I assume it just happens to be whenever the clock happens to tick. However, the XP client always ACKs every 1-2 packets, even mid-stream; there is never a sequence of 3 packets. It's almost like the XP client is configured to ACK every packet rather than every other one (strange, nothing in the registry suggests this...perhaps something under a different key). So, with EnableSendfile On, it looks like Apache is still waiting for an ACK, but because it appears the client is trying to ACK every packet there is no delay. With EnableSendfile Off, the traffic appears the same, but again it's really hard to draw any conclusions given the low precision...
So, there you have it.
I think it's really unfortunate that a default install of Apache on Windows has this issue -- and because the symptoms vary so drastically based on the client OS, it's very easy to get mislead and think the problem is with the client...or the application...or the server...or the network.
Does anyone know if there is even a valid scenario when EnableSendfile MUST be turned On on a Windows machine? Seems like Off should be the default on Windows...
Thanks for the feedback and helping us double-check our work!
-----Message Disclaimer-----
This e-mail message is intended only for the use of the individual or
entity to which it is addressed, and may contain information that is
privileged, confidential and exempt from disclosure under applicable law.
If you are not the intended recipient, any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify us immediately by
reply email to Connect@principal.com and delete or destroy all copies of
the original message and attachments thereto. Email sent to or from the
Principal Financial Group or any of its member companies may be retained
as required by law or regulation.
Nothing in this message is intended to constitute an Electronic signature
for purposes of the Uniform Electronic Transactions Act (UETA) or the
Electronic Signatures in Global and National Commerce Act ("E-Sign")
unless a specific statement to the contrary is included in this message.
While this communication may be used to promote or market a transaction
or an idea that is discussed in the publication, it is intended to provide
general information about the subject matter covered and is provided with
the understanding that The Principal is not rendering legal, accounting,
or tax advice. It is not a marketed opinion and may not be used to avoid
penalties under the Internal Revenue Code. You should consult with
appropriate counsel or other advisors on all matters pertaining to legal,
tax, or accounting obligations and requirements.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Slow performance of Apache on Windows
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 3/9/2011 4:44 PM, William A. Rowe Jr. wrote:
> On 3/9/2011 3:15 PM, Feldhacker, Chris wrote:
>>
>> It was still odd that the problem only seemed to occur from our internal server, HTTP downloads from the internet seemed fine. Out of curiosity we setup a Windows Server 2008 running Apache, and a RedHat Linux server running Apache. The performance of the 2008 server was the same as the 2003 server, but the performance of the Linux server was double that of the Windows machines (10-11MB/s)! In fact, not only did the Windows 7 client maintain 10-11MB/s, but the Windows XP client had the same 10-11MB/s performance as well!
>
> On this one issue, did you try 'Enable Sendfile off' on the windows box?
Sorry, asked and answered in the original summary, I just missed it the first
time through.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Slow performance of Apache on Windows
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 3/9/2011 3:15 PM, Feldhacker, Chris wrote:
>
> It was still odd that the problem only seemed to occur from our internal server, HTTP downloads from the internet seemed fine. Out of curiosity we setup a Windows Server 2008 running Apache, and a RedHat Linux server running Apache. The performance of the 2008 server was the same as the 2003 server, but the performance of the Linux server was double that of the Windows machines (10-11MB/s)! In fact, not only did the Windows 7 client maintain 10-11MB/s, but the Windows XP client had the same 10-11MB/s performance as well!
On this one issue, did you try 'Enable Sendfile off' on the windows box?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Slow performance of Apache on Windows
Posted by Eric Covener <co...@gmail.com>.
> It is probably present in all that info somewhere, but how do we know
> that the TCP window size for IIS is not larger than the one for
> Apache?
my 2 cents:
I'm aware of anecdote that Apache setting SendBufferSize, despite
successfully hitting the socket, on some systems Windows does not have
the same affect as setting the global registry parameter (despite MSDN
docs)
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Slow performance of Apache on Windows
Posted by Jeff Trawick <tr...@gmail.com>.
On Wed, Mar 9, 2011 at 5:19 PM, Feldhacker, Chris
<Fe...@principal.com> wrote:
> -----Original Message-----
> From: Jeff Trawick [mailto:trawick@gmail.com]
> Sent: Wednesday, March 09, 2011 4:05 PM
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] Slow performance of Apache on Windows
>
> On Wed, Mar 9, 2011 at 4:15 PM, Feldhacker, Chris <Fe...@principal.com> wrote:
>>> Summary:
>>> Apache HTTP Server 2.2.17 running on Windows Server 2003 & 2008 has slight HTTP transfer performance problems with Windows XP clients, and *extreme* performance problems with Windows 7 clients.
>>> Apache HTTP Server running on RedHat Linux 5 performs fine with both XP and W7 clients.
>>> IIS and Small HTTP Server on Windows 2003/2008 perform fine with both XP and W7 clients.
>
>> Did you ever get a network trace with IIS as the server to see how it differs from the trace with Apache?
>
> Yes, the delayed acknowledgements by the client for the packet with the push flag were no longer present -- acks were being sent very quickly. (The time deltas for pretty much all network traffic was basically next to zero.) Other than the shortened delays, there were no other obvious differences between the traces.
ok; there must be something different though up until the point where
the client does/does not delay before sending the packet
odd number of packets up until this point with Apache, but even with
the other servers? (odd is where the delay kicks in, right?)
--/--
It is probably present in all that info somewhere, but how do we know
that the TCP window size for IIS is not larger than the one for
Apache?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Slow performance of Apache on Windows
Posted by "Feldhacker, Chris" <Fe...@principal.com>.
-----Original Message-----
From: Jeff Trawick [mailto:trawick@gmail.com]
Sent: Wednesday, March 09, 2011 4:05 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Slow performance of Apache on Windows
On Wed, Mar 9, 2011 at 4:15 PM, Feldhacker, Chris <Fe...@principal.com> wrote:
>> Summary:
>> Apache HTTP Server 2.2.17 running on Windows Server 2003 & 2008 has slight HTTP transfer performance problems with Windows XP clients, and *extreme* performance problems with Windows 7 clients.
>> Apache HTTP Server running on RedHat Linux 5 performs fine with both XP and W7 clients.
>> IIS and Small HTTP Server on Windows 2003/2008 perform fine with both XP and W7 clients.
> Did you ever get a network trace with IIS as the server to see how it differs from the trace with Apache?
Yes, the delayed acknowledgements by the client for the packet with the push flag were no longer present -- acks were being sent very quickly. (The time deltas for pretty much all network traffic was basically next to zero.) Other than the shortened delays, there were no other obvious differences between the traces.
-----Message Disclaimer-----
This e-mail message is intended only for the use of the individual or
entity to which it is addressed, and may contain information that is
privileged, confidential and exempt from disclosure under applicable law.
If you are not the intended recipient, any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify us immediately by
reply email to Connect@principal.com and delete or destroy all copies of
the original message and attachments thereto. Email sent to or from the
Principal Financial Group or any of its member companies may be retained
as required by law or regulation.
Nothing in this message is intended to constitute an Electronic signature
for purposes of the Uniform Electronic Transactions Act (UETA) or the
Electronic Signatures in Global and National Commerce Act ("E-Sign")
unless a specific statement to the contrary is included in this message.
While this communication may be used to promote or market a transaction
or an idea that is discussed in the publication, it is intended to provide
general information about the subject matter covered and is provided with
the understanding that The Principal is not rendering legal, accounting,
or tax advice. It is not a marketed opinion and may not be used to avoid
penalties under the Internal Revenue Code. You should consult with
appropriate counsel or other advisors on all matters pertaining to legal,
tax, or accounting obligations and requirements.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Slow performance of Apache on Windows
Posted by Jeff Trawick <tr...@gmail.com>.
On Wed, Mar 9, 2011 at 4:15 PM, Feldhacker, Chris
<Fe...@principal.com> wrote:
> Summary:
> Apache HTTP Server 2.2.17 running on Windows Server 2003 & 2008 has slight HTTP transfer performance problems with Windows XP clients, and *extreme* performance problems with Windows 7 clients.
> Apache HTTP Server running on RedHat Linux 5 performs fine with both XP and W7 clients.
> IIS and Small HTTP Server on Windows 2003/2008 perform fine with both XP and W7 clients.
Did you ever get a network trace with IIS as the server to see how it
differs from the trace with Apache?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org