You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Phillip Susi <ps...@cfl.rr.com> on 2005/08/02 22:51:05 UTC
SSL downloads faster than non SSL?
I decided to do some informal benchmark comparisons between using
windows SMBFS and apache/webdav for file transfers. I ended up finding
that apache is actually faster at sending files over an SSL connection
than a plain connection. I downloaded a ~600 meg test file from the
server using windows explorer webfolders, IE, and firefox. Firefox
downloads the file in the 4000-5000 KB/s range when using an SSL
connection. IE gets over 10,000 KB/s downloading over the secure
connection. Both only are able to download at 300-600 KB/s using the
non SSL connection though. This is, of course, all done over a 100 Mbps
ethernet network that is minimally loaded, and I repeated the test a few
times, clearing the browsers caches each time.
Needless to say, this is very strange. My client machine is running
windows 2000 pro, and the server is running windows 2003 server, with
apache 2.0.52 installed. Does anyone have any ideas to possibly explain
and correct these results?
Re: SSL downloads faster than non SSL?
Posted by Joost de Heer <jo...@sanguis.xs4all.nl>.
Phillip Susi wrote:
> I decided to do some informal benchmark comparisons between using
> windows SMBFS and apache/webdav for file transfers. I ended up finding
> that apache is actually faster at sending files over an SSL connection
> than a plain connection. I downloaded a ~600 meg test file from the
> server using windows explorer webfolders, IE, and firefox. Firefox
> downloads the file in the 4000-5000 KB/s range when using an SSL
> connection. IE gets over 10,000 KB/s downloading over the secure
> connection. Both only are able to download at 300-600 KB/s using the
> non SSL connection though. This is, of course, all done over a 100 Mbps
> ethernet network that is minimally loaded, and I repeated the test a few
> times, clearing the browsers caches each time.
Did you restart Apache too, to clear the memory cache of the OS?
Joost
Re: SSL downloads faster than non SSL?
Posted by Bill Stoddard <st...@apache.org>.
Nick Kew wrote:
> William A. Rowe, Jr. wrote:
>
>>At 09:01 PM 8/3/2005, Bill Stoddard wrote:
>>
>>
>>>A monitor thread would periodically check for a transmitfile
>>>completion status; if the completion status is too slow in
>>>coming, the monitor thread cancels the io and closes the socket.
>>
>>
>>We really need not wait ;-) Driving home from your neck of the
>>woods in NC (well, a bit west in fact, near Fontana) it struck me
>>that for all the individuals wishing for 'absolute' timeouts on
>>unix platforms, it would be rather cool to cache the start time
>>and run a parent thread against the scoreboard, killing all the
>>lingering processes subject to byte-at-a-time DoS attacks in the
>>headers. We would obviously need to be careful of lengthy req
>>bodies which would take more time than the 'absolute' timeout, but
>>your comment reminded me that perhaps, we can kill two birds with
>>one stone :)
>
>
> We can kill processes/threads that have spent too long in any given
> scoreboard state: that's exactly what I needed to do when I proposed
> what is now ap_hook_monitor.
>
> But as for byte-at-a-time DOS attacks, I don't think (OTTOMH) it's
> a good solution.
Of course your right, anything we do at the HTTP layer against DoS attacks is easy for a clever attacker to
circumvent. Need to go into deeper into the TCP stack to mount an effective defense against DoS attacks. OTOH,
blocking the simple stuff at the HTTP layer is often 'good enough' in practice.
Re: SSL downloads faster than non SSL?
Posted by Nick Kew <ni...@webthing.com>.
William A. Rowe, Jr. wrote:
> At 09:01 PM 8/3/2005, Bill Stoddard wrote:
>
>>A monitor thread would periodically check for a transmitfile
>>completion status; if the completion status is too slow in
>>coming, the monitor thread cancels the io and closes the socket.
>
>
> We really need not wait ;-) Driving home from your neck of the
> woods in NC (well, a bit west in fact, near Fontana) it struck me
> that for all the individuals wishing for 'absolute' timeouts on
> unix platforms, it would be rather cool to cache the start time
> and run a parent thread against the scoreboard, killing all the
> lingering processes subject to byte-at-a-time DoS attacks in the
> headers. We would obviously need to be careful of lengthy req
> bodies which would take more time than the 'absolute' timeout, but
> your comment reminded me that perhaps, we can kill two birds with
> one stone :)
We can kill processes/threads that have spent too long in any given
scoreboard state: that's exactly what I needed to do when I proposed
what is now ap_hook_monitor.
But as for byte-at-a-time DOS attacks, I don't think (OTTOMH) it's
a good solution. If we have a shortish timeout but up it for long
request bodies, the DOS simply mutates to sending
Content-Length: $BIG_RANDOM ... [byte] .... [byte] ....
--
Nick Kew
Re: SSL downloads faster than non SSL?
Posted by Bill Stoddard <st...@apache.org>.
William A. Rowe, Jr. wrote:
> At 09:01 PM 8/3/2005, Bill Stoddard wrote:
>
>>A monitor thread would periodically check for a transmitfile
>>completion status; if the completion status is too slow in
>>coming, the monitor thread cancels the io and closes the socket.
>
>
> We really need not wait ;-) Driving home from your neck of the
> woods in NC (well, a bit west in fact, near Fontana) it struck me
> that for all the individuals wishing for 'absolute' timeouts on
> unix platforms, it would be rather cool to cache the start time
> and run a parent thread against the scoreboard, killing all the
> lingering processes subject to byte-at-a-time DoS attacks in the
> headers. We would obviously need to be careful of lengthy req
> bodies which would take more time than the 'absolute' timeout, but
> your comment reminded me that perhaps, we can kill two birds with
> one stone :)
>
Yes, I like it. +1.
Bill
Re: SSL downloads faster than non SSL?
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 09:01 PM 8/3/2005, Bill Stoddard wrote:
>A monitor thread would periodically check for a transmitfile
>completion status; if the completion status is too slow in
>coming, the monitor thread cancels the io and closes the socket.
We really need not wait ;-) Driving home from your neck of the
woods in NC (well, a bit west in fact, near Fontana) it struck me
that for all the individuals wishing for 'absolute' timeouts on
unix platforms, it would be rather cool to cache the start time
and run a parent thread against the scoreboard, killing all the
lingering processes subject to byte-at-a-time DoS attacks in the
headers. We would obviously need to be careful of lengthy req
bodies which would take more time than the 'absolute' timeout, but
your comment reminded me that perhaps, we can kill two birds with
one stone :)
Re: SSL downloads faster than non SSL?
Posted by Bill Stoddard <st...@apache.org>.
William A. Rowe, Jr. wrote:
> At 10:48 AM 8/3/2005, Phillip Susi wrote:
>
>>William A. Rowe, Jr. wrote:
>>
>>
>>>In the APR library, yes, we translate 'apr_sendfile' to TransmitFile()
>>>on win32. Some other magic occurs to obtain a file handle which can be passed to TransmitFile. But there are enough flaws in the TF() api
>>>that perhaps this would be better defaulted to 'off'.
>>
>>Really? Are you quite sure? I wonder what's hosing it all up. Once you hand TransmitFile() the socket and file handles, it should blast the file over the network nice and fast.
>
>
> Yes of course :) However, sadly, Microsoft has a number of bugs
Search the archives of this or the apr mailing list... other than bugs (which can be reported to MS with
reasonable expectation that they will be fixed. maybe :-), the most serious flaw is that there is no way to
timeout calls to TransmitFile. If I call TransmitFile to send a file and the client chooses to not read any
of the bytes I send him, transmitfile will fill-up the send buffers in the TCP stack then block forever. I've
never found a way to check the 'status' of the call to TransmitFile, to see if it was making 'acceptable'
progress sending bytes to the client.
To solve (by some definition of solve) this timeout problem, we made TransmitFile (under apr_sendfile) send no
more than 64K bytes at a time. The call to transmitfile is non-blocking and the calling thread blocks on
WaitForSingleObject for 'timeout' seconds. If the call completes before the WaitFor call times out, we send
the next 64K byte chunk of the file. Repeat until all the file is sent. Whoever came up with the brilliant
idea of making multiple calls to TransmitFile to send files over 64K bytes needs to be dragged behind a bus ;-)
Now if Apache 2 supported asynchronous (or event driven) writes to the network (like IIS), we could just call
apr_sendfile/TransmitFile once to send the whole shaboozie and not worry (too much) about whether the client
is broken or is running a DoS attack. A monitor thread would periodically check for a transmitfile completion
status; if the completion status is too slow in coming, the monitor thread cancels the io and closes the socket.
Re: SSL downloads faster than non SSL?
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 10:48 AM 8/3/2005, Phillip Susi wrote:
>William A. Rowe, Jr. wrote:
>
>>In the APR library, yes, we translate 'apr_sendfile' to TransmitFile()
>>on win32. Some other magic occurs to obtain a file handle which can be passed to TransmitFile. But there are enough flaws in the TF() api
>>that perhaps this would be better defaulted to 'off'.
>Really? Are you quite sure? I wonder what's hosing it all up. Once you hand TransmitFile() the socket and file handles, it should blast the file over the network nice and fast.
Yes of course :) However, sadly, Microsoft has a number of bugs in
various versions of Windows NT kernels which have 1) messed up any
transmissions from NFS files mounted from other network file stores,
2) corrupted the output from TransmitFile() on specific kernel builds,
and 3) have some indeterminate behaviors when additional third party
drivers are injected into the socket stack.
The principal is wonderful, the practice has led to many bug reports
and inquires over the users@ support channel.
Bill
Re: SSL downloads faster than non SSL?
Posted by Brian Akins <ba...@web.turner.com>.
Phillip Susi wrote:
> My understanding is that the current code will memory map the data file,
> optionally encrypt it with SSL, and then call a conventional send().
> Using send() on a memory mapped file view instead of read() eliminates
It does not change the file in place.
--
Brian Akins
Lead Systems Engineer
CNN Internet Technologies
Re: SSL downloads faster than non SSL?
Posted by Phillip Susi <ps...@cfl.rr.com>.
William A. Rowe, Jr. wrote:
>In the APR library, yes, we translate 'apr_sendfile' to TransmitFile()
>on win32. Some other magic occurs to obtain a file handle which can
>be passed to TransmitFile. But there are enough flaws in the TF() api
>that perhaps this would be better defaulted to 'off'.
>
>
>
Really? Are you quite sure? I wonder what's hosing it all up. Once
you hand TransmitFile() the socket and file handles, it should blast the
file over the network nice and fast.
>That is also available. As you are aware TransmitFile() lives entirely
>in the kernel, so there are far fewer user<->kernel mode transitions.
>
>
Yes, there are fewer user-kernel transitions, but not that many and they
are relatively inexpensive. By far the largest savings that
TransmitFile() gains is from not having to copy the data from user to
kernel buffers before it can be sent over the network. A conventional
read() and send() call pair ends up making a copy from kernel FS buffer
memory to user buffer, then back to kernel socket buffer memory. That's
where most of the CPU time is wasted. A few years ago I wrote an FTP
server and tried using both TransmitFile() and using overlapped IO. By
disabling kernel buffering on the socket and memory mapping 2 32 KB
views of the file at once and overlapping both sends, I was able to
match both the network throughput and low CPU load of TransmitFile().
Specifically, I developed this on a PII-233 system with two fast
ethernet NICs installed. Using several other FTP servers popular at the
time, I was only able to manage around 5500 KB/s though one NIC using
100% of the CPU. Using either TransmitFile() or zero copy overlapped
IO, I was able to push 11,820 KB/s over one NIC and 8,500 KB/s over the
other ( not as good of a card apparently ) simultaneously using 1% of
the CPU. There was no noticeable difference between TransmitFile() and
the overlapped IO. Oh, and also I had to find a registry setting to
make TransmitFile() behave on my NT 4 workstation system the way it does
by default on NT server to get it to perform well. By default on
workstation it was not nearly so good.
>But if you turn off sendfile, and leave mmap on, Win32 (on Apache 2.0,
>but not back in Apache 1.3) does use memory mapped I/O.
>
>You suggest this works with SSL to create zero-copy? That's not quite
>correct, since there is the entire translation phase required.
>
>
>
My understanding is that the current code will memory map the data file,
optionally encrypt it with SSL, and then call a conventional send().
Using send() on a memory mapped file view instead of read() eliminates
one copy, but there is still another one made when you call send(), so
you're only half way there. To eliminate that second copy you have to
ask the kernel to set the socket buffer size to 0 ( I can't remember if
that was done with setsockopt or ioctlsocket ) and then use overlapped
IO ( preferably with IO completion ports for notification ) to give the
kernel multiple pending buffers to send. That way you eliminate the
second buffer copy and the NIC always has a locked buffer from which it
can DMA.
>:) We seriously appreciate all efforts. If you are very familiar with
>Win32 internals, the mpm_winnt.c does need work; I hope to change this
>mpm to follow unix in setting up/tearing down threads when we hit min
>and max thresholds. Obviously many other things in the (fairly simple)
>win32 implementation can be improved. Support for multiple processes
>is high on the list, since a fault in a single thread brings down the
>process and many established connections, and introduces a large latency
>until the next worker process is respawned and accepting connections.
>
>Bill
>
>
Well, ideally you just need a small number of worker threads using an IO
completion port. This yields much better results than allocating one
thread to each request, even if those threads are created in advance. I
have been trying to gain some understanding of the Apache 2 bucket
brigade system, but it's been a bit difficult just perusing the docs on
the web site in my spare time. From what I've been able to pick up so
far though, it looks like the various processing stages have the option
to either hold onto a bucket to process asynchronously, process the
bucket synchronously, or simply pass it down to the next layer
immediately. What I have not been able to figure out is if any of the
processing layers tend to make system calls to block the thread.
Provided that you don't do very much to block the thread while
processing a request, then if you were to use an IO completion port
model, a small handful of threads could service potentially thousands of
requests at once.
Also a fault in one thread does not have to kill the entire process, you
can catch the fault and handle it more gracefully. I'd love to dig into
mpm_winnt but at the moment my plate is a bit full. Maybe in another
month or two I'll be able to take a week off from work and dig into it.
Of course, if someone else who is already familiar with the code wants
to work on it, I'd be quite happy to consult ;)
Re: SSL downloads faster than non SSL?
Posted by Rici Lake <ri...@ricilake.net>.
On 3-Aug-05, at 8:11 AM, William A. Rowe, Jr. wrote:
> In the APR library, yes, we translate 'apr_sendfile' to TransmitFile()
> on win32. Some other magic occurs to obtain a file handle which can
> be passed to TransmitFile. But there are enough flaws in the TF() api
> that perhaps this would be better defaulted to 'off'.
>
+1
Re: SSL downloads faster than non SSL?
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 04:54 PM 8/2/2005, Phillip Susi wrote:
>That's what it was, thanks. Say, the name says 'Sendfile' but that doesn't exist on win32. Does apache call TransmitFile() instead or does it use some very poor library emulation of sendfile()? If it were using TransmitFile() I would expect it to not have any trouble saturating the network.
In the APR library, yes, we translate 'apr_sendfile' to TransmitFile()
on win32. Some other magic occurs to obtain a file handle which can
be passed to TransmitFile. But there are enough flaws in the TF() api
that perhaps this would be better defaulted to 'off'.
>FYI, I used wget to download the file without SSL and with sendfile disabled and it took 55 seconds and reported a rate of 11 MB/s. I used a copy command to download the file from the command prompt via the SMB mapped drive and it took 90 seconds. I wonder what the heck the deal is with all that fancy kernel mode redirector stuff not working out the way they designed?
>
>I don't even see why the option for sendfile should exist on the win32 build, seeing as how you can achieve the same performance using memory mapped zero copy overlapped IO, which also would work with non disk files, such as an SSL encrypted connection.
That is also available. As you are aware TransmitFile() lives entirely
in the kernel, so there are far fewer user<->kernel mode transitions.
But if you turn off sendfile, and leave mmap on, Win32 (on Apache 2.0,
but not back in Apache 1.3) does use memory mapped I/O.
You suggest this works with SSL to create zero-copy? That's not quite
correct, since there is the entire translation phase required.
>One of these days I'm going to have to take a little vacation from work and write an MPM to do just that.
:) We seriously appreciate all efforts. If you are very familiar with
Win32 internals, the mpm_winnt.c does need work; I hope to change this
mpm to follow unix in setting up/tearing down threads when we hit min
and max thresholds. Obviously many other things in the (fairly simple)
win32 implementation can be improved. Support for multiple processes
is high on the list, since a fault in a single thread brings down the
process and many established connections, and introduces a large latency
until the next worker process is respawned and accepting connections.
Bill
Re: SSL downloads faster than non SSL?
Posted by Phillip Susi <ps...@cfl.rr.com>.
That's what it was, thanks. Say, the name says 'Sendfile' but that
doesn't exist on win32. Does apache call TransmitFile() instead or does
it use some very poor library emulation of sendfile()? If it were using
TransmitFile() I would expect it to not have any trouble saturating the
network.
FYI, I used wget to download the file without SSL and with sendfile
disabled and it took 55 seconds and reported a rate of 11 MB/s. I used
a copy command to download the file from the command prompt via the SMB
mapped drive and it took 90 seconds. I wonder what the heck the deal is
with all that fancy kernel mode redirector stuff not working out the way
they designed?
I don't even see why the option for sendfile should exist on the win32
build, seeing as how you can achieve the same performance using memory
mapped zero copy overlapped IO, which also would work with non disk
files, such as an SSL encrypted connection.
One of these days I'm going to have to take a little vacation from work
and write an MPM to do just that.
Rici Lake wrote:
> Try setting
>
> EnableSendfile off
>
> in your apache config file and rerunning the tests. There have been
> other reports of surprisingly slow transfer speeds with sendfile
> enabled on Windows. This won't affect SSL, of course, since the SSL
> transfer cannot use sendfile.
>
>
Re: SSL downloads faster than non SSL?
Posted by Rici Lake <ri...@ricilake.net>.
Try setting
EnableSendfile off
in your apache config file and rerunning the tests. There have been
other reports of surprisingly slow transfer speeds with sendfile
enabled on Windows. This won't affect SSL, of course, since the SSL
transfer cannot use sendfile.
On 2-Aug-05, at 3:51 PM, Phillip Susi wrote:
> I decided to do some informal benchmark comparisons between using
> windows SMBFS and apache/webdav for file transfers. I ended up finding
> that apache is actually faster at sending files over an SSL connection
> than a plain connection. I downloaded a ~600 meg test file from the
> server using windows explorer webfolders, IE, and firefox. Firefox
> downloads the file in the 4000-5000 KB/s range when using an SSL
> connection. IE gets over 10,000 KB/s downloading over the secure
> connection. Both only are able to download at 300-600 KB/s using the
> non SSL connection though. This is, of course, all done over a 100
> Mbps
> ethernet network that is minimally loaded, and I repeated the test a
> few
> times, clearing the browsers caches each time.
>
> Needless to say, this is very strange. My client machine is running
> windows 2000 pro, and the server is running windows 2003 server, with
> apache 2.0.52 installed. Does anyone have any ideas to possibly
> explain
> and correct these results?
>
>