You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Bill Stoddard <bi...@wstoddard.com> on 2001/01/31 23:51:48 UTC
AP_MIN_SENDFILE_BYTES
Ryan recently committed a patch to disable sendfile if the number of bytes to
send was less than AP_MIN_SENDFILE_BYTES. Had a couple of private notes
exchanges with Ryan and wanted to bring the discussion public. Here is an
example supporting the patch (from Ryan):
telnet localhost 80
GET /index.html HTTP/1.1
Host: localhost
Range: bytes= 0-850, 840-900
Without this change, this requires three calls to sendfile. One for bytes
0-850, one for bytes 840-850, and one for 850-900.
The patch get around this nicely, however at a steep price on Apache for
Windows. We take a 50% performance hit when serving files smaller than
AP_MIN_SENDFILE_BYTES because we cannot use apr_sendfile() and reuse the
accept socket. I would like to consider alternate solutions such as:
1. back this patch out entirely and suffer the small packets on the network
for the cases above (which are not that common).
2. Ryan suggests tuning AP_MIN_SENDFILE_BYTES. To what? 500 bytes? The number
needs to be much smaller than 8192 where it is set now. Thoughts?
3. Bypassing this optimizationonly for byterange requests (or perhaps multiple
byterange requests)
Bill
Re: AP_MIN_SENDFILE_BYTES
Posted by Greg Ames <gr...@raleigh.ibm.com>.
rbb@covalent.net wrote:
>
> > The patch get around this nicely, however at a steep price on Apache for
> > Windows. We take a 50% performance hit when serving files smaller than
> > AP_MIN_SENDFILE_BYTES because we cannot use apr_sendfile() and reuse the
> > accept socket. I would like to consider alternate solutions such as:
> >
> > 1. back this patch out entirely and suffer the small packets on the network
> > for the cases above (which are not that common).
>
> It is common. Take a look at what the Adobe browser plug-in
> requests. The last time I looked this made a lot of overlapping range
> requests.
>
Yes, but what percentage of web traffic on the Internet is .pdf/Adobe,
or other byterange requests? I suspect a small fraction, compared to
GETs for entire files. We need to keep a sharp focus on what our
mainline path is if we ever hope to compete with other high end web
servers.
> > 2. Ryan suggests tuning AP_MIN_SENDFILE_BYTES. To what? 500 bytes? The number
> > needs to be much smaller than 8192 where it is set now. Thoughts?
>
> As I said when I committed the change, we need to run tests, and tune the
> number accordingly. I guessed at 8192, it is likely wrong, and I said
> that when I committed. I haven't had the time to run any tests. Feel
> free to run your own. If nobody runs them, I will run the tests in a few
> weeks, after we go beta. Arguing over the correct value without running
> tests doesn't really do us any good. Let's run the tests, and use real
> numbers to get the answers.
>
That's probably a good idea to do in any case, byterange or no. The
optimal answer will no doubt change depending on the platforms.
> > 3. Bypassing this optimizationonly for byterange requests (or perhaps multiple
> > byterange requests)
>
> Again, let's get some numbers, and speak with real numbers and real
> performance. I really believe this patch is required, we just need to get
> the cut-off right.
>
The optimal solution may very well be one number for mainline, and a
different number for byterange. But who knows? yeah, I agree 100%
about benchmarking it.
Greg
Re: AP_MIN_SENDFILE_BYTES
Posted by Jeff Trawick <tr...@bellsouth.net>.
"Bill Stoddard" <bi...@wstoddard.com> writes:
> > lots of byte ranges; I dunno how many overlapping ones
> >
> > [trawick@daedalus www]$ grep "HTTP/1... 200" weblog | wc
> > 26067 352253 3359819
> > [trawick@daedalus www]$ grep "HTTP/1... 206" weblog | wc
> > 9979 129769 1207219
>
> All but 7 are from the same ip address.
darnit, my brain is disintegrating...
That was *me* yesterday :) I had a little program banging on
apache.org:8092 and told it to generate lots of silly byte range
request ("Range: bytes=0-0,1-5" or something like that).
--
Jeff Trawick | trawickj@bellsouth.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: AP_MIN_SENDFILE_BYTES
Posted by Bill Stoddard <bi...@wstoddard.com>.
> rbb@covalent.net writes:
>
> > > > It is common. Take a look at what the Adobe browser plug-in
> > > > requests. The last time I looked this made a lot of overlapping range
> > > > requests.
> > >
> > > But how prevalent is the adobe browser plug-in? I really don't know.
> >
> > It;s not just Adobe, that was an example. Remember, that Brian was able
> > to generate over 50 core files in under 5 minutes the first time we put
> > 2.0 on Apache.org. All of those that I looked at were for byterange
> > requests. I believe we are all under estimating how many requests are
> > byterange.
>
> lots of byte ranges; I dunno how many overlapping ones
>
> [trawick@daedalus www]$ grep "HTTP/1... 200" weblog | wc
> 26067 352253 3359819
> [trawick@daedalus www]$ grep "HTTP/1... 206" weblog | wc
> 9979 129769 1207219
All but 7 are from the same ip address.
Bill
Re: AP_MIN_SENDFILE_BYTES
Posted by Jeff Trawick <tr...@bellsouth.net>.
rbb@covalent.net writes:
> > > It is common. Take a look at what the Adobe browser plug-in
> > > requests. The last time I looked this made a lot of overlapping range
> > > requests.
> >
> > But how prevalent is the adobe browser plug-in? I really don't know.
>
> It;s not just Adobe, that was an example. Remember, that Brian was able
> to generate over 50 core files in under 5 minutes the first time we put
> 2.0 on Apache.org. All of those that I looked at were for byterange
> requests. I believe we are all under estimating how many requests are
> byterange.
lots of byte ranges; I dunno how many overlapping ones
[trawick@daedalus www]$ grep "HTTP/1... 200" weblog | wc
26067 352253 3359819
[trawick@daedalus www]$ grep "HTTP/1... 206" weblog | wc
9979 129769 1207219
--
Jeff Trawick | trawickj@bellsouth.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: AP_MIN_SENDFILE_BYTES
Posted by James Sutherland <ja...@cam.ac.uk>.
On Thu, 1 Feb 2001 rbb@covalent.net wrote:
>
> > > It is common. Take a look at what the Adobe browser plug-in
> > > requests. The last time I looked this made a lot of overlapping range
> > > requests.
> >
> > But how prevalent is the adobe browser plug-in? I really don't know.
>
> It;s not just Adobe, that was an example. Remember, that Brian was able
> to generate over 50 core files in under 5 minutes the first time we put
> 2.0 on Apache.org. All of those that I looked at were for byterange
> requests. I believe we are all under estimating how many requests are
> byterange.
Apart from anything else, anyone resuming a download from /dist will be
generating byterange traffic. A download manager may also generate a
byterange request, to check it's supported properly - at which point,
every Windows user downloading the binary via Go!Zilla will be donating
another "core"...
James.
Re: AP_MIN_SENDFILE_BYTES
Posted by rb...@covalent.net.
> > It is common. Take a look at what the Adobe browser plug-in
> > requests. The last time I looked this made a lot of overlapping range
> > requests.
>
> But how prevalent is the adobe browser plug-in? I really don't know.
It;s not just Adobe, that was an example. Remember, that Brian was able
to generate over 50 core files in under 5 minutes the first time we put
2.0 on Apache.org. All of those that I looked at were for byterange
requests. I believe we are all under estimating how many requests are
byterange.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: AP_MIN_SENDFILE_BYTES
Posted by Bill Stoddard <bi...@wstoddard.com>.
>
> > The patch get around this nicely, however at a steep price on Apache for
> > Windows. We take a 50% performance hit when serving files smaller than
> > AP_MIN_SENDFILE_BYTES because we cannot use apr_sendfile() and reuse the
> > accept socket. I would like to consider alternate solutions such as:
> >
> > 1. back this patch out entirely and suffer the small packets on the network
> > for the cases above (which are not that common).
>
> It is common. Take a look at what the Adobe browser plug-in
> requests. The last time I looked this made a lot of overlapping range
> requests.
But how prevalent is the adobe browser plug-in? I really don't know.
>
> > 2. Ryan suggests tuning AP_MIN_SENDFILE_BYTES. To what? 500 bytes? The number
> > needs to be much smaller than 8192 where it is set now. Thoughts?
>
> As I said when I committed the change, we need to run tests, and tune the
> number accordingly. I guessed at 8192, it is likely wrong, and I said
> that when I committed. I haven't had the time to run any tests. Feel
> free to run your own. If nobody runs them, I will run the tests in a few
> weeks, after we go beta. Arguing over the correct value without running
> tests doesn't really do us any good. Let's run the tests, and use real
> numbers to get the answers.
>
> > 3. Bypassing this optimizationonly for byterange requests (or perhaps multiple
> > byterange requests)
>
> Again, let's get some numbers, and speak with real numbers and real
> performance. I really believe this patch is required, we just need to get
> the cut-off right.
>
What are we testing? Performance of the byterange request? I am sure the performance difference
will be substantial on Windows (in favor of using sendfile whenever possible), even with byterange
requests. I'll do some testing tomorrow on Windows. Have any ideas on: how many byte ranges the
adobe plug-in 'typically' request (1, 2 many)? How large are the byte ranges?
Bill
Re: AP_MIN_SENDFILE_BYTES
Posted by rb...@covalent.net.
> > > read()+write() for "normal" GETs on tiny files
> > > sendfile() on "normal" GETs on bigger files
> > > mmap()+writev() for multiple byteranges, whatever the size
> >
> > When I talk about read, I expect us to use writev, so we will merge
> > packets.
>
> Yes - will multiple small reads be merged, though?
>
> If we get a request like bytes 1-5, 10-20, 25-75..., unless the read is
> merged into a single bigger read, mmap() will probably be the winner
> there. Just fault in the first page, and access directly.
>
> Of course, logic to merge reads like that together into a 4k read would be
> better still...
That is done with the ap_f* functionst that I have been trying to get into
the code. The MMAP and read options will both use the exact same writev
logic, so the packet merging will be exactly the same.
> > Also, even for byterange requests, we will want to use
> > sendfile. Imagine a byterange request for 15000 bytes from a 10Meg
> > file. This can be sent using sendfile instead of mmap.
>
> Yes - that's a single range, so it would be anyway under the logic
> above. If you're hopping around, though, multiple sendfile()s will hurt
> much more than an mmap().
Now extend that, to something like
Range: bytes=0-10000, 15000-25000
Sendfile will help with that case. It really has to do with the length of
the data, not the type of the request.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: AP_MIN_SENDFILE_BYTES
Posted by James Sutherland <ja...@cam.ac.uk>.
On Fri, 2 Feb 2001 rbb@covalent.net wrote:
> > read()+write() for "normal" GETs on tiny files
> > sendfile() on "normal" GETs on bigger files
> > mmap()+writev() for multiple byteranges, whatever the size
>
> When I talk about read, I expect us to use writev, so we will merge
> packets.
Yes - will multiple small reads be merged, though?
If we get a request like bytes 1-5, 10-20, 25-75..., unless the read is
merged into a single bigger read, mmap() will probably be the winner
there. Just fault in the first page, and access directly.
Of course, logic to merge reads like that together into a 4k read would be
better still...
> Also, even for byterange requests, we will want to use
> sendfile. Imagine a byterange request for 15000 bytes from a 10Meg
> file. This can be sent using sendfile instead of mmap.
Yes - that's a single range, so it would be anyway under the logic
above. If you're hopping around, though, multiple sendfile()s will hurt
much more than an mmap().
James.
Re: AP_MIN_SENDFILE_BYTES
Posted by rb...@covalent.net.
> > Yep. Basically, there needs to be a cut-off, so that we don't sendfile a
> > lot of little files. We are much better off with a hierarchy:
> >
> > read if it's really small,
> > mmap or sendfile if it's a bit bigger.
>
> I think for byterange requests, mmap()+writev() would be the best - the
> other two options won't merge packets, which will be a performance
> hit. Even for tiny files, getting bytes 5-10,10-15 would be better under
> mmap()+writev() - read/write and sendfile will both send two tiny packets,
> it seems.
>
> So:
>
> read()+write() for "normal" GETs on tiny files
> sendfile() on "normal" GETs on bigger files
> mmap()+writev() for multiple byteranges, whatever the size
When I talk about read, I expect us to use writev, so we will merge
packets. Also, even for byterange requests, we will want to use
sendfile. Imagine a byterange request for 15000 bytes from a 10Meg
file. This can be sent using sendfile instead of mmap.
This is all dependant on what the response data looks like.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: AP_MIN_SENDFILE_BYTES
Posted by James Sutherland <ja...@cam.ac.uk>.
On Fri, 2 Feb 2001 rbb@covalent.net wrote:
> > > >Incidentally, does anyone know if a writev() will combine the chunks into
> > > >large packets, or send each section individually??
> > >
> > > The former.
> >
> > Whereas using sendfile() for each chunk will generate lots of little
> > packetlets... mmap()+writev() seems like the way to go, then??
>
> Yep. Basically, there needs to be a cut-off, so that we don't sendfile a
> lot of little files. We are much better off with a hierarchy:
>
> read if it's really small,
> mmap or sendfile if it's a bit bigger.
I think for byterange requests, mmap()+writev() would be the best - the
other two options won't merge packets, which will be a performance
hit. Even for tiny files, getting bytes 5-10,10-15 would be better under
mmap()+writev() - read/write and sendfile will both send two tiny packets,
it seems.
So:
read()+write() for "normal" GETs on tiny files
sendfile() on "normal" GETs on bigger files
mmap()+writev() for multiple byteranges, whatever the size
I think that's the right structure??
James.
Re: AP_MIN_SENDFILE_BYTES
Posted by rb...@covalent.net.
> > >Incidentally, does anyone know if a writev() will combine the chunks into
> > >large packets, or send each section individually??
> >
> > The former.
>
> Whereas using sendfile() for each chunk will generate lots of little
> packetlets... mmap()+writev() seems like the way to go, then??
Yep. Basically, there needs to be a cut-off, so that we don't sendfile a
lot of little files. We are much better off with a hierarchy:
read if it's really small,
mmap or sendfile if it's a bit bigger.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: AP_MIN_SENDFILE_BYTES
Posted by James Sutherland <ja...@cam.ac.uk>.
On Thu, 1 Feb 2001, Tony Finch wrote:
> James Sutherland <ja...@cam.ac.uk> wrote:
> >
> >Incidentally, does anyone know if a writev() will combine the chunks into
> >large packets, or send each section individually??
>
> The former.
Whereas using sendfile() for each chunk will generate lots of little
packetlets... mmap()+writev() seems like the way to go, then??
James.
Re: AP_MIN_SENDFILE_BYTES
Posted by Tony Finch <do...@dotat.at>.
James Sutherland <ja...@cam.ac.uk> wrote:
>
>Incidentally, does anyone know if a writev() will combine the chunks into
>large packets, or send each section individually??
The former.
Tony.
--
f.a.n.finch fanf@covalent.net dot@dotat.at
" ``Well, let's go down and find out who's grave it is.''
``How?'' ``By going down and finding out!'' "
Re: AP_MIN_SENDFILE_BYTES
Posted by James Sutherland <ja...@cam.ac.uk>.
On Thu, 1 Feb 2001 rbb@covalent.net wrote:
>
> > I suspect the answer is 'it depends on the OS'. Apache 1.3 does not mmap files under a certain
> > threshold size (a few thousand bytes if I recall). Sendfile on Windows is very efficient (by
> > Windows standards :-) even for very small files, especially when you can do the 'reuse the accept
> > socket' trick. Worth investigating on Unix for sure.
>
> Again, it isn't the sendfile call that will hurt us. If that is all you
> measure, you will miss the problem. The problem is the small packet that
> is going to be sent. That is not a fast operation on any platform.
Yes. If I have some spare time, I might try doing some benchmarks of this.
Incidentally, does anyone know if a writev() will combine the chunks into
large packets, or send each section individually?? If the latter, using
mmap()+writev() would avoid this problem completely...
James.
Re: AP_MIN_SENDFILE_BYTES
Posted by rb...@covalent.net.
> I suspect the answer is 'it depends on the OS'. Apache 1.3 does not mmap files under a certain
> threshold size (a few thousand bytes if I recall). Sendfile on Windows is very efficient (by
> Windows standards :-) even for very small files, especially when you can do the 'reuse the accept
> socket' trick. Worth investigating on Unix for sure.
Again, it isn't the sendfile call that will hurt us. If that is all you
measure, you will miss the problem. The problem is the small packet that
is going to be sent. That is not a fast operation on any platform.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: AP_MIN_SENDFILE_BYTES
Posted by Bill Stoddard <bi...@wstoddard.com>.
> On Wed, 31 Jan 2001 rbb@covalent.net wrote:
>
> >
> > > The patch get around this nicely, however at a steep price on Apache for
> > > Windows. We take a 50% performance hit when serving files smaller than
> > > AP_MIN_SENDFILE_BYTES because we cannot use apr_sendfile() and reuse the
> > > accept socket. I would like to consider alternate solutions such as:
> > >
> > > 1. back this patch out entirely and suffer the small packets on the network
> > > for the cases above (which are not that common).
> >
> > It is common. Take a look at what the Adobe browser plug-in
> > requests. The last time I looked this made a lot of overlapping range
> > requests.
> >
> > > 2. Ryan suggests tuning AP_MIN_SENDFILE_BYTES. To what? 500 bytes? The number
> > > needs to be much smaller than 8192 where it is set now. Thoughts?
> >
> > As I said when I committed the change, we need to run tests, and tune the
> > number accordingly. I guessed at 8192, it is likely wrong, and I said
> > that when I committed. I haven't had the time to run any tests. Feel
> > free to run your own. If nobody runs them, I will run the tests in a few
> > weeks, after we go beta. Arguing over the correct value without running
> > tests doesn't really do us any good. Let's run the tests, and use real
> > numbers to get the answers.
> >
> > > 3. Bypassing this optimizationonly for byterange requests (or perhaps multiple
> > > byterange requests)
> >
> > Again, let's get some numbers, and speak with real numbers and real
> > performance. I really believe this patch is required, we just need to get
> > the cut-off right.
>
> Question: how does mmap()+writev() compare with sendfile() for this kind
> of usage? On the example given, the first page (4k/8k) of the file will be
> faulted in, then a simple writev() will send the whole body, avoiding all
> the seeks, syscalls etc?
>
I suspect the answer is 'it depends on the OS'. Apache 1.3 does not mmap files under a certain
threshold size (a few thousand bytes if I recall). Sendfile on Windows is very efficient (by
Windows standards :-) even for very small files, especially when you can do the 'reuse the accept
socket' trick. Worth investigating on Unix for sure.
Bill
Bill
Re: AP_MIN_SENDFILE_BYTES
Posted by James Sutherland <ja...@cam.ac.uk>.
On Wed, 31 Jan 2001 rbb@covalent.net wrote:
>
> > The patch get around this nicely, however at a steep price on Apache for
> > Windows. We take a 50% performance hit when serving files smaller than
> > AP_MIN_SENDFILE_BYTES because we cannot use apr_sendfile() and reuse the
> > accept socket. I would like to consider alternate solutions such as:
> >
> > 1. back this patch out entirely and suffer the small packets on the network
> > for the cases above (which are not that common).
>
> It is common. Take a look at what the Adobe browser plug-in
> requests. The last time I looked this made a lot of overlapping range
> requests.
>
> > 2. Ryan suggests tuning AP_MIN_SENDFILE_BYTES. To what? 500 bytes? The number
> > needs to be much smaller than 8192 where it is set now. Thoughts?
>
> As I said when I committed the change, we need to run tests, and tune the
> number accordingly. I guessed at 8192, it is likely wrong, and I said
> that when I committed. I haven't had the time to run any tests. Feel
> free to run your own. If nobody runs them, I will run the tests in a few
> weeks, after we go beta. Arguing over the correct value without running
> tests doesn't really do us any good. Let's run the tests, and use real
> numbers to get the answers.
>
> > 3. Bypassing this optimizationonly for byterange requests (or perhaps multiple
> > byterange requests)
>
> Again, let's get some numbers, and speak with real numbers and real
> performance. I really believe this patch is required, we just need to get
> the cut-off right.
Question: how does mmap()+writev() compare with sendfile() for this kind
of usage? On the example given, the first page (4k/8k) of the file will be
faulted in, then a simple writev() will send the whole body, avoiding all
the seeks, syscalls etc?
James.
Re: AP_MIN_SENDFILE_BYTES
Posted by rb...@covalent.net.
> The patch get around this nicely, however at a steep price on Apache for
> Windows. We take a 50% performance hit when serving files smaller than
> AP_MIN_SENDFILE_BYTES because we cannot use apr_sendfile() and reuse the
> accept socket. I would like to consider alternate solutions such as:
>
> 1. back this patch out entirely and suffer the small packets on the network
> for the cases above (which are not that common).
It is common. Take a look at what the Adobe browser plug-in
requests. The last time I looked this made a lot of overlapping range
requests.
> 2. Ryan suggests tuning AP_MIN_SENDFILE_BYTES. To what? 500 bytes? The number
> needs to be much smaller than 8192 where it is set now. Thoughts?
As I said when I committed the change, we need to run tests, and tune the
number accordingly. I guessed at 8192, it is likely wrong, and I said
that when I committed. I haven't had the time to run any tests. Feel
free to run your own. If nobody runs them, I will run the tests in a few
weeks, after we go beta. Arguing over the correct value without running
tests doesn't really do us any good. Let's run the tests, and use real
numbers to get the answers.
> 3. Bypassing this optimizationonly for byterange requests (or perhaps multiple
> byterange requests)
Again, let's get some numbers, and speak with real numbers and real
performance. I really believe this patch is required, we just need to get
the cut-off right.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------