You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2021/11/17 01:26:30 UTC
[Bug 65686] New: sendfile() doesn't work
https://bz.apache.org/bugzilla/show_bug.cgi?id=65686
Bug ID: 65686
Summary: sendfile() doesn't work
Product: Apache httpd-2
Version: 2.4.51
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Core
Assignee: bugs@httpd.apache.org
Reporter: gary@cpanel.net
Target Milestone: ---
I'm not sure this is a regression, as I haven't had time to dig into it. On
Apache 2.4.51 linked against APR-1.7.0 (CentOS8, 64bit), sendfile() is not used
even though the following is set in httpd.conf;
EnableSendfile on
EnableMMAP on
Using this shows sendfile() is not used at all on a moderately busy server with
mixed traffic (PHP and html files, gzip/zip/etc)
strace -e sendfile,sendfile64 -f $(ps auxfww | egrep [h]ttpd | awk '{print $2}'
| sed 's/^/-p/g') 2>&1 ^C
I attached gdb to httpd and set a breakpoint on sendfile and sendfile64 and the
breakpoints were not triggered. I also hacked in some code into core.c to show
sendfile support is enabled:
[Wed Nov 17 01:10:52.550460 2021] [core:info] [pid 12459:tid 140518256277632]
EnableSendfile: on
Loglevel trace8 shows writev_nonblocking() is being used all the time, ie:
[Wed Nov 17 01:14:46.774434 2021] [core:trace6] [pid 12496:tid 139630070900480]
core_filters.c(806): [client 10.3.9.151:54629] writev_nonblocking: 8022/8022
Apache is compiled with -D APR_HAS_SENDFILE (according to httpd -V)
I've seen this behavior on CentOS7, CentOS8, and Ubuntu 20.x so far with stock
kernels et al.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65686] sendfile() doesn't work
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65686
--- Comment #2 from Ruediger Pluem <rp...@apache.org> ---
How large are the static files? Are they delivered directly by Apache?
File buckets below 256 byte of size are not sent via sendfile.
If would be helpful if you could break with gdb in can_sendfile_bucket and do
dump_bucket(b)
(requires the .gdbinit delivered with the source code to be loaded in gdb).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65686] sendfile() doesn't work
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65686
--- Comment #1 from Gary S <ga...@cpanel.net> ---
This may not be related;
But after each writev() call, mmap() is called and throws INVAL. An strace of
an active connection to a dummy index.html page:
7006 mmap(NULL, 7950, PROT_READ, MAP_SHARED, 17, 0) = 0x2aab472b1000
7006 munmap(0x2aab472b1000, 7950) = 0
7006 writev(13, [{iov_base="\27\3\3
\21\362|k\202\250&S\274\343\272\217\356r\375\322\233G\232Eu1\215B\236sc\321"...,
iov_len=8214}], 1) = 8214
7006 mmap(NULL, 72624, PROT_READ, MAP_SHARED, 17, 0x1f0e) = -1 EINVAL (Invalid
argument)
7006 lseek(17, 7950, SEEK_SET) = 7950
7006 read(17, "ackground-dim.has-background-dim"..., 8000) = 8000
7006 writev(13,
[{iov_base="\27\3\3\37Q\336\310}\v2b\335B\373\216F\266\200b\2\1\361\375\2608\214\213\3\10\355:n"...,
iov_len=8022}], 1) = 8022
7006 mmap(NULL, 64624, PROT_READ, MAP_SHARED, 17, 0x3e4e) = -1 EINVAL (Invalid
argument)
7006 lseek(17, 15950, SEEK_SET) = 15950
7006 read(17, "argin-bottom:0;max-width:840px;p"..., 8000) = 8000
7006 writev(13,
[{iov_base="\27\3\3\37Q\n\2076@\364\211\261OsS8\253\321&\201uK\203\5X\t\27\370\253\f\332M"...,
iov_len=8022}], 1) = 8022
7006 mmap(NULL, 56624, PROT_READ, MAP_SHARED, 17, 0x5d8e) = -1 EINVAL (Invalid
argument)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65686] sendfile() doesn't work
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65686
--- Comment #7 from Stefan Eissing <ic...@apache.org> ---
If I read this correctly, everything is working without TLS.
The thing is that a TLS encrypted connection cannot make use of sendfile() on
common operation systems.
There have been efforts by Netflix in the past to have an encrypted TLS
sendfile()
(<https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99>),
but that requires OS specific support.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65686] sendfile() doesn't work
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65686
--- Comment #3 from Gary S <ga...@cpanel.net> ---
I've tried static files that range in size from 512 bytes to 16,000 bytes in
size (html files)
(In reply to Ruediger Pluem from comment #2)
> How large are the static files? Are they delivered directly by Apache?
> File buckets below 256 byte of size are not sent via sendfile.
>
> If would be helpful if you could break with gdb in can_sendfile_bucket and do
>
> dump_bucket(b)
>
>
> (requires the .gdbinit delivered with the source code to be loaded in gdb).
Looks like b() is being optimized out, I guess I'll need to recompile
everything without any opts at all;
Breakpoint 1, can_sendfile_bucket (b=<optimized out>) at core_filters.c:621
621 if (APR_BUCKET_IS_FILE(b) && b->length >= AP_MIN_SENDFILE_BYTES) {
(gdb) dump_bucket(b)
value has been optimized out
I'll work on it tonight and get you the information.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65686] sendfile() doesn't work
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65686
--- Comment #4 from Gary S <ga...@cpanel.net> ---
(In reply to Ruediger Pluem from comment #2)
> How large are the static files? Are they delivered directly by Apache?
> File buckets below 256 byte of size are not sent via sendfile.
>
> If would be helpful if you could break with gdb in can_sendfile_bucket and do
>
> dump_bucket(b)
>
>
> (requires the .gdbinit delivered with the source code to be loaded in gdb).
Here's the requested information:
Breakpoint 1, can_sendfile_bucket (b=0x558cded27068) at core_filters.c:621
621 if (APR_BUCKET_IS_FILE(b) && b->length >= AP_MIN_SENDFILE_BYTES) {
(gdb) dump_bucket(b)
bucket=HEAP (558cded27068) length=1555 data=558cded26de8
contents=[~~~~z~~~v~~~~~l~Z...] rc=1
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65686] sendfile() doesn't work
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65686
Gary Stanley <ga...@cpanel.net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |INVALID
Status|NEW |RESOLVED
--- Comment #8 from Gary Stanley <ga...@cpanel.net> ---
Ahh.. Makes sense now. I'll close this out, as I was unaware of sendfile() not
working with TLS (maybe kernel TLS on Linux could solve this issue)
Closing this out.
Thank you.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65686] sendfile() doesn't work
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65686
--- Comment #6 from Gary S <ga...@cpanel.net> ---
Interesting discovery. a breakpoint on sendfile64/sendfile without SSL (in a
virtualhost), sendfile is working:
(gdb) c
Continuing.
Breakpoint 1, sendfile64 () at ../sysdeps/unix/syscall-template.S:81
81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) c
Continuing.
#1 0x00007f80c891e5af in apr_socket_sendfile (sock=0x55d4ecb81a20,
file=0x55d4ecd3c4e0, hdtr=0x7f80c8b33e40 <no_hdtr>, offset=<optimized out>,
len=0x7ffd49aa0c08, flags=<optimized out>)
at network_io/unix/sendrecv.c:333
#2 0x000055d4eb2b3185 in sendfile_nonblocking (s=0x55d4ecb81a20,
bucket=0x55d4ecd20d48, ctx=0x55d4ecb82150, c=0x55d4ecb81c10) at
core_filters.c:828
#3 0x000055d4eb2b296d in send_brigade_nonblocking (s=0x55d4ecb81a20,
bb=0x55d4ecd3c5e0, ctx=0x55d4ecb82150, c=0x55d4ecb81c10) at core_filters.c:660
#4 0x000055d4eb2b243d in ap_core_output_filter (f=0x55d4ecb82098,
new_bb=0x55d4ecd3c5e0) at core_filters.c:505
#5 0x000055d4eb28fb3c in ap_pass_brigade (next=0x55d4ecb82098,
bb=0x55d4ecd3c5e0) at util_filter.c:590
#6 0x000055d4eb2e029f in ap_http_outerror_filter (f=0x55d4ecd2b298,
b=0x55d4ecd3c5e0) at http_filters.c:1907
#7 0x000055d4eb28fb3c in ap_pass_brigade (next=0x55d4ecd2b298,
bb=0x55d4ecd3c5e0) at util_filter.c:590
#8 0x000055d4eb2df2d0 in ap_http_header_filter (f=0x55d4ecd2b270,
b=0x55d4ecd3c5e0) at http_filters.c:1541
#9 0x000055d4eb28fb3c in ap_pass_brigade (next=0x55d4ecd2b270,
bb=0x55d4ecd3c5e0) at util_filter.c:590
#10 0x000055d4eb2996e0 in ap_content_length_filter (f=0x55d4ecd2b248,
b=0x55d4ecd3c5e0) at protocol.c:1956
I confirmed this with an strace as well:
# strace -f $(ps auxfww | egrep [h]ttpd | awk '{print $2}' | sed 's/^/-p/g')
2>&1 -e sendfile64,sendfile
strace: Process 17680 attached
sendfile(13, 17, [0] => [77416], 83306) = 77416
sendfile(13, 17, [77416], 5890) = -1 EAGAIN (Resource temporarily
unavailable)
sendfile(13, 17, [77416], 5890) = -1 EAGAIN (Resource temporarily
unavailable)
sendfile(13, 17, [77416] => [83306], 5890) = 5890
Reproduction steps I'm using to demonstrate this behavior, Create a large
index.html, ie:
php -i > index.html
Use curl locally / remotely (or a browser)
curl -k https://virtualhost/index.html <-- self signed certificate host
This works, though:
curl http://virtualhost/index.htm
Do you still need the dump_brigade(bb) with the above inforation?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65686] sendfile() doesn't work
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65686
--- Comment #5 from Ruediger Pluem <rp...@apache.org> ---
(In reply to Gary S from comment #4)
> (In reply to Ruediger Pluem from comment #2)
> > How large are the static files? Are they delivered directly by Apache?
> > File buckets below 256 byte of size are not sent via sendfile.
> >
> > If would be helpful if you could break with gdb in can_sendfile_bucket and do
> >
> > dump_bucket(b)
> >
> >
> > (requires the .gdbinit delivered with the source code to be loaded in gdb).
>
> Here's the requested information:
>
> Breakpoint 1, can_sendfile_bucket (b=0x558cded27068) at core_filters.c:621
> 621 if (APR_BUCKET_IS_FILE(b) && b->length >= AP_MIN_SENDFILE_BYTES) {
> (gdb) dump_bucket(b)
> bucket=HEAP (558cded27068) length=1555 data=558cded26de8
> contents=[~~~~z~~~v~~~~~l~Z...] rc=1
Thanks. The above is a heap bucket which cannot be sent via sendfile, but this
does not mean that there is no issue. I think I just used a bad breakpoint. Can
you please break in send_brigade_nonblocking and do
dump_brigade(bb)
? As a the response might be split over multiple brigades you need to do this
until the file is delivered.
Another thing: Can you check (probably via strace) if that happens as well for
a virtual host that is
- http/1.1 (no ssl)
- Has no PHP, content compression or alike setup, but just delivers static
files?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org