pipelined HTTP/1.1 request

[dean gaudet <de...@arctic.org>]

% cat >req
GET /index.html.en HTTP/1.1
Host: arctic.org:8888

GET /index.html.en HTTP/1.1
Host: arctic.org:8888

^D
% nc arctic.org 8888 < req > resp

results in this strace:

accept(5, {sin_family=AF_INET, sin_port=htons(2151), sin_addr=inet_addr("204.107.140.52")}}, [16]) = 8
rt_sigaction(SIGUSR1, {SIG_IGN}, {0x807f774, [], SA_INTERRUPT|0x4000000}, 8) = 0
setsockopt(8, IPPROTO_TCP1, [1], 4)     = 0
getsockname(8, {sin_family=AF_INET, sin_port=htons(8888), sin_addr=inet_addr("204.107.140.52")}}, [16]) = 0
fcntl(8, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(8, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
read(8, "GET /index.html.en HTTP/1.1\nHost"..., 8192) = 102
gettimeofday({978894801, 277613}, NULL) = 0
stat("/home/dean/ap2/htdocs/index.html.en", {st_mode=S_IFREG|0664, st_size=1349, ...}) = 0
open("/home/dean/ap2/htdocs/index.html.en", O_RDONLY) = 9
write(4, "204.107.140.52 - - [07/Jan/2001:"..., 87) = 87
gettimeofday({978894801, 278770}, NULL) = 0
stat("/home/dean/ap2/htdocs/index.html.en", {st_mode=S_IFREG|0664, st_size=1349, ...}) = 0
open("/home/dean/ap2/htdocs/index.html.en", O_RDONLY) = 10
getsockopt(8, IPPROTO_TCP, 1, [1], [4]) = 0
setsockopt(8, IPPROTO_TCP1, [0], 4)     = 0
setsockopt(8, IPPROTO_TCP3, [1], 4)     = 0
writev(8, [{"HTTP/1.1 200 OK\r\nDate: Sun, 07 J"..., 275}], 1) = 275
sendfile(8, 9, [0], 1349)               = 1349
writev(8, [{"HTTP/1.1 200 OK\r\nDate: Sun, 07 J"..., 275}], 1) = 275
setsockopt(8, IPPROTO_TCP3, [0], 4)     = 0
setsockopt(8, IPPROTO_TCP1, [1], 4)     = 0
read(8, 0x814cfb0, 8192)                = -1 EAGAIN (Resource temporarily unavailable)
sendfile(8, 10, [0], 1349)              = 1349
write(4, "204.107.140.52 - - [07/Jan/2001:"..., 87) = 87
read(8, 0x814cfb0, 8192)                = -1 EAGAIN (Resource temporarily unavailable)
select(9, [8], NULL, NULL, {15, 0})     = 0 (Timeout)
gettimeofday({978894816, 275011}, NULL) = 0
shutdown(8, 1 /* send */)               = 0
gettimeofday({978894816, 275329}, NULL) = 0
read(8, "", 512)                        = 0
close(8)                                = 0
rt_sigaction(SIGUSR1, {0x807f774, [], SA_INTERRUPT|0x4000000}, {SIG_IGN}, 8) = 0
close(10)                               = 0
close(9)                                = 0
accept(5,  <unfinished ...>

weird --  the "setsockopt(8, IPPROTO_TCP3, [0], 4)" is the popping of the
CORK.  why does it occur between the second writev and the second sendfile?
this will result in suboptimal network packets.

as i suggested in the last message i think the solution is to remember
the cork/NODELAY state in the socket and delay changes as long as possible.

note that this is the case where the "halfduplex"/"saferead" stuff in
1.3 comes into play.  i haven't yet looked through the i/o layer, but
the only way to properly pack network packets in a pipelined request is
to be able to test for a blocking read, and then cause a flush (to the
network) if the read would block.

in the context of 2.0 this means the CORK pretty much needs to be
controlled by httpd, not by APR...  or httpd needs to be able to set
something equivalent to SAFEREAD on the socket.

another bug is that the close(9) of the /index.html.en file is delayed
until the end of the connection rather than the end of the request.
this is a resource leak... i bet it's just registered into the wrong pool.
it should be on the request pool, not the connection pool.  (or maybe
we're just lacking a request cleanup?  it happens even when un-pipelined)

-dean

Re: pipelined HTTP/1.1 request

[rb...@covalent.net]

> another bug is that the close(9) of the /index.html.en file is delayed
> until the end of the connection rather than the end of the request.
> this is a resource leak... i bet it's just registered into the wrong pool.
> it should be on the request pool, not the connection pool.  (or maybe
> we're just lacking a request cleanup?  it happens even when un-pipelined)

Actually, this was done on purpose, and we need to find another way to do
this.  The basic issue is that it is possible for the file to not be sent
before the request pool is cleaned.  This problem can be easily seen by
just requesting the "It Works" page from a standard 2.0 build, after
changing the pool back to the request_rec.  You will get no data at all.

The reason this happens is that we wait for 9K of data before sending out
a request, but if this is the first request in a pipeline, and we have
under 9K, we won't send any data until either we get 9K of data, or there
are no more requests.  Well, we do our best to use sendfile for files, so
we put the file bucket on the brigade in the core_output_filter, but the
file itself was allocated out of the request pool, so when the request
goes away, so does the file, even if it hasn't been sent yet.  That is
VERY bad.

There are a couple of possible solutions:

1)  If we have a file in the brigade, just call sendfile, whether we have
enough for a full packet or not.  Of course, this will need to be more
general, so it will be anything allocated out of a pool other than a pool
bucket.

I dislike this, because it is a special case and it will usually be wrong.

2)  During the check for the pipelined request, if there is data allocated
out of a pool, send it then.

This is better, but it is still a special case.

3)  Keep things in the connection pool.

This is annoying, because it is a resource leak.

4)  Register a cleanup with the request pool.  This cleanup will basically
take the file bucket and convert it to a malloc bucket.

I kind of like this.  I just thought of it, but it shouldn't be THAT hard
to do.  The idea is simple enough.  If we have a file bucket in the
core_output, we know it is less than 9K, so we are pretty safe to just
read that into memory.  This will also allow us to move the file (and thus
MMAP) back into the request pool.

In case you can't tell, I like option 4 best.  :-)

Ryan

_______________________________________________________________________________
Ryan Bloom                        	rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------