FreeBSD-CURRENT with threads...

[Justin Erenkrantz <je...@ebuilt.com>]

Well, I've now had a chance to bring up FreeBSD-CURRENT (aka 5.0) and
am going to *try* to see what we can do to get threading working with
APR and FreeBSD.  I figure I should start with -CURRENT and see if I
can work my way backwards through this sucker.

Basically, I'm using test/sendfile to see if it'll work.  It looks 
like sendfile (perhaps other functions) is losing the buffer in 
non-blocking mode.  It works fine when the client is in blocking mode.

What is odd is that it is failing at the end of the first writev call
required for the headers.

57324 byte in hdr 3 = 57344 overall (15+5+57324)

Which is how much we sent in the first call to sendfile.  We don't 
notice the failure until two more sendfile operations have occurred 
(oh, well).  Note that there are a couple of fixes recently in 
sendfile on FreeBSD - notably to the return value when using writev 
for the headers/trailers (looks like a few months ago).  This may or 
may not have any impact on daedalus's error conditions.  I've cleaned
up my local copy of apr_sendfile (WTF do we add the header length to 
bytes_to_send - this looks like a remanant of an old FreeBSD version.
AFAICT, -CURRENT and -STABLE don't need this and look like they
would be adversely affected by having this - I dunno...)

I checked the client and it definitely is getting 0 from the other
end.  I may try to hack with this sendfile test to see if I can
capture the generated network traffic.  I don't think this problem
is with our receive logic - just with our sendfile logic.

I have a feeling once this test works, other things will magically
work too.  =-)

Any thoughts, tips, or suggestions are welcomed.  My gut tells me
that this is something remarkable simple (i.e. build system) not a 
logic error in either APR or FreeBSD.  I've been known to be wrong
before though.  -- justin

jerenkrantz@walla% ./sendfile server
Waiting for a client to connect...
Processing a client...
problem with data read (byte 57324 of hdr 3):
read `0' (0x30) from client; expected `^'
---
jerenkrantz@walla% ./sendfile client timeout
Creating a test file...
Sending the file...
Calling apr_sendfile()...
Headers:
        15 bytes   
        5 bytes
        80000 bytes
File: 200000 bytes from offset 0
Trailers:
        19 bytes
        10 bytes
        90000 bytes
apr_sendfile()->0, sent 57344 bytes
Calling apr_sendfile()...
Headers:
        22676 bytes
File: 200000 bytes from offset 0
Trailers:
        19 bytes
        10 bytes
        90000 bytes
apr_sendfile()->0, sent 57344 bytes
Calling apr_sendfile()...
Headers:
        22676 bytes
File: 200000 bytes from offset 0
Trailers:
        19 bytes   
        10 bytes   
        90000 bytes
apr_sendfile()->0, sent 22676 bytes
Calling apr_sendfile()...
Headers:
File: 200000 bytes from offset 0
Trailers:
        19 bytes   
        10 bytes   
        90000 bytes
apr_sendfile()->32, sent 0 bytes
client problem: sent 80020 of 370049 bytes