You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by "William A. Rowe, Jr." <wr...@rowe-clan.net> on 2003/04/23 22:03:42 UTC
Win32 apr_sockets; Random notes from the 'oh, duh' department
Folks,
we have a number of real issues with the win32 implementation of sockets
which I think I can fix over the next day or so, but I'm just dropping off this
note early enough for folks to bitch^H^H^H^H^H comment.
Serious Problem #1: Blocking Socket, apr_sendv() invokes WSASend()
with NULL completion arguments... according to WinSock2 this should
be a sync, blocking send. Put a huge (4MB), mmap'ed region into that
iovec, and it will be sent - BUT IT IS SENT ASYNC! So later we close the
socket and the transmission is aborted.
Solution? Apparently we just can't trust sync send/recv, and I guess
it is time to simply surrender and make them all async with a completion
event, irrespective of timeouts. Then wait on that event INFINITE or by
the so_timeout value.
Question; do we simply use the socket itself as the event handle?
Or do we create one event for each socket that we can later use for all
sorts of goodness such as a WSAAsyncSelect()-style poll? Two threads
won't be looking at the same socket, so I think an event handle in each
apr_socket_t would be useful.
Serious Problem #2: Win32 doesn't support socket timeouts. We are
using raw WSASend/WSARecv, blocking. This just isn't good.
Solution? Same as for #1 above.
Serious Problem #3: Using WaitForSingleObject() with our 'socket'
timeout value is rather bogus - it would measure the total elapsed time,
not the interval between packet transmissions.
Solution? Well, if WinSock2 had some sort of 'progress' indication, bytes
sent or received so far for a completion-based request - then we could just
sample and assure ourselves that something had happened. As it is we
just can't do that. Effectively, it looks like any huge send over a slow wire
will be timed out based on any sane setting for a timeout value.
Those are just the random observations of the day. If you have a Win32
sockets patch - now would be the time to toss it back up for consideration
since I'm already up to my neck in WinSock2.
Bill
Re: Win32 apr_sockets; Random notes from the 'oh, duh'
department
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 08:11 PM 4/23/2003, Bill Stoddard wrote:
>>>This is horribly inefficient on Windows where sys call overhead is high.
>>>Windows uses the builtin setsockopt(SO_RCVTIMEOUT) timeout mechanisms (which are part of the BSD socket spec but is not implemented on Unix). If, as you are reporting, WSA* calls are not really working as they should, then we should probably just do them async and wait for the io completion.
>>
>>Ok... according to every document on WinSock2 I've read, it appears that
>>SO_RCVTIMEOUT isn't supported on Win32. I could be misreading, or
>>just out of my mind, but it appears clearly documented - as unsupported.
>
>I am floored... got any pointers to the doc?
Well now my brain is twisted...
Here, it's documented as 'supported';
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/socket_options_for_windows_nt_and_windows_95_98_me_2.asp
Here is a terrific grid;
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/socket_options.asp
But here is something completely different, documented as 'unsupported';
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/setsockopt_2.asp
"BSD options not supported for setsockopt are shown in the following table. "
Now what does that mean? Supported on win32, unsupported on BSD?
Or options from BSD not supported on Win32?
Even worse;
"SO_RCVTIMEO and SO_SNDTIMEO"
"When using the <http://msdn.microsoft.com/library/en-us/winsock/winsock/recv_2.asp>recv function, if no data arrives during the period specified in SO_RCVTIMEO,
the recv function completes. In Windows versions prior to Windows 2000, any data received
subsequently fails with WSAETIMEDOUT. In Windows 2000 and later, if no data arrives within
the period specified in SO_RCVTIMEO the recv function returns WSAETIMEDOUT, and if data
is received, recv returns SUCCESS."
"If a send or receive operation times out on a socket, the socket state is indeterminate,
and should not be used; TCP sockets in this state have a potential for data loss, since
the operation could be canceled at the same moment the operation was to be completed."
Take your pick. What scares the bejezus out of me is that if the
docs contradict, what will your average TCP/IP Filter author do?
>>>>Serious Problem #3: Using WaitForSingleObject() with our 'socket'
>>>>timeout value is rather bogus - it would measure the total elapsed time,
>>>>not the interval between packet transmissions.
>>>>Solution? Well, if WinSock2 had some sort of 'progress' indication, bytes
>>>>sent or received so far for a completion-based request - then we could just
>>>>sample and assure ourselves that something had happened. As it is we
>>>>just can't do that. Effectively, it looks like any huge send over a slow wire
>>>>will be timed out based on any sane setting for a timeout value.
>>>
>>>That should not be the case. Last time I checked, I though we set the timeout
>>>then sent 64K (?) chunks then reset the timeout for the next 64k chunk. This
>>>is -eaxactly- what 1.3 does on Unix (where signals are used as the timeout
>>>mechanism). This has someone changed that w/o my noticing? I'll check later...
>>
>>No, especially not in the case of mmaps. The way filters, brigades and
>>buckets work, you can end up with a 16MB sendfile or 4MB mmap. In
>>the later case, you may have 4MB * 16 iovecs in a single WSASend.
>
>Yaaaarrrggggg.... right you are! The assumption all along was the the application
>(httpd in the case) would not send down huge buffers to send. I did not realize that
>mmaps broke this assumption. Ouch...
It must be up to APR - we can't expect application authors to make low-level
decisions when all of the low-level implementations differ ;-)
>It should not be too difficult to make the mods to make WSASend and WSARecv do the async thing.
I concur - patch on it's way tomorrow.
Bill
Re: Win32 apr_sockets; Random notes from the 'oh, duh' department
Posted by Bill Stoddard <bi...@wstoddard.com>.
William A. Rowe, Jr. wrote:
> At 01:15 PM 4/24/2003, Bill Stoddard wrote:
>
>
>>>>That should not be the case. Last time I checked, I though we set the timeout then sent 64K (?) chunks then reset the timeout for the next 64k chunk. This is -eaxactly- what 1.3 does on Unix (where signals are used as the timeout mechanism). This has someone changed that w/o my noticing? I'll check later...
>>>
>>>No, especially not in the case of mmaps. The way filters, brigades and
>>>buckets work, you can end up with a 16MB sendfile or 4MB mmap. In
>>>the later case, you may have 4MB * 16 iovecs in a single WSASend.
>>
>>Humm... I am wondering if breaking up sends should really be the responsibility of APR. Consider that the application sets the send timeout, the application understands how its expected to behave and what kind of link it is talking over. Seems it should be able to control the size of the buffer to send and the time it will allow for it to be sent. Using that line of reasoning, one could argue that if the app (httpd in this case) sent to APR a 4MB mmap to send but did not set an appropriate timeout, that it is httpd that is at fault if the send does not complete before the timeout fires.
>
>
> I partially agree that *apr* networking shouldn't worry about this. Clearly,
> *apr-util* brigades and buckets need to consider this impact. But I am
> confused by one thing...
>
> When I implement timeout mechanisms, they are always 'inactivity' timers.
> E.g., I presumed that SO_SNDTIMEO/SO_RCVTIMEO were timing out
> based on the intra-transmission delay, not on the total time of the operation.
> Stevens didn't help on this - so I'm hoping someone can share if those
> options either;
>
> * timeout clocked from the start of the entire transmission, or
Yep, pretty sure this is what it means. You do a WSASend for 100,000
bytes and you have SO_SNDTIMEO for the stack to take ownership of your
buffers to complete the send.
Bill
Re: Win32 apr_sockets; Random notes from the 'oh, duh'
department
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 01:15 PM 4/24/2003, Bill Stoddard wrote:
>>>That should not be the case. Last time I checked, I though we set the timeout then sent 64K (?) chunks then reset the timeout for the next 64k chunk. This is -eaxactly- what 1.3 does on Unix (where signals are used as the timeout mechanism). This has someone changed that w/o my noticing? I'll check later...
>>
>>No, especially not in the case of mmaps. The way filters, brigades and
>>buckets work, you can end up with a 16MB sendfile or 4MB mmap. In
>>the later case, you may have 4MB * 16 iovecs in a single WSASend.
>
>Humm... I am wondering if breaking up sends should really be the responsibility of APR. Consider that the application sets the send timeout, the application understands how its expected to behave and what kind of link it is talking over. Seems it should be able to control the size of the buffer to send and the time it will allow for it to be sent. Using that line of reasoning, one could argue that if the app (httpd in this case) sent to APR a 4MB mmap to send but did not set an appropriate timeout, that it is httpd that is at fault if the send does not complete before the timeout fires.
I partially agree that *apr* networking shouldn't worry about this. Clearly,
*apr-util* brigades and buckets need to consider this impact. But I am
confused by one thing...
When I implement timeout mechanisms, they are always 'inactivity' timers.
E.g., I presumed that SO_SNDTIMEO/SO_RCVTIMEO were timing out
based on the intra-transmission delay, not on the total time of the operation.
Stevens didn't help on this - so I'm hoping someone can share if those
options either;
* timeout clocked from the start of the entire transmission, or
* restart the timeout clock from each successful packet transmission
Bill
Re: Win32 apr_sockets; Random notes from the 'oh, duh' department
Posted by Bill Stoddard <bi...@wstoddard.com>.
>>That should not be the case. Last time I checked, I though we set the timeout
>>then sent 64K (?) chunks then reset the timeout for the next 64k chunk.
>>This is -eaxactly- what 1.3 does on Unix (where signals are used as the
>>timeout mechanism). This has someone changed that w/o my noticing?
>>I'll check later...
>
>
> No, especially not in the case of mmaps. The way filters, brigades and
> buckets work, you can end up with a 16MB sendfile or 4MB mmap. In
> the later case, you may have 4MB * 16 iovecs in a single WSASend.
Humm... I am wondering if breaking up sends should really be the
responsibility of APR. Consider that the application sets the send
timeout, the application understands how its expected to behave and what
kind of link it is talking over. Seems it should be able to control the
size of the buffer to send and the time it will allow for it to be sent.
Using that line of reasoning, one could argue that if the app (httpd
in this case) sent to APR a 4MB mmap to send but did not set an
appropriate timeout, that it is httpd that is at fault if the send does
not complete before the timeout fires.
I agree that some kind of progress indicator would be the best solution.
Bill
Re: Win32 apr_sockets; Random notes from the 'oh, duh' department
Posted by Bill Stoddard <bi...@wstoddard.com>.
>
>>This is horribly inefficient on Windows where sys call overhead is high.
>>Windows uses the builtin setsockopt(SO_RCVTIMEOUT) timeout
>>mechanisms (which are part of the BSD socket spec but is not
>>implemented on Unix). If, as you are reporting, WSA* calls are not
>>really working as they should, then we should probably just do them
>>async and wait for the io completion.
>
>
> Ok... according to every document on WinSock2 I've read, it appears that
> SO_RCVTIMEOUT isn't supported on Win32. I could be misreading, or
> just out of my mind, but it appears clearly documented - as unsupported.
I am floored... got any pointers to the doc?
>>> Solution? Same as for #1 above.
>
>
> Of course if we solve the problems above, this one comes free even if
> SO_RCVTIMEOUT isn't implemented.
yep.
>
>
>>> Serious Problem #3: Using WaitForSingleObject() with our 'socket'
>>>timeout value is rather bogus - it would measure the total elapsed time,
>>>not the interval between packet transmissions.
>>> Solution? Well, if WinSock2 had some sort of 'progress' indication, bytes
>>>sent or received so far for a completion-based request - then we could just
>>>sample and assure ourselves that something had happened. As it is we
>>>just can't do that. Effectively, it looks like any huge send over a slow wire
>>>will be timed out based on any sane setting for a timeout value.
>>
>>That should not be the case. Last time I checked, I though we set the timeout
>>then sent 64K (?) chunks then reset the timeout for the next 64k chunk.
>>This is -eaxactly- what 1.3 does on Unix (where signals are used as the
>>timeout mechanism). This has someone changed that w/o my noticing?
>>I'll check later...
>
>
> No, especially not in the case of mmaps. The way filters, brigades and
> buckets work, you can end up with a 16MB sendfile or 4MB mmap. In
> the later case, you may have 4MB * 16 iovecs in a single WSASend.
Yaaaarrrggggg.... right you are! The assumption all along was the the
application (httpd in the case) would not send down huge buffers to
send. I did not realize that mmaps broke this assumption. Ouch...
It should not be too difficult to make the mods to make WSASend and
WSARecv do the async thing.
Bill
Re: Win32 apr_sockets; Random notes from the 'oh, duh'
department
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 05:24 PM 4/23/2003, Bill Stoddard wrote:
>William A. Rowe, Jr. wrote;
>> Serious Problem #1: Blocking Socket, apr_sendv() invokes WSASend()
>>with NULL completion arguments... according to WinSock2 this should
>>be a sync, blocking send. Put a huge (4MB), mmap'ed region into that iovec, and it will be sent - BUT IT IS SENT ASYNC! So later we close the
>>socket and the transmission is aborted.
>Humm... I'll ruminate on this some... there may be another solution. You sure the WSASend call is not returning a timeout expiration?
No, the strangest bit is that I'm running on W2K SP3... but this sounds
very much like the XP afd.sys bug. We begin the transfer and the function
returns immediately ... it does continue to send the entire 4MB.
Upon socket close, Win32 blocks for the file to complete. But it doesn't
seem to do so predictably. No matter what, the buffer could move out
from under WSASend, so it's a big bug. But the truncation could also
have to do with my need to do a linger close.
/me wonders if we linger correctly when doing a TransmitFile and
attempting to reuse the socket.
>> Solution? Apparently we just can't trust sync send/recv, and I guess
>>it is time to simply surrender and make them all async with a completion
>>event, irrespective of timeouts. Then wait on that event INFINITE or by the so_timeout value.
>> Question; do we simply use the socket itself as the event handle?
>
>Should be fine as long as multiple threads never attempt to touch the socket at the same time. apr_sendfile uses the socket as the event handle and AFAIK it has never caused a problem.
Right. Once an MPM passes a socket to a given thread, it must be
hands-off.
>>Or do we create one event for each socket that we can later use for all
>>sorts of goodness such as a WSAAsyncSelect()-style poll? Two threads
>>won't be looking at the same socket, so I think an event handle in each
>>apr_socket_t would be useful.
>I'd prefer to not use a seperate event handle (for performance reasons) unless we have a clear demonstrated fully thought out need to. MHO.
I'm also rewriting poll using WSAEventSelect() (not AsyncSelect - that's
the one that posts messages to hWnds.) I will toss in some ifdefs that
let us play with both models (socket-as-event v.s. discrete event handle.)
>> Serious Problem #2: Win32 doesn't support socket timeouts. We are
>>using raw WSASend/WSARecv, blocking. This just isn't good.
>
>Sure it does, just not the same way as most unix systems. Unix uses nonblocking i/o followed by a select to do timeout.
Huh? I'm describing what happens if we timeout within a single operation.
Not leading 'up to' that operation.
>This is horribly inefficient on Windows where sys call overhead is high.
>Windows uses the builtin setsockopt(SO_RCVTIMEOUT) timeout
>mechanisms (which are part of the BSD socket spec but is not
>implemented on Unix). If, as you are reporting, WSA* calls are not
>really working as they should, then we should probably just do them
>async and wait for the io completion.
Ok... according to every document on WinSock2 I've read, it appears that
SO_RCVTIMEOUT isn't supported on Win32. I could be misreading, or
just out of my mind, but it appears clearly documented - as unsupported.
>> Solution? Same as for #1 above.
Of course if we solve the problems above, this one comes free even if
SO_RCVTIMEOUT isn't implemented.
>> Serious Problem #3: Using WaitForSingleObject() with our 'socket'
>>timeout value is rather bogus - it would measure the total elapsed time,
>>not the interval between packet transmissions.
>> Solution? Well, if WinSock2 had some sort of 'progress' indication, bytes
>>sent or received so far for a completion-based request - then we could just
>>sample and assure ourselves that something had happened. As it is we
>>just can't do that. Effectively, it looks like any huge send over a slow wire
>>will be timed out based on any sane setting for a timeout value.
>
>That should not be the case. Last time I checked, I though we set the timeout
>then sent 64K (?) chunks then reset the timeout for the next 64k chunk.
>This is -eaxactly- what 1.3 does on Unix (where signals are used as the
>timeout mechanism). This has someone changed that w/o my noticing?
>I'll check later...
No, especially not in the case of mmaps. The way filters, brigades and
buckets work, you can end up with a 16MB sendfile or 4MB mmap. In
the later case, you may have 4MB * 16 iovecs in a single WSASend.
Re: Win32 apr_sockets; Random notes from the 'oh, duh' department
Posted by Bill Stoddard <bi...@wstoddard.com>.
William A. Rowe, Jr. wrote:
> Folks,
>
> we have a number of real issues with the win32 implementation of sockets
> which I think I can fix over the next day or so, but I'm just dropping off this
> note early enough for folks to bitch^H^H^H^H^H comment.
>
> Serious Problem #1: Blocking Socket, apr_sendv() invokes WSASend()
> with NULL completion arguments... according to WinSock2 this should
> be a sync, blocking send. Put a huge (4MB), mmap'ed region into that
> iovec, and it will be sent - BUT IT IS SENT ASYNC! So later we close the
> socket and the transmission is aborted.
Humm... I'll ruminate on this some... there may be another solution.
You sure the WSASend call is not returning a timeout expiration?
> Solution? Apparently we just can't trust sync send/recv, and I guess
> it is time to simply surrender and make them all async with a completion
> event, irrespective of timeouts. Then wait on that event INFINITE or by
> the so_timeout value.
>
> Question; do we simply use the socket itself as the event handle?
Should be fine as long as multiple threads never attempt to touch the
socket at the same time. apr_sendfile uses the socket as the event
handle and AFAIK it has never caused a problem.
> Or do we create one event for each socket that we can later use for all
> sorts of goodness such as a WSAAsyncSelect()-style poll? Two threads
> won't be looking at the same socket, so I think an event handle in each
> apr_socket_t would be useful.
I'd prefer to not use a seperate event handle (for performance reasons)
unless we have a clear demonstrated fully thought out need to. MHO.
>
> Serious Problem #2: Win32 doesn't support socket timeouts. We are
> using raw WSASend/WSARecv, blocking. This just isn't good.
Sure it does, just not the same way as most unix systems. Unix uses
nonblocking i/o followed by a select to do timeout. This is horribly
inefficient on Windows where sys call overhead is high. Windows uses
the builtin setsockopt(SO_RCVTIMEOUT) timeout mechanisms (which are part
of the BSD socket spec but is not implemented on Unix). If, as you are
reporting, WSA* calls are not really working as they should, then we
should probably just do them async and wait for the io completion.
>
> Solution? Same as for #1 above.
>
> Serious Problem #3: Using WaitForSingleObject() with our 'socket'
> timeout value is rather bogus - it would measure the total elapsed time,
> not the interval between packet transmissions.
>
> Solution? Well, if WinSock2 had some sort of 'progress' indication, bytes
> sent or received so far for a completion-based request - then we could just
> sample and assure ourselves that something had happened. As it is we
> just can't do that. Effectively, it looks like any huge send over a slow wire
> will be timed out based on any sane setting for a timeout value.
That should not be the case. Last time I checked, I though we set the
timeout then sent 64K (?) chunks then reset the timeout for the next 64k
chunk. This is -eaxactly- what 1.3 does on Unix (where signals are used
as the timeout mechanism). This has someone changed that w/o my
noticing? I'll check later...
Bill