You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Brian Pane <br...@apache.org> on 2005/08/22 01:25:19 UTC
TCP_CORK in Linux 2.6: not broken, but it doesn't work with TCP_NODELAY
There was some discussion a few weeks ago about whether TCP_CORK was
broken on Linux 2.6.
I did some tests on Linux 2.6.12 and found that TCP_CORK *does*
appear to
work. However, it doesn't work if TCP_NODELAY is also set on the
socket (as
the tcp(7) manual page warns).
I've attached my test program in case it's useful to others. It's
run as
sendfile_test [--cork] [--nodelay] [--nonblock] filename
listener_port_number.
It listens endlessly for connections on the specified port. When it
gets a connection,
it write(2)s a short HTTP response header and then calls sendfile to
transmit the
entire contents of the file. The optional arguments tell it whether
to set TCP_CORK,
TCP_NODELAY, and/or O_NONBLOCK on the accepted connection; the default
is not to set any of these.
Looking at packet traces, the pattern I see is:
* TCP_NODELAY not set, TCP_CORK not set: the HTTP response header
arrives in a packet all by itself, followed by the start of the
file in the next packet.
* TCP_NODELAY not set, TCP_CORK set: the first packet of the response
contains
the HTTP response header and the first part of the file. The
packet is filled with
data up to the MSS, unless the total header+file content is
smaller than the MSS.
* TCP_NODELAY set, TCP_CORK set: the HTTP response header
arrives in a packet all by itself, followed by the start of the
file in the next packet.
* TCP_NODELAY set, TCP_CORK not set: the HTTP response header
arrives in a packet all by itself, followed by the start of the
file in the next packet.
I tested with both blocking or nonblocking sockets, and both modes
yielded
these same results.
httpd-2.x always sets TCP_NODELAY, so it makes sense that people are
observing it sending the response header in a separate packet under
Linux
2.6. (I'm guessing that TCP_CORK works even in the presence of
TCP_NODELAY under 2.4, but I don't have a 2.4 system to test.)
Rather than automatically setting TCP_NODELAY in core_pre_connection(),
perhaps we should set it conditionally in core_output_filter(), where
we have
enough information to tell whether it's needed.
Brian
Re: TCP_CORK in Linux 2.6: not broken, but it doesn't work with TCP_NODELAY
Posted by Joe Orton <jo...@redhat.com>.
On Wed, Aug 24, 2005 at 08:56:38AM -0700, Justin Erenkrantz wrote:
> --On August 23, 2005 10:59:47 PM +0100 Joe Orton <jo...@redhat.com> wrote:
>
> >The patch won't (shouldn't ;) make any difference to non-Linux
> >platforms, since the autoconf test checks for TCP_CORK and TCP_NODELAY
> >explicitly. I don't know whether FreeBSD has ever or does currently
> >allow TCP_NOPUSH and TCP_NODELAY together, but it would be easy enough
> >to adapt the code appropriately if someone tested that stuff out.
>
> Ah, right. So, the autoconf test will just fail on FreeBSD/Darwin due to
> TCP_CORK not being present - thus, having no real change to that platform.
> If we can figure out what FreeBSD/Darwin can do, we can then write some
> m4-fu to try to optimize it if desired.
yup.
> Isn't there a test program somewhere that I could use to see what it'll do
> in this scenario?
You can start with the test program Brian wrote, I've uploaded it to
http://people.apache.org/~jorton/sendfile.c but it would need to be
adapted to either use APR directly and to test APR behaviour, or to use
the FreeBSD sendfile() interface, TCP_NOPUSH, etc.
(I may possibly write an APR-based test program to demonstrate the APR
behaviour at some point and I'll post that if I do, but it might not be
any time soon)
joe
Re: TCP_CORK in Linux 2.6: not broken, but it doesn't work with
TCP_NODELAY
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On August 23, 2005 10:59:47 PM +0100 Joe Orton <jo...@redhat.com> wrote:
> The patch won't (shouldn't ;) make any difference to non-Linux
> platforms, since the autoconf test checks for TCP_CORK and TCP_NODELAY
> explicitly. I don't know whether FreeBSD has ever or does currently
> allow TCP_NOPUSH and TCP_NODELAY together, but it would be easy enough
> to adapt the code appropriately if someone tested that stuff out.
Ah, right. So, the autoconf test will just fail on FreeBSD/Darwin due to
TCP_CORK not being present - thus, having no real change to that platform. If
we can figure out what FreeBSD/Darwin can do, we can then write some m4-fu to
try to optimize it if desired.
Isn't there a test program somewhere that I could use to see what it'll do in
this scenario?
>> FWIW, I just couldn't make sense of the way that diff split up the context
>> without applying the patch locally and reading the new function. -- justin
>
> Ah, sorry; can you see any way to clean the function up or is it just
> that the diff is going to be confusing either way?
Not that I can tell. The unified diff algorithm just breaks down horribly in
this particular situation. Oh, well. No biggie.
> Thanks for the review!
Thank you! (Reviewing is the 'easy' part.) -- justin
Re: TCP_CORK in Linux 2.6: not broken, but it doesn't work with TCP_NODELAY
Posted by Joe Orton <jo...@redhat.com>.
On Tue, Aug 23, 2005 at 12:25:55PM -0700, Justin Erenkrantz wrote:
> --On August 23, 2005 4:51:52 PM +0100 Joe Orton <jo...@redhat.com> wrote:
>
> >Since 2.6 can enable both TCP_CORK and TCP_NODELAY at the same time, and
> >TCP_CORK will take precedence, APR can save two syscalls in the critical
> >path. Objections?
>
> Looks fine. One question: how does this apply to other OSes which also
> have TCP_NOPUSH and TCP_NODELAY (such as FreeBSD)?
The patch won't (shouldn't ;) make any difference to non-Linux
platforms, since the autoconf test checks for TCP_CORK and TCP_NODELAY
explicitly. I don't know whether FreeBSD has ever or does currently
allow TCP_NOPUSH and TCP_NODELAY together, but it would be easy enough
to adapt the code appropriately if someone tested that stuff out.
> FWIW, I just couldn't make sense of the way that diff split up the context
> without applying the patch locally and reading the new function. -- justin
Ah, sorry; can you see any way to clean the function up or is it just
that the diff is going to be confusing either way?
Thanks for the review!
joe
Re: TCP_CORK in Linux 2.6: not broken, but it doesn't work with
TCP_NODELAY
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On August 23, 2005 4:51:52 PM +0100 Joe Orton <jo...@redhat.com> wrote:
> Since 2.6 can enable both TCP_CORK and TCP_NODELAY at the same time, and
> TCP_CORK will take precedence, APR can save two syscalls in the critical
> path. Objections?
Looks fine. One question: how does this apply to other OSes which also
have TCP_NOPUSH and TCP_NODELAY (such as FreeBSD)?
FWIW, I just couldn't make sense of the way that diff split up the context
without applying the patch locally and reading the new function. -- justin
Re: TCP_CORK in Linux 2.6: not broken, but it doesn't work with TCP_NODELAY
Posted by Joe Orton <jo...@redhat.com>.
On Mon, Aug 22, 2005 at 03:19:58PM +0100, Joe Orton wrote:
> On Mon, Aug 22, 2005 at 02:52:38PM +0100, Joe Orton wrote:
> > With the FC4 2.6.12-ish kernel I am seeing:
> >
> > - stuff sent by httpd fails to cork properly; partial frames are sent
> > ... i.e. what Greg reported.
> >
> > - stuff sent by your test case *does* get corked properly; the partial
> > header frame is suppressed; using "--cork --nonblock", which *should* be
> > equivalent to what httpd does.
> >
> > so I think there is something subtle here.
>
> It looks like the fact that TCP_NODELAY was ever set to 1 on the socket
> is triggering the problem, even if it is set to 0 at the time TCP_CORK
> is set.
Dave Miller confirmed this and there's a fix working its way through.
Meanwhile...
Since 2.6 can enable both TCP_CORK and TCP_NODELAY at the same time, and
TCP_CORK will take precedence, APR can save two syscalls in the critical
path. Objections?
Index: network_io/unix/sockopt.c
===================================================================
--- network_io/unix/sockopt.c (revision 239389)
+++ network_io/unix/sockopt.c (working copy)
@@ -242,7 +242,13 @@
break;
case APR_TCP_NOPUSH:
#if APR_TCP_NOPUSH_FLAG
+ /* TCP_NODELAY and TCP_CORK are mutually exclusive on Linux
+ * kernels < 2.6; on newer kernels they can be used together
+ * and TCP_CORK takes preference, which is the desired
+ * behaviour. On older kernels, TCP_NODELAY must be toggled
+ * to "off" whilst TCP_CORK is in effect. */
if (apr_is_option_set(sock, APR_TCP_NOPUSH) != on) {
+#ifndef HAVE_TCP_NODELAY_WITH_CORK
int optlevel = IPPROTO_TCP;
int optname = TCP_NODELAY;
@@ -253,11 +259,9 @@
}
#endif
/* OK we're going to change some settings here... */
- /* TCP_NODELAY is mutually exclusive, so do we have it set? */
if (apr_is_option_set(sock, APR_TCP_NODELAY) == 1 && on) {
- /* If we want to set NOPUSH then if we have the TCP_NODELAY
- * flag set we need to switch it off...
- */
+ /* Now toggle TCP_NODELAY to off, if TCP_CORK is being
+ * turned on: */
int tmpflag = 0;
if (setsockopt(sock->socketdes, optlevel, optname,
(void*)&tmpflag, sizeof(int)) == -1) {
@@ -268,13 +272,19 @@
} else if (on) {
apr_set_option(sock, APR_RESET_NODELAY, 0);
}
+#endif /* HAVE_TCP_NODELAY_WITH_CORK */
+
/* OK, now we can just set the TCP_NOPUSH flag accordingly...*/
if (setsockopt(sock->socketdes, IPPROTO_TCP, APR_TCP_NOPUSH_FLAG,
(void*)&on, sizeof(int)) == -1) {
return errno;
}
apr_set_option(sock, APR_TCP_NOPUSH, on);
+#ifndef HAVE_TCP_NODELAY_WITH_CORK
if (!on && apr_is_option_set(sock, APR_RESET_NODELAY)) {
+ /* Now, if turning TCP_CORK was just turned off, turn
+ * TCP_NODELAY back on again if it was earlier toggled
+ * to off: */
int tmpflag = 1;
if (setsockopt(sock->socketdes, optlevel, optname,
(void*)&tmpflag, sizeof(int)) == -1) {
@@ -283,6 +293,7 @@
apr_set_option(sock, APR_RESET_NODELAY,0);
apr_set_option(sock, APR_TCP_NODELAY, 1);
}
+#endif /* HAVE_TCP_NODELAY_WITH_CORK */
}
#else
return APR_ENOTIMPL;
Index: configure.in
===================================================================
--- configure.in (revision 239389)
+++ configure.in (working copy)
@@ -1875,6 +1875,7 @@
APR_CHECK_TCP_NODELAY_INHERITED
APR_CHECK_O_NONBLOCK_INHERITED
+APR_CHECK_TCP_NODELAY_WITH_CORK
# Look for a way of corking TCP...
APR_CHECK_DEFINE(TCP_CORK, netinet/tcp.h)
Index: build/apr_network.m4
===================================================================
--- build/apr_network.m4 (revision 239389)
+++ build/apr_network.m4 (working copy)
@@ -368,6 +368,60 @@
])
dnl
+dnl Determine whether TCP_NODELAY and TCP_CORK can both be set
+dnl on a TCP socket.
+dnl
+AC_DEFUN([APR_CHECK_TCP_NODELAY_WITH_CORK], [
+AC_CACHE_CHECK([whether TCP_NODELAY and TCP_CORK can both be enabled],
+[apr_cv_tcp_nodelay_with_cork],
+[AC_RUN_IFELSE([AC_LANG_PROGRAM([[
+#ifdef HAVE_SYS_TYPES_H
+#include <sys/types.h>
+#endif
+#ifdef HAVE_SYS_SOCKET_H
+#include <sys/socket.h>
+#endif
+#ifdef HAVE_NETINET_IN_H
+#include <netinet/in.h>
+#endif
+#ifdef HAVE_NETINET_TCP_H
+#include <netinet/tcp.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+]], [[
+ int fd, flag, rc;
+
+ fd = socket(AF_INET, SOCK_STREAM, 0);
+ if (fd < 0) {
+ exit(1);
+ }
+
+ flag = 1;
+ rc = setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof flag);
+ if (rc < 0) {
+ perror("setsockopt TCP_NODELAY");
+ exit(2);
+ }
+
+ flag = 1;
+ rc = setsockopt(fd, IPPROTO_TCP, TCP_CORK, &flag, sizeof flag);
+ if (rc < 0) {
+ perror("setsockopt TCP_CORK");
+ exit(3);
+ }
+
+ exit(0);
+]])], [apr_cv_tcp_nodelay_with_cork=yes], [apr_cv_tcp_nodelay_with_cork=no])])
+
+if test "$apr_cv_tcp_nodelay_with_cork" = "yes"; then
+ AC_DEFINE([HAVE_TCP_NODELAY_WITH_CORK], 1,
+ [Define if TCP_NODELAY and TCP_CORK can be enabled at the same time])
+fi
+])
+
+
+dnl
dnl see if O_NONBLOCK setting is inherited from listening sockets
dnl
AC_DEFUN(APR_CHECK_O_NONBLOCK_INHERITED,[
Re: TCP_CORK in Linux 2.6: not broken, but it doesn't work with TCP_NODELAY
Posted by Joe Orton <jo...@redhat.com>.
On Mon, Aug 22, 2005 at 03:19:58PM +0100, Joe Orton wrote:
> On Mon, Aug 22, 2005 at 02:52:38PM +0100, Joe Orton wrote:
> > With the FC4 2.6.12-ish kernel I am seeing:
> >
> > - stuff sent by httpd fails to cork properly; partial frames are sent
> > ... i.e. what Greg reported.
> >
> > - stuff sent by your test case *does* get corked properly; the partial
> > header frame is suppressed; using "--cork --nonblock", which *should* be
> > equivalent to what httpd does.
> >
> > so I think there is something subtle here.
>
> It looks like the fact that TCP_NODELAY was ever set to 1 on the socket
> is triggering the problem, even if it is set to 0 at the time TCP_CORK
> is set.
Dave Miller confirmed this and there's a fix working its way through.
Meanwhile...
Since 2.6 can enable both TCP_CORK and TCP_NODELAY at the same time, and
TCP_CORK will take precedence, APR can save two syscalls in the critical
path. Objections?
Index: network_io/unix/sockopt.c
===================================================================
--- network_io/unix/sockopt.c (revision 239389)
+++ network_io/unix/sockopt.c (working copy)
@@ -242,7 +242,13 @@
break;
case APR_TCP_NOPUSH:
#if APR_TCP_NOPUSH_FLAG
+ /* TCP_NODELAY and TCP_CORK are mutually exclusive on Linux
+ * kernels < 2.6; on newer kernels they can be used together
+ * and TCP_CORK takes preference, which is the desired
+ * behaviour. On older kernels, TCP_NODELAY must be toggled
+ * to "off" whilst TCP_CORK is in effect. */
if (apr_is_option_set(sock, APR_TCP_NOPUSH) != on) {
+#ifndef HAVE_TCP_NODELAY_WITH_CORK
int optlevel = IPPROTO_TCP;
int optname = TCP_NODELAY;
@@ -253,11 +259,9 @@
}
#endif
/* OK we're going to change some settings here... */
- /* TCP_NODELAY is mutually exclusive, so do we have it set? */
if (apr_is_option_set(sock, APR_TCP_NODELAY) == 1 && on) {
- /* If we want to set NOPUSH then if we have the TCP_NODELAY
- * flag set we need to switch it off...
- */
+ /* Now toggle TCP_NODELAY to off, if TCP_CORK is being
+ * turned on: */
int tmpflag = 0;
if (setsockopt(sock->socketdes, optlevel, optname,
(void*)&tmpflag, sizeof(int)) == -1) {
@@ -268,13 +272,19 @@
} else if (on) {
apr_set_option(sock, APR_RESET_NODELAY, 0);
}
+#endif /* HAVE_TCP_NODELAY_WITH_CORK */
+
/* OK, now we can just set the TCP_NOPUSH flag accordingly...*/
if (setsockopt(sock->socketdes, IPPROTO_TCP, APR_TCP_NOPUSH_FLAG,
(void*)&on, sizeof(int)) == -1) {
return errno;
}
apr_set_option(sock, APR_TCP_NOPUSH, on);
+#ifndef HAVE_TCP_NODELAY_WITH_CORK
if (!on && apr_is_option_set(sock, APR_RESET_NODELAY)) {
+ /* Now, if turning TCP_CORK was just turned off, turn
+ * TCP_NODELAY back on again if it was earlier toggled
+ * to off: */
int tmpflag = 1;
if (setsockopt(sock->socketdes, optlevel, optname,
(void*)&tmpflag, sizeof(int)) == -1) {
@@ -283,6 +293,7 @@
apr_set_option(sock, APR_RESET_NODELAY,0);
apr_set_option(sock, APR_TCP_NODELAY, 1);
}
+#endif /* HAVE_TCP_NODELAY_WITH_CORK */
}
#else
return APR_ENOTIMPL;
Index: configure.in
===================================================================
--- configure.in (revision 239389)
+++ configure.in (working copy)
@@ -1875,6 +1875,7 @@
APR_CHECK_TCP_NODELAY_INHERITED
APR_CHECK_O_NONBLOCK_INHERITED
+APR_CHECK_TCP_NODELAY_WITH_CORK
# Look for a way of corking TCP...
APR_CHECK_DEFINE(TCP_CORK, netinet/tcp.h)
Index: build/apr_network.m4
===================================================================
--- build/apr_network.m4 (revision 239389)
+++ build/apr_network.m4 (working copy)
@@ -368,6 +368,60 @@
])
dnl
+dnl Determine whether TCP_NODELAY and TCP_CORK can both be set
+dnl on a TCP socket.
+dnl
+AC_DEFUN([APR_CHECK_TCP_NODELAY_WITH_CORK], [
+AC_CACHE_CHECK([whether TCP_NODELAY and TCP_CORK can both be enabled],
+[apr_cv_tcp_nodelay_with_cork],
+[AC_RUN_IFELSE([AC_LANG_PROGRAM([[
+#ifdef HAVE_SYS_TYPES_H
+#include <sys/types.h>
+#endif
+#ifdef HAVE_SYS_SOCKET_H
+#include <sys/socket.h>
+#endif
+#ifdef HAVE_NETINET_IN_H
+#include <netinet/in.h>
+#endif
+#ifdef HAVE_NETINET_TCP_H
+#include <netinet/tcp.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+]], [[
+ int fd, flag, rc;
+
+ fd = socket(AF_INET, SOCK_STREAM, 0);
+ if (fd < 0) {
+ exit(1);
+ }
+
+ flag = 1;
+ rc = setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof flag);
+ if (rc < 0) {
+ perror("setsockopt TCP_NODELAY");
+ exit(2);
+ }
+
+ flag = 1;
+ rc = setsockopt(fd, IPPROTO_TCP, TCP_CORK, &flag, sizeof flag);
+ if (rc < 0) {
+ perror("setsockopt TCP_CORK");
+ exit(3);
+ }
+
+ exit(0);
+]])], [apr_cv_tcp_nodelay_with_cork=yes], [apr_cv_tcp_nodelay_with_cork=no])])
+
+if test "$apr_cv_tcp_nodelay_with_cork" = "yes"; then
+ AC_DEFINE([HAVE_TCP_NODELAY_WITH_CORK], 1,
+ [Define if TCP_NODELAY and TCP_CORK can be enabled at the same time])
+fi
+])
+
+
+dnl
dnl see if O_NONBLOCK setting is inherited from listening sockets
dnl
AC_DEFUN(APR_CHECK_O_NONBLOCK_INHERITED,[
Re: TCP_CORK in Linux 2.6: not broken, but it doesn't work with TCP_NODELAY
Posted by Joe Orton <jo...@redhat.com>.
On Mon, Aug 22, 2005 at 02:52:38PM +0100, Joe Orton wrote:
> With the FC4 2.6.12-ish kernel I am seeing:
>
> - stuff sent by httpd fails to cork properly; partial frames are sent
> ... i.e. what Greg reported.
>
> - stuff sent by your test case *does* get corked properly; the partial
> header frame is suppressed; using "--cork --nonblock", which *should* be
> equivalent to what httpd does.
>
> so I think there is something subtle here.
It looks like the fact that TCP_NODELAY was ever set to 1 on the socket
is triggering the problem, even if it is set to 0 at the time TCP_CORK
is set.
Modified version of your test case attached; passing --toggle-nodelay
sets TCP_NODELAY to 1 and then back to 0, which should be equivalent to
never setting it at all but isn't. I've reported this to our kernel
guys.
joe
Re: TCP_CORK in Linux 2.6: not broken, but it doesn't work with TCP_NODELAY
Posted by Joe Orton <jo...@redhat.com>.
On Sun, Aug 21, 2005 at 04:25:19PM -0700, Brian Pane wrote:
> I've attached my test program in case it's useful to others. It's
> run as
> sendfile_test [--cork] [--nodelay] [--nonblock] filename
> listener_port_number.
This is rather useful, thanks Brian.
I think you're missing the fact that 2.0 never actually tries to enable
both TCP_CORK *and* TCP_NODELAY - this wasn't possible with a 2.4
kernel, the second setsockopt call would fail. APR knows not to try
this and toggles NODELAY off when CORK is enabled.
With the FC4 2.6.12-ish kernel I am seeing:
- stuff sent by httpd fails to cork properly; partial frames are sent
... i.e. what Greg reported.
- stuff sent by your test case *does* get corked properly; the partial
header frame is suppressed; using "--cork --nonblock", which *should* be
equivalent to what httpd does.
so I think there is something subtle here.
Comments in the 2.6 kernel source imply that TCP_CORK and TCP_NODELAY
should actually be usable together than that TCP_CORK would trump
TCP_NODELAY. But this isn't what httpd is trying, so maybe should be
treated as a separate issue for now.
joe
Re: TCP_CORK in Linux 2.6: not broken, but it doesn't work with TCP_NODELAY
Posted by Greg Ames <gr...@remulak.net>.
Brian Pane wrote:
> Rather than automatically setting TCP_NODELAY in core_pre_connection(),
> perhaps we should set it conditionally in core_output_filter(), where
> we have
> enough information to tell whether it's needed.
I'm +1 on the concept of being more lazy about setting the sockopts. the way we
do it now on Linux is pretty silly. we set it one way, discover we're doing
sendfile and change our minds and flip it the other way, then set another
option, then undo that and go back to step 1. these syscalls seem cheap on
Linux but half of them could be eliminated. doing so might bypass the 2.6 cork
bug.
Greg