Better TCP performance for dialup customers...

["Philip A. Prindeville" <ph...@enteka.com>]

[ Beware: there be "jargon" here!  If you are a networking
  person with some knowledge of TCP mechanics, you might
  find this useful.  Otherwise, I apologize for the posting
  as it is a little incidental to this list... ]

By the way, for all of you sites that have Solaris 2.3 to 2.5.1
installed and get a lot of hits from dialup clients, you can
seriously improve performance (and eliminate wasted bandwidth
on your precious T-1 [or T-3 for the lucky bastards] connection)
by installing Sun patches 103582-xx and 103630-xx, as well as
installing the following file:

#
# /etc/rc2.d/S60tweak
#
# set the TCP initial RTT estimate to be something less braindead.
#

ndd -set /dev/tcp tcp_rexmit_interval_initial 5000

which increases the TCP initial round-trip time to 5 seconds
from 500 milliseconds.  Otherwise, if someone is calling from a
overbooked ISP, they are likely to easily exceed the 500ms
RTT on the first packet.  TCP will timeout, and retransmit the
first packet (while the original copy is still on the wire).
The whole thing cascades, causing excessive retransmissions,
slowing things down further, meanwhile TCP thinks packets are
being dropped and keeps the congestion window at 1, so you
never have more than a packet-full of data on the wire (though
often that same packet multiple times!).  Because Solaris is
so broken, it also doesn't exponentially increase its RTT
estimate as it is supposed to, so you stay in this pathetic
state for the first 50-80 packets sent (depending on how high
the RTT really is).  Note: for a good many transfers on an
HTTP server (thumbnail GIFs, etc.), the connection never
lasts long enough to get the congestion window opened up to
full size (or even hit 2 packets!).  We saw cases where for
anything less than a 24K file, the congestion window was
staying at 1 packet for the whole transfer.

With a 3.5 - 5 second RTT (over packet radio), the transfer
crawled.  Once we increased the initial RTT and installed these
patches, we started seeing 3-8 times better performance for
the same radio modems.

One last note:  for non-dialup customers, when the first packet
comes back from the remote end (in 100 - 200ms), your RTT is
immediately set to that, so there is no negative impact when
you raise the initial RTT estimate.

The only time things are bad is when the first packet is
dropped (due to an ARP expiration or congestion), and you have
to retransmit the packet 5 seconds later.  In the case of
congestion, there isn't much you can do.  And ARP data is valid
for 4 hours, so this isn't a regular occurence anyway.

If you are wondering why Sun did this by the way, so is everyone
else.  The best explanation I heard was that they were trying to
squeeze a few extra percentage points utilisation out of their
host/host transfer rates when two machines are connected to the
same LAN.  In my opinion, they made the tradeoff in the wrong
area...  The impact on WAN performance is catastrophic.

-Philip

P.S.	Even fractional T-1 frame-relay customers with traffic
	shaping (i.e. "smoothing") can experience this
	pathological behaviour since smoothing can cause your
	RTT to momentarily hit 200ms or more.  Smoothing is
	when you insert small delays between back-to-back
	packets so as to not exceed a predefined threshold in
	traffic "burstiness" that you and your ISP agree to
	when you sign the contract.  Fair-queueing also can
	provoke this.  Ciscos do fair-queueing on serial
	synchronous interfaces by default...  Don't ask me to
	explain fair-queueing -- it would take too long.

Re: Better TCP performance for dialup customers...

[Dean Gaudet <dg...@arctic.org>]

... this is fixed with the latest tcp/ip patches for 2.5.1.  At least I'm
pretty sure it is.

Dean

On Thu, 4 Sep 1997, Philip A. Prindeville wrote:

> [ Beware: there be "jargon" here!  If you are a networking
>   person with some knowledge of TCP mechanics, you might
>   find this useful.  Otherwise, I apologize for the posting
>   as it is a little incidental to this list... ]
> 
> By the way, for all of you sites that have Solaris 2.3 to 2.5.1
> installed and get a lot of hits from dialup clients, you can
> seriously improve performance (and eliminate wasted bandwidth
> on your precious T-1 [or T-3 for the lucky bastards] connection)
> by installing Sun patches 103582-xx and 103630-xx, as well as
> installing the following file:
> 
> #
> # /etc/rc2.d/S60tweak
> #
> # set the TCP initial RTT estimate to be something less braindead.
> #
> 
> ndd -set /dev/tcp tcp_rexmit_interval_initial 5000
> 
> which increases the TCP initial round-trip time to 5 seconds
> from 500 milliseconds.  Otherwise, if someone is calling from a
> overbooked ISP, they are likely to easily exceed the 500ms
> RTT on the first packet.  TCP will timeout, and retransmit the
> first packet (while the original copy is still on the wire).
> The whole thing cascades, causing excessive retransmissions,
> slowing things down further, meanwhile TCP thinks packets are
> being dropped and keeps the congestion window at 1, so you
> never have more than a packet-full of data on the wire (though
> often that same packet multiple times!).  Because Solaris is
> so broken, it also doesn't exponentially increase its RTT
> estimate as it is supposed to, so you stay in this pathetic
> state for the first 50-80 packets sent (depending on how high
> the RTT really is).  Note: for a good many transfers on an
> HTTP server (thumbnail GIFs, etc.), the connection never
> lasts long enough to get the congestion window opened up to
> full size (or even hit 2 packets!).  We saw cases where for
> anything less than a 24K file, the congestion window was
> staying at 1 packet for the whole transfer.
> 
> With a 3.5 - 5 second RTT (over packet radio), the transfer
> crawled.  Once we increased the initial RTT and installed these
> patches, we started seeing 3-8 times better performance for
> the same radio modems.
> 
> One last note:  for non-dialup customers, when the first packet
> comes back from the remote end (in 100 - 200ms), your RTT is
> immediately set to that, so there is no negative impact when
> you raise the initial RTT estimate.
> 
> The only time things are bad is when the first packet is
> dropped (due to an ARP expiration or congestion), and you have
> to retransmit the packet 5 seconds later.  In the case of
> congestion, there isn't much you can do.  And ARP data is valid
> for 4 hours, so this isn't a regular occurence anyway.
> 
> If you are wondering why Sun did this by the way, so is everyone
> else.  The best explanation I heard was that they were trying to
> squeeze a few extra percentage points utilisation out of their
> host/host transfer rates when two machines are connected to the
> same LAN.  In my opinion, they made the tradeoff in the wrong
> area...  The impact on WAN performance is catastrophic.
> 
> -Philip
> 
> P.S.	Even fractional T-1 frame-relay customers with traffic
> 	shaping (i.e. "smoothing") can experience this
> 	pathological behaviour since smoothing can cause your
> 	RTT to momentarily hit 200ms or more.  Smoothing is
> 	when you insert small delays between back-to-back
> 	packets so as to not exceed a predefined threshold in
> 	traffic "burstiness" that you and your ISP agree to
> 	when you sign the contract.  Fair-queueing also can
> 	provoke this.  Ciscos do fair-queueing on serial
> 	synchronous interfaces by default...  Don't ask me to
> 	explain fair-queueing -- it would take too long.
>