You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by Erik Frey <er...@last.fm> on 2008/06/12 12:18:54 UTC
strange network latency... something about TSocket?
Hi All,
I'm trying to use thrift in a low-latency scenario. Even at low loads (just a
few hundred requests a second) occasionally (a few times a second) a round
trip from the client to the server takes 250 milliseconds. I'm using the
default TSocket settings, so it's probably not nodelay, and I'm using
TNonblockingServer, using 20080411p1 from the web site.
Just curious if anyone's seen this before?
Interestingly, when I measure the round trip time on the server end (inserting
timing code before and after processor->process(in, out)), it's what I would
expect. The delay seems to be on the client side.
Erik
Re: strange network latency... something about TSocket?
Posted by Erik Frey <er...@last.fm>.
Turned out to be a hardware issue. Thanks guys.
Erik
On Thursday 12 June 2008 20:17:35 Mark Slee wrote:
> Also, which client language are you using? It's possible that you may be
> using a TSocket without buffering -- depending upon the language you
> probably want to wrap the Tsocket in a TBufferedTransport to avoid lots
> of syscalls to incrementally read data off the socket -- rather than out
> of a decently sized buffer in application space.
>
> -----Original Message-----
> From: David Reiss [mailto:dreiss@facebook.com]
> Sent: Thursday, June 12, 2008 11:05 AM
> To: thrift-dev@incubator.apache.org
> Subject: Re: strange network latency... something about TSocket?
>
> Dropped TCP packets maybe? I've heard the retransmit timeout is about
> 200 ms. I think you can get retransmit counts from the kernel with
> netstat.
>
> --David
>
> Erik Frey wrote:
> > Hi All,
> >
> > I'm trying to use thrift in a low-latency scenario. Even at low loads
> >
> > (just a few hundred requests a second) occasionally (a few times a
> > second) a round trip from the client to the server takes 250
> > milliseconds. I'm using the default TSocket settings, so it's
> > probably not nodelay, and I'm using TNonblockingServer, using
> > 20080411p1 from the web site.
> >
> > Just curious if anyone's seen this before?
> >
> > Interestingly, when I measure the round trip time on the server end
> > (inserting timing code before and after processor->process(in, out)),
> > it's what I would expect. The delay seems to be on the client side.
> >
> > Erik
RE: strange network latency... something about TSocket?
Posted by Mark Slee <ms...@facebook.com>.
Also, which client language are you using? It's possible that you may be
using a TSocket without buffering -- depending upon the language you
probably want to wrap the Tsocket in a TBufferedTransport to avoid lots
of syscalls to incrementally read data off the socket -- rather than out
of a decently sized buffer in application space.
-----Original Message-----
From: David Reiss [mailto:dreiss@facebook.com]
Sent: Thursday, June 12, 2008 11:05 AM
To: thrift-dev@incubator.apache.org
Subject: Re: strange network latency... something about TSocket?
Dropped TCP packets maybe? I've heard the retransmit timeout is about
200 ms. I think you can get retransmit counts from the kernel with
netstat.
--David
Erik Frey wrote:
> Hi All,
>
> I'm trying to use thrift in a low-latency scenario. Even at low loads
> (just a few hundred requests a second) occasionally (a few times a
> second) a round trip from the client to the server takes 250
> milliseconds. I'm using the default TSocket settings, so it's
> probably not nodelay, and I'm using TNonblockingServer, using
> 20080411p1 from the web site.
>
> Just curious if anyone's seen this before?
>
> Interestingly, when I measure the round trip time on the server end
> (inserting timing code before and after processor->process(in, out)),
> it's what I would expect. The delay seems to be on the client side.
>
> Erik
>
Re: strange network latency... something about TSocket?
Posted by David Reiss <dr...@facebook.com>.
Dropped TCP packets maybe? I've heard the retransmit timeout is about
200 ms. I think you can get retransmit counts from the kernel with
netstat.
--David
Erik Frey wrote:
> Hi All,
>
> I'm trying to use thrift in a low-latency scenario. Even at low loads
> (just a
> few hundred requests a second) occasionally (a few times a second) a round
> trip from the client to the server takes 250 milliseconds. I'm using the
> default TSocket settings, so it's probably not nodelay, and I'm using
> TNonblockingServer, using 20080411p1 from the web site.
>
> Just curious if anyone's seen this before?
>
> Interestingly, when I measure the round trip time on the server end
> (inserting
> timing code before and after processor->process(in, out)), it's what I would
> expect. The delay seems to be on the client side.
>
> Erik
>