Network latency on Cassandra 0.7 (TFramedTransport)

[Michal Augustýn <au...@gmail.com>]

Hello,

I'm experiencing big network latency when using TFramedTransport.

The latency is about 200 ms on every request when I'm connected to another
computer. On localhost, all goes well.

I can now solve this issue by changing "thrift_framed_transport_size_in_mb"
to 0 (so disable framed transport) and use TBufferedTransport (then all is
ok). But I know that this is deprecated scenario...

This conversation deals with the same problem (but I cannot use the
solution):
http://www.mail-archive.com/user@cassandra.apache.org/msg01966.html

Cassandra version: 0.7 (build from trunk)
Thrift: recommended revision (959516)
Client: C# on Windows7
Server OS: Windows 2008
Cassandra Configuration (yaml): default ("data_file_directories",
"commitlog_directory" and "rpc_address" changed only)


Super-simple C# example:

var sw = Stopwatch.StartNew();
var coreTransport = new Thrift.Transport.TSocket("some.server", 9160);
var transport = new Thrift.Transport.TFramedTransport(coreTransport);
var protocol = new Thrift.Protocol.TBinaryProtocol(transport);
var client = new Apache.Cassandra.Cassandra.Client(protocol);
transport.Open();
sw.Stop();
Console.WriteLine("Connection: {0} ms", sw.ElapsedMilliseconds);

sw = Stopwatch.StartNew();
var v = client.describe_cluster_name();
sw.Stop();
Console.WriteLine("Time: {0} ms", sw.ElapsedMilliseconds);
Console.WriteLine("Value: {0}", v);
Console.Read();

Maybe I'm using Thrift in bad way.

Thank you.

Augi

Re: Network latency on Cassandra 0.7 (TFramedTransport)

[Michal Augustýn <au...@gmail.com>]

Thank you very much. This solves my issue.

Augi

2010/9/17 Michael Greene <mi...@gmail.com>

> This is the correct cause.  Reproducing your test gives 38-45ms in each of
> 10 runs.  If you run a profiler against it, you can see that the time is
> entirely spent blocking on receive in TStreamTransport.Read.
>
> Your test can be modified with the following line:
> coreTransport.TcpClient.NoDelay = true;
>
> After disabling Nagle's each test runs in < 1ms.
>
> Michael
>
> On Fri, Sep 17, 2010 at 9:57 AM, Peter Schuller <
> peter.schuller@infidyne.com> wrote:
>
>> > The latency is about 200 ms on every request when I'm connected to
>> another
>>
>> My first thought here was that maybe you're seeing the effects of
>> nagle[1] + delayed acks on the other side. On Unix, normally something
>> like a thrift client would set TCP_NODELAY on its socket to avoid the
>> problem. I'm not sure whether or not this is a reasonable hypothesis
>> with C# + Thrift on Windows. Sorry, I don't know what the easy fix is
>> either but perhaps it might help dig further.
>>
>> [1] http://en.wikipedia.org/wiki/Nagle's_algorithm
>>
>> --
>> / Peter Schuller
>>
>
>

Re: Network latency on Cassandra 0.7 (TFramedTransport)

[Jonathan Ellis <jb...@gmail.com>]

Created https://issues.apache.org/jira/browse/THRIFT-904

On Fri, Sep 17, 2010 at 12:16 PM, Michael Greene
<mi...@gmail.com> wrote:
> This is the correct cause.  Reproducing your test gives 38-45ms in each of
> 10 runs.  If you run a profiler against it, you can see that the time is
> entirely spent blocking on receive in TStreamTransport.Read.
> Your test can be modified with the following line:
> coreTransport.TcpClient.NoDelay = true;
> After disabling Nagle's each test runs in < 1ms.
> Michael
> On Fri, Sep 17, 2010 at 9:57 AM, Peter Schuller
> <pe...@infidyne.com> wrote:
>>
>> > The latency is about 200 ms on every request when I'm connected to
>> > another
>>
>> My first thought here was that maybe you're seeing the effects of
>> nagle[1] + delayed acks on the other side. On Unix, normally something
>> like a thrift client would set TCP_NODELAY on its socket to avoid the
>> problem. I'm not sure whether or not this is a reasonable hypothesis
>> with C# + Thrift on Windows. Sorry, I don't know what the easy fix is
>> either but perhaps it might help dig further.
>>
>> [1] http://en.wikipedia.org/wiki/Nagle's_algorithm
>>
>> --
>> / Peter Schuller
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: Network latency on Cassandra 0.7 (TFramedTransport)

[Michael Greene <mi...@gmail.com>]

This is the correct cause.  Reproducing your test gives 38-45ms in each of
10 runs.  If you run a profiler against it, you can see that the time is
entirely spent blocking on receive in TStreamTransport.Read.

Your test can be modified with the following line:
coreTransport.TcpClient.NoDelay = true;

After disabling Nagle's each test runs in < 1ms.

Michael

On Fri, Sep 17, 2010 at 9:57 AM, Peter Schuller <peter.schuller@infidyne.com
> wrote:

> > The latency is about 200 ms on every request when I'm connected to
> another
>
> My first thought here was that maybe you're seeing the effects of
> nagle[1] + delayed acks on the other side. On Unix, normally something
> like a thrift client would set TCP_NODELAY on its socket to avoid the
> problem. I'm not sure whether or not this is a reasonable hypothesis
> with C# + Thrift on Windows. Sorry, I don't know what the easy fix is
> either but perhaps it might help dig further.
>
> [1] http://en.wikipedia.org/wiki/Nagle's_algorithm
>
> --
> / Peter Schuller
>

Re: Network latency on Cassandra 0.7 (TFramedTransport)

[Peter Schuller <pe...@infidyne.com>]

> The latency is about 200 ms on every request when I'm connected to another

My first thought here was that maybe you're seeing the effects of
nagle[1] + delayed acks on the other side. On Unix, normally something
like a thrift client would set TCP_NODELAY on its socket to avoid the
problem. I'm not sure whether or not this is a reasonable hypothesis
with C# + Thrift on Windows. Sorry, I don't know what the easy fix is
either but perhaps it might help dig further.

[1] http://en.wikipedia.org/wiki/Nagle's_algorithm

-- 
/ Peter Schuller