You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Даниел Симеонов <ds...@gmail.com> on 2010/05/21 18:16:52 UTC
Cassandra thrift question
Hi,
I have a question about the thrift protocol used to connect to Cassandra,
I saw in class CassandraDaemon that TServerSocket is being used,
why TNonblockingServerSocket is not being used? Thank you very much!
Best regards, Daniel.
Re: Cassandra thrift question
Posted by Carlos Alvarez <cb...@gmail.com>.
On Mon, May 24, 2010 at 7:43 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> with C# you need to be sure to tell thrift to use client-side
> buffering. http://wiki.apache.org/cassandra/ThriftExamples#C.23 shows
> this (but didn't until recently)
Yes, I am unsing TBufferedTransport. However the high times continues.
When I run a load test (not against cassandra directly but against the
client application which uses cassandra) I see a lower number of
anomalies.
I see the slowness in org.apache.cassandra.thrift.Column when the
column value is read throught a TProtocol.readBinary. The column value
are somewhat big (10k) but I think 2s is still too much.
I am not pooling at all (I had to go productive before ending my
pooling client), so I have a lot of sockets open per server (I use a
threadlocal conection). Also I have period of very low activity and on
those periods I see the higher number of anomalies. I suspect windows
is messing up a with a lot of open sockets unused by long periods of
time since I don't see any other troubles in the network.
In the next days I will deploy the pooling client, so will test my hypothesis.
Carlos.
Re: Cassandra thrift question
Posted by Jonathan Ellis <jb...@gmail.com>.
with C# you need to be sure to tell thrift to use client-side
buffering. http://wiki.apache.org/cassandra/ThriftExamples#C.23 shows
this (but didn't until recently)
On Mon, May 24, 2010 at 4:09 PM, Carlos Alvarez <cb...@gmail.com> wrote:
> I have the same issue in my cluster: 0,5% of requests are extremely
> slow because the time it takes to read the data from the socket.
>
> However in my case it is not related to the load. Actually the
> percentage of anomalies drop as the load increases.
>
> On the other hand the nio is actually slow than blocking io when you
> have enough resources to handle all connections (which is my situation
> in the server).
>
> I tend to blame the client and my unoptimized client connection scheme
> ( c# client) which opens a lot of sockets and doenst pool at all.
>
> I am currently pinpointing this issue. I will let you know if find something.
>
>
> Carlos.e
>
>
> On 5/22/10, Даниел Симеонов <ds...@gmail.com> wrote:
>> Hi Jonathan,
>> This sounds a little bit strange, I would expect that NIO server sockets
>> to be more scalable than regular socket, well I don't have much experience
>> with tcp programming. What was the scenario you used? The problem I observe
>> is that most of the requests to Cassandra are quite performant (lets say
>> miliseconds) yet some requests (~ 0.2 % of all requests) are slow ~2-5
>> seconds. Profiling Cassandra showed that thrift socket read operations
>> contributed for most of the time spent.
>> Best regards, Daniel.
>>
>> 2010/5/22 Jonathan Ellis <jb...@gmail.com>
>>
>>> Because when we tested it, it was slower.
>>>
>>> 2010/5/21 Даниел Симеонов <ds...@gmail.com>:
>>> > Hi,
>>> > I have a question about the thrift protocol used to connect to
>>> Cassandra,
>>> > I saw in class CassandraDaemon that TServerSocket is being used,
>>> > why TNonblockingServerSocket is not being used? Thank you very much!
>>> > Best regards, Daniel.
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of Riptano, the source for professional Cassandra support
>>> http://riptano.com
>>>
>>
>
> --
> Sent from my mobile device
>
> Tal vez hubo un error en la grafía. O en la articulación del Sacro Nombre.
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Cassandra thrift question
Posted by Carlos Alvarez <cb...@gmail.com>.
I have the same issue in my cluster: 0,5% of requests are extremely
slow because the time it takes to read the data from the socket.
However in my case it is not related to the load. Actually the
percentage of anomalies drop as the load increases.
On the other hand the nio is actually slow than blocking io when you
have enough resources to handle all connections (which is my situation
in the server).
I tend to blame the client and my unoptimized client connection scheme
( c# client) which opens a lot of sockets and doenst pool at all.
I am currently pinpointing this issue. I will let you know if find something.
Carlos.e
On 5/22/10, Даниел Симеонов <ds...@gmail.com> wrote:
> Hi Jonathan,
> This sounds a little bit strange, I would expect that NIO server sockets
> to be more scalable than regular socket, well I don't have much experience
> with tcp programming. What was the scenario you used? The problem I observe
> is that most of the requests to Cassandra are quite performant (lets say
> miliseconds) yet some requests (~ 0.2 % of all requests) are slow ~2-5
> seconds. Profiling Cassandra showed that thrift socket read operations
> contributed for most of the time spent.
> Best regards, Daniel.
>
> 2010/5/22 Jonathan Ellis <jb...@gmail.com>
>
>> Because when we tested it, it was slower.
>>
>> 2010/5/21 Даниел Симеонов <ds...@gmail.com>:
>> > Hi,
>> > I have a question about the thrift protocol used to connect to
>> Cassandra,
>> > I saw in class CassandraDaemon that TServerSocket is being used,
>> > why TNonblockingServerSocket is not being used? Thank you very much!
>> > Best regards, Daniel.
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>>
>
--
Sent from my mobile device
Tal vez hubo un error en la grafía. O en la articulación del Sacro Nombre.
Re: Cassandra thrift question
Posted by Даниел Симеонов <ds...@gmail.com>.
Hi Jonathan,
This sounds a little bit strange, I would expect that NIO server sockets
to be more scalable than regular socket, well I don't have much experience
with tcp programming. What was the scenario you used? The problem I observe
is that most of the requests to Cassandra are quite performant (lets say
miliseconds) yet some requests (~ 0.2 % of all requests) are slow ~2-5
seconds. Profiling Cassandra showed that thrift socket read operations
contributed for most of the time spent.
Best regards, Daniel.
2010/5/22 Jonathan Ellis <jb...@gmail.com>
> Because when we tested it, it was slower.
>
> 2010/5/21 Даниел Симеонов <ds...@gmail.com>:
> > Hi,
> > I have a question about the thrift protocol used to connect to
> Cassandra,
> > I saw in class CassandraDaemon that TServerSocket is being used,
> > why TNonblockingServerSocket is not being used? Thank you very much!
> > Best regards, Daniel.
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: Cassandra thrift question
Posted by Jonathan Ellis <jb...@gmail.com>.
Because when we tested it, it was slower.
2010/5/21 Даниел Симеонов <ds...@gmail.com>:
> Hi,
> I have a question about the thrift protocol used to connect to Cassandra,
> I saw in class CassandraDaemon that TServerSocket is being used,
> why TNonblockingServerSocket is not being used? Thank you very much!
> Best regards, Daniel.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com