Re: Hintedhandoff will never complete when a BIG rowmutation

[Schubert Zhang <zs...@gmail.com>]

Yes, I think current HintedHandOff implementation in 0.6.x cannot support
large hints, it is a risk in a production system.


On Tue, Jun 29, 2010 at 12:31 AM, albert_e <do...@gmail.com> wrote:

> In 0.6.2, HH sending MUTATION message using the same OutboundTcpConnection
> with READ message. When HH transfering big mutation data, read operation
> will be blocked and read storm may cause 100% disk I/O of the dest node.
>
>
> 2010/6/28 Jonathan Ellis <jb...@gmail.com>
>
> Yes, you should increase your timeout if you are hinting big mutations
>> (or big rows that were built from smaller mutations).
>>
>> 2010/6/28 Lu Ming <xl...@live.com>:
>> > Hi:
>> >      These days I found my Cassandra is strange, much slower than
>> before.
>> > And I Spent much time to figure it out and today I got the answer.
>> >
>> >     Some bad buy keeps on writing many data day and night, then made a
>> very
>> > big row mutation which size is about 140M.
>> > In this period I restarted some Cassandra nodes, and when the nodes is
>> alive
>> > again, them got some hintedhandoff messages.
>> > HintedHandOffManager.sendMessage() will send the rowmutations to these
>> > nodes, but the rowmutation is too big to finish transferring in
>> > 8 seconds (defined in DatabaseDescriptor.getRpcTimeout()), and
>> sendMessage()
>> > return false when got a TimeoutException.
>> >
>> > Every one hour HintedHandOffManager will check hintedhandoff
>> ColumnFamily
>> > then send out the big rowmutations to alive nodes,
>> > It fails again because of the TimeoutException, so the task will never
>> > finish and the big rowmutation is sending again and again.
>> >
>> >    In multi-datacenters,  a big rowmutation can not be transferred
>> > in several seconds. so It is a potential risk when  a big
>> > rowmutation occurs.
>> >
>> >
>> >
>> > Luke
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>>
>
>