Re: cassandra for a inbox search with high reading qps

[Chen Xinli <ch...@gmail.com>]

Hi,

Despite our use cases, is't it a good feature to disable reading when a node
is doing hinted handoff, just like bootstraping?
It will be very useful for READ ONE consitency level.

Or it can be an option in storage-conf.xml, and user can set it when
necessary.

I'd like to implement this feature if it's useful.


2010/8/18 Chen Xinli <ch...@gmail.com>

> Thanks for your reply.
>
> Cassandra, in our case, is used  for searching purposes not for data
> storage.
> We can build the cassandra keyspace data daily/weekly when system load is
> lower.
>
> We have modified the cassandra code to add a value filter which makes the
> data-repair not working.
> The value filter, as I say, is to filter the columns of a key, and only the
> desired column is returned.
> The filter is done in local cassandra, not in thrift client; So we have to
> disable data-repair.
>
> Cassandra has met most of our needs except that:
> if a node fails, after a while, recovers, joins the cluster and doing
> hinted handoff, then a reading is forward to this node, the data returned is
> out of date.
>
> The node failure is not frequently; if it happens unfortunately, we should
> keep the reading consitency.
>
>
> 2010/8/18 Benjamin Black <b...@b3k.us>
>
> On Tue, Aug 17, 2010 at 7:55 PM, Chen Xinli <ch...@gmail.com> wrote:
>> > Hi,
>> >
>> > We are going to use cassandra for searching purpose like inbox search.
>> > The reading qps is very high, we'd like to use ConsitencyLevel.One for
>> > reading and disable read-repair at the same time.
>> >
>>
>> In 0.7 you can set a probability for read repair, but disabling it is
>> a spectacularly bad idea.  Any write problems on a node will result in
>> persistent inconsistency.
>>
>> > For reading consistency in this condition, the writing should use
>> > ConsistencyLevel.ALL. But the writing will fail if one node fails.
>>
>> You are free to read and write with consistency levels where R+W < N,
>> it just means you have weaker consistency guarantees.
>>
>> > We want such a ConsistencyLevel for writing/reading that :
>> > 1. writing will success if there is node alive for this key
>> > 2. reading will not forward to a node that's just recovered and doing
>> hinted
>> > handoff
>> >
>> > So that, if some node fails, others nodes for replica will receive the
>> data
>> > and surve reading successfully;
>> > when the failure node recovers,  it will receive hinted handoff from
>> other
>> > nodes and it'll not surve reading until hinted handoff is done.
>> >
>> > Does cassandra support the cases already? or should I modify the code to
>> > meet our requirements?
>> >
>>
>> You are phrasing these requirements in terms of a specific
>> implementation.  What are your actual consistency goals?  If node
>> failure is such a common occurrence in your system, you are going to
>> have _numerous_ problems.
>>
>>
>> b
>>
>
>
>
> --
> Best Regards,
> Chen Xinli
>



-- 
Best Regards,
Chen Xinli

Re: cassandra for a inbox search with high reading qps

[Rob Coli <rc...@digg.com>]

On 8/19/10 7:18 AM, Chen Xinli wrote:
> 2010/8/19 Oleg Anastasjev<ol...@gmail.com>
>
>> Chen Xinli<chen.daqi<at>  gmail.com>  writes:
>>
>>>
>>> Hi,
>>>
>>> Despite our use cases, is't it a good feature to disable reading when a
>> node
>>> is doing hinted handoff, just like bootstraping?
>>> It will be very useful for READ ONE consitency level.
>>>
>>> Or it can be an option in storage-conf.xml, and user can set it when
>>> necessary.
>>>
>>> I'd like to implement this feature if it's useful.
>>
>> Yes, we also use this failback policy, but currently doing it manually -
>> returning node back to clients nodelist only after hinted handoff is
>> completed.

FWIW, this has been proposed before, and resolved "Not A Problem."

https://issues.apache.org/jira/browse/CASSANDRA-768

=Rob

Re: cassandra for a inbox search with high reading qps

[Chen Xinli <ch...@gmail.com>]

Thanks for the update.
I got the idea; it also works in our case.
I read the last post by Rob; this feature has been posted before, and there
seems no such an easy way to figure it out.
Any way, the manual operation can solve our problem. Thanks a lot!


2010/8/20 Oleg Anastasjev <ol...@gmail.com>

> Chen Xinli <chen.daqi <at> gmail.com> writes:
>
> > would you pls describe the manual operation with more details?
> > I have not found any related information.
> >
> Um, this is code of our in-house implementation of cassandra client
> libraries.
> The main idea is that normally clients query ring and work directly with
> nodes
> found in ring until they detect failure or slow down of a particular node.
> Then clients fail over to the next node in ring automatically. Failed node
> is
> placed by clients to failed nodes list and will not ever be used by clients
> operator's command. This command is not to cassandra servers, but to
> clients
> saying to exclude node from failed nodes list.
>
> And there is procedure, that operator should inspect, did failed node
> finished
> hinted handoff prior issuing this command to clients.
>
> If we'd have possibility to inspect this condition automatically, we'd
> eliminated this manual inspection from our workflow.
>
>
>
>


-- 
Best Regards,
Chen Xinli

Re: cassandra for a inbox search with high reading qps

[Oleg Anastasjev <ol...@gmail.com>]

Chen Xinli <chen.daqi <at> gmail.com> writes:

> would you pls describe the manual operation with more details?
> I have not found any related information.
> 
Um, this is code of our in-house implementation of cassandra client libraries.
The main idea is that normally clients query ring and work directly with nodes
found in ring until they detect failure or slow down of a particular node. 
Then clients fail over to the next node in ring automatically. Failed node is
placed by clients to failed nodes list and will not ever be used by clients
operator's command. This command is not to cassandra servers, but to clients
saying to exclude node from failed nodes list.

And there is procedure, that operator should inspect, did failed node finished
hinted handoff prior issuing this command to clients.

If we'd have possibility to inspect this condition automatically, we'd
eliminated this manual inspection from our workflow.

Re: cassandra for a inbox search with high reading qps

[Chen Xinli <ch...@gmail.com>]

2010/8/19 Oleg Anastasjev <ol...@gmail.com>

> Chen Xinli <chen.daqi <at> gmail.com> writes:
>
> >
> > Hi,
> >
> > Despite our use cases, is't it a good feature to disable reading when a
> node
> > is doing hinted handoff, just like bootstraping?
> > It will be very useful for READ ONE consitency level.
> >
> > Or it can be an option in storage-conf.xml, and user can set it when
> > necessary.
> >
> > I'd like to implement this feature if it's useful.
>
> Yes, we also use this failback policy, but currently doing it manually -
> returning node back to clients nodelist only after hinted handoff is
> completed.
> So at least for us this should be handy, if you'd implement this to happen
> automatically.
>
>
would you pls describe the manual operation with more details?
I have not found any related information.

-- 
Best Regards,
Chen Xinli

Re: cassandra for a inbox search with high reading qps

[Oleg Anastasjev <ol...@gmail.com>]

Chen Xinli <chen.daqi <at> gmail.com> writes:

> 
> Hi,
> 
> Despite our use cases, is't it a good feature to disable reading when a node
> is doing hinted handoff, just like bootstraping?
> It will be very useful for READ ONE consitency level.
> 
> Or it can be an option in storage-conf.xml, and user can set it when
> necessary.
> 
> I'd like to implement this feature if it's useful.

Yes, we also use this failback policy, but currently doing it manually -
returning node back to clients nodelist only after hinted handoff is completed.
So at least for us this should be handy, if you'd implement this to happen
automatically.