You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Chen Xinli <ch...@gmail.com> on 2010/08/18 04:55:58 UTC
cassandra for a inbox search with high reading qps
Hi,
We are going to use cassandra for searching purpose like inbox search.
The reading qps is very high, we'd like to use ConsitencyLevel.One for
reading and disable read-repair at the same time.
For reading consistency in this condition, the writing should use
ConsistencyLevel.ALL. But the writing will fail if one node fails.
We want such a ConsistencyLevel for writing/reading that :
1. writing will success if there is node alive for this key
2. reading will not forward to a node that's just recovered and doing hinted
handoff
So that, if some node fails, others nodes for replica will receive the data
and surve reading successfully;
when the failure node recovers, it will receive hinted handoff from other
nodes and it'll not surve reading until hinted handoff is done.
Does cassandra support the cases already? or should I modify the code to
meet our requirements?
Thanks for any advices!
--
Best Regards,
Chen Xinli
Re: cassandra for a inbox search with high reading qps
Posted by Chen Xinli <ch...@gmail.com>.
I'm using cassandra 0.6.4; there's a configuration option
DoConsistencyChecksBoolean in storage-conf.xml.
Is't that for read-repair ?
I will do a test for WRITE QUORUM, READ.ONE if it can meet our requirements.
2010/8/18 Edward Capriolo <ed...@gmail.com>
> On Tue, Aug 17, 2010 at 10:55 PM, Chen Xinli <ch...@gmail.com> wrote:
> > Hi,
> >
> > We are going to use cassandra for searching purpose like inbox search.
> > The reading qps is very high, we'd like to use ConsitencyLevel.One for
> > reading and disable read-repair at the same time.
> >
> > For reading consistency in this condition, the writing should use
> > ConsistencyLevel.ALL. But the writing will fail if one node fails.
> > We want such a ConsistencyLevel for writing/reading that :
> > 1. writing will success if there is node alive for this key
> > 2. reading will not forward to a node that's just recovered and doing
> hinted
> > handoff
> >
> > So that, if some node fails, others nodes for replica will receive the
> data
> > and surve reading successfully;
> > when the failure node recovers, it will receive hinted handoff from
> other
> > nodes and it'll not surve reading until hinted handoff is done.
> >
> > Does cassandra support the cases already? or should I modify the code to
> > meet our requirements?
> >
> > Thanks for any advices!
> >
> > --
> > Best Regards,
> > Chen Xinli
> >
>
> >>The reading qps is very high, we'd like to use ConsitencyLevel.One for
> reading and disable read-repair at the same time.
> You can not disable read repair, all reads no matter the
> ConsistencyLevel always repair. The CL only controls how many nodes to
> read from before returning data to the client.
>
> These two statements contradict.
> > For reading consistency in this condition, the writing should use
> ConsistencyLevel.ALL. But the writing will fail if one node fails.
> > 1. writing will success if there is node alive for this key
>
> Also regardless of the write ConsistencyLevel all writes are written
> to all nodes. If a target node is down HintedHandoff will queue the
> write up for when the node restarts * in (6.3 you can turn off Hinted
> Handoff)
>
> You may want to WRITE QUORUM, READ.ONE, with RF=3 you would need two
> failed nodes before seeing an UnavailableException. You still have
> pretty strong consistency (depending how you look at it) and fast
> reads.
>
> Check the IRC logs Ben "schooled" me over this a couple of days ago.
>
--
Best Regards,
Chen Xinli
Re: cassandra for a inbox search with high reading qps
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Aug 17, 2010 at 10:55 PM, Chen Xinli <ch...@gmail.com> wrote:
> Hi,
>
> We are going to use cassandra for searching purpose like inbox search.
> The reading qps is very high, we'd like to use ConsitencyLevel.One for
> reading and disable read-repair at the same time.
>
> For reading consistency in this condition, the writing should use
> ConsistencyLevel.ALL. But the writing will fail if one node fails.
> We want such a ConsistencyLevel for writing/reading that :
> 1. writing will success if there is node alive for this key
> 2. reading will not forward to a node that's just recovered and doing hinted
> handoff
>
> So that, if some node fails, others nodes for replica will receive the data
> and surve reading successfully;
> when the failure node recovers, it will receive hinted handoff from other
> nodes and it'll not surve reading until hinted handoff is done.
>
> Does cassandra support the cases already? or should I modify the code to
> meet our requirements?
>
> Thanks for any advices!
>
> --
> Best Regards,
> Chen Xinli
>
>>The reading qps is very high, we'd like to use ConsitencyLevel.One for reading and disable read-repair at the same time.
You can not disable read repair, all reads no matter the
ConsistencyLevel always repair. The CL only controls how many nodes to
read from before returning data to the client.
These two statements contradict.
> For reading consistency in this condition, the writing should use ConsistencyLevel.ALL. But the writing will fail if one node fails.
> 1. writing will success if there is node alive for this key
Also regardless of the write ConsistencyLevel all writes are written
to all nodes. If a target node is down HintedHandoff will queue the
write up for when the node restarts * in (6.3 you can turn off Hinted
Handoff)
You may want to WRITE QUORUM, READ.ONE, with RF=3 you would need two
failed nodes before seeing an UnavailableException. You still have
pretty strong consistency (depending how you look at it) and fast
reads.
Check the IRC logs Ben "schooled" me over this a couple of days ago.
Re: cassandra for a inbox search with high reading qps
Posted by Chen Xinli <ch...@gmail.com>.
what's the average size of a user?
As I know, lucandra will first poll the data from cassandra, then do
computation in the client. That's ok for small rows.
But we have 1M row in average, and some rows scale to 100M; at the same
time, we expect high reading qps.
Polling these data to client machine through network is unacceptable.
I have setup a demo system which do the searching locally in cassandra; it
seems working with reading qps 1000+ per node.
2010/8/31 Todd Nine <to...@spidertracks.co.nz>
> We use Lucandra as well for searching for users, as well as geo-encoding.
> It really works well except for numeric fields.
>
> https://issues.apache.org/jira/browse/CASSANDRA-1235
>
> That bug may be a bit of an issue, but after they release 0.6.5 all the
> Lucene functionality will be available to you.
>
> Todd
>
>
>
>
>
>
>
> On Mon, 2010-08-30 at 05:49 -0700, Mike Peters wrote:
>
> Chen,
>
> Have you considered using http://www.slideshare.net/otisg/lucandra Lucandra
> for Inbox search?
>
> We have a similar setup and are currently looking into using Lucandra over
> implementing the searching ourselves with pure Cassandra.
>
>
--
Best Regards,
Chen Xinli
Re: cassandra for a inbox search with high reading qps
Posted by Todd Nine <to...@spidertracks.co.nz>.
We use Lucandra as well for searching for users, as well as
geo-encoding. It really works well except for numeric fields.
https://issues.apache.org/jira/browse/CASSANDRA-1235
That bug may be a bit of an issue, but after they release 0.6.5 all the
Lucene functionality will be available to you.
Todd
On Mon, 2010-08-30 at 05:49 -0700, Mike Peters wrote:
> Chen,
>
> Have you considered using http://www.slideshare.net/otisg/lucandra Lucandra
> for Inbox search?
>
> We have a similar setup and are currently looking into using Lucandra over
> implementing the searching ourselves with pure Cassandra.
Re: cassandra for a inbox search with high reading qps
Posted by Mike Peters <ca...@softwareprojects.com>.
Chen,
Have you considered using http://www.slideshare.net/otisg/lucandra Lucandra
for Inbox search?
We have a similar setup and are currently looking into using Lucandra over
implementing the searching ourselves with pure Cassandra.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/cassandra-for-a-inbox-search-with-high-reading-qps-tp5434900p5478060.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: cassandra for a inbox search with high reading qps
Posted by Rob Coli <rc...@digg.com>.
On 8/19/10 7:18 AM, Chen Xinli wrote:
> 2010/8/19 Oleg Anastasjev<ol...@gmail.com>
>
>> Chen Xinli<chen.daqi<at> gmail.com> writes:
>>
>>>
>>> Hi,
>>>
>>> Despite our use cases, is't it a good feature to disable reading when a
>> node
>>> is doing hinted handoff, just like bootstraping?
>>> It will be very useful for READ ONE consitency level.
>>>
>>> Or it can be an option in storage-conf.xml, and user can set it when
>>> necessary.
>>>
>>> I'd like to implement this feature if it's useful.
>>
>> Yes, we also use this failback policy, but currently doing it manually -
>> returning node back to clients nodelist only after hinted handoff is
>> completed.
FWIW, this has been proposed before, and resolved "Not A Problem."
https://issues.apache.org/jira/browse/CASSANDRA-768
=Rob
Re: cassandra for a inbox search with high reading qps
Posted by Chen Xinli <ch...@gmail.com>.
Thanks for the update.
I got the idea; it also works in our case.
I read the last post by Rob; this feature has been posted before, and there
seems no such an easy way to figure it out.
Any way, the manual operation can solve our problem. Thanks a lot!
2010/8/20 Oleg Anastasjev <ol...@gmail.com>
> Chen Xinli <chen.daqi <at> gmail.com> writes:
>
> > would you pls describe the manual operation with more details?
> > I have not found any related information.
> >
> Um, this is code of our in-house implementation of cassandra client
> libraries.
> The main idea is that normally clients query ring and work directly with
> nodes
> found in ring until they detect failure or slow down of a particular node.
> Then clients fail over to the next node in ring automatically. Failed node
> is
> placed by clients to failed nodes list and will not ever be used by clients
> operator's command. This command is not to cassandra servers, but to
> clients
> saying to exclude node from failed nodes list.
>
> And there is procedure, that operator should inspect, did failed node
> finished
> hinted handoff prior issuing this command to clients.
>
> If we'd have possibility to inspect this condition automatically, we'd
> eliminated this manual inspection from our workflow.
>
>
>
>
--
Best Regards,
Chen Xinli
Re: cassandra for a inbox search with high reading qps
Posted by Oleg Anastasjev <ol...@gmail.com>.
Chen Xinli <chen.daqi <at> gmail.com> writes:
> would you pls describe the manual operation with more details?
> I have not found any related information.
>
Um, this is code of our in-house implementation of cassandra client libraries.
The main idea is that normally clients query ring and work directly with nodes
found in ring until they detect failure or slow down of a particular node.
Then clients fail over to the next node in ring automatically. Failed node is
placed by clients to failed nodes list and will not ever be used by clients
operator's command. This command is not to cassandra servers, but to clients
saying to exclude node from failed nodes list.
And there is procedure, that operator should inspect, did failed node finished
hinted handoff prior issuing this command to clients.
If we'd have possibility to inspect this condition automatically, we'd
eliminated this manual inspection from our workflow.
Re: cassandra for a inbox search with high reading qps
Posted by Chen Xinli <ch...@gmail.com>.
2010/8/19 Oleg Anastasjev <ol...@gmail.com>
> Chen Xinli <chen.daqi <at> gmail.com> writes:
>
> >
> > Hi,
> >
> > Despite our use cases, is't it a good feature to disable reading when a
> node
> > is doing hinted handoff, just like bootstraping?
> > It will be very useful for READ ONE consitency level.
> >
> > Or it can be an option in storage-conf.xml, and user can set it when
> > necessary.
> >
> > I'd like to implement this feature if it's useful.
>
> Yes, we also use this failback policy, but currently doing it manually -
> returning node back to clients nodelist only after hinted handoff is
> completed.
> So at least for us this should be handy, if you'd implement this to happen
> automatically.
>
>
would you pls describe the manual operation with more details?
I have not found any related information.
--
Best Regards,
Chen Xinli
Re: cassandra for a inbox search with high reading qps
Posted by Oleg Anastasjev <ol...@gmail.com>.
Chen Xinli <chen.daqi <at> gmail.com> writes:
>
> Hi,
>
> Despite our use cases, is't it a good feature to disable reading when a node
> is doing hinted handoff, just like bootstraping?
> It will be very useful for READ ONE consitency level.
>
> Or it can be an option in storage-conf.xml, and user can set it when
> necessary.
>
> I'd like to implement this feature if it's useful.
Yes, we also use this failback policy, but currently doing it manually -
returning node back to clients nodelist only after hinted handoff is completed.
So at least for us this should be handy, if you'd implement this to happen
automatically.
Re: cassandra for a inbox search with high reading qps
Posted by Chen Xinli <ch...@gmail.com>.
Hi,
Despite our use cases, is't it a good feature to disable reading when a node
is doing hinted handoff, just like bootstraping?
It will be very useful for READ ONE consitency level.
Or it can be an option in storage-conf.xml, and user can set it when
necessary.
I'd like to implement this feature if it's useful.
2010/8/18 Chen Xinli <ch...@gmail.com>
> Thanks for your reply.
>
> Cassandra, in our case, is used for searching purposes not for data
> storage.
> We can build the cassandra keyspace data daily/weekly when system load is
> lower.
>
> We have modified the cassandra code to add a value filter which makes the
> data-repair not working.
> The value filter, as I say, is to filter the columns of a key, and only the
> desired column is returned.
> The filter is done in local cassandra, not in thrift client; So we have to
> disable data-repair.
>
> Cassandra has met most of our needs except that:
> if a node fails, after a while, recovers, joins the cluster and doing
> hinted handoff, then a reading is forward to this node, the data returned is
> out of date.
>
> The node failure is not frequently; if it happens unfortunately, we should
> keep the reading consitency.
>
>
> 2010/8/18 Benjamin Black <b...@b3k.us>
>
> On Tue, Aug 17, 2010 at 7:55 PM, Chen Xinli <ch...@gmail.com> wrote:
>> > Hi,
>> >
>> > We are going to use cassandra for searching purpose like inbox search.
>> > The reading qps is very high, we'd like to use ConsitencyLevel.One for
>> > reading and disable read-repair at the same time.
>> >
>>
>> In 0.7 you can set a probability for read repair, but disabling it is
>> a spectacularly bad idea. Any write problems on a node will result in
>> persistent inconsistency.
>>
>> > For reading consistency in this condition, the writing should use
>> > ConsistencyLevel.ALL. But the writing will fail if one node fails.
>>
>> You are free to read and write with consistency levels where R+W < N,
>> it just means you have weaker consistency guarantees.
>>
>> > We want such a ConsistencyLevel for writing/reading that :
>> > 1. writing will success if there is node alive for this key
>> > 2. reading will not forward to a node that's just recovered and doing
>> hinted
>> > handoff
>> >
>> > So that, if some node fails, others nodes for replica will receive the
>> data
>> > and surve reading successfully;
>> > when the failure node recovers, it will receive hinted handoff from
>> other
>> > nodes and it'll not surve reading until hinted handoff is done.
>> >
>> > Does cassandra support the cases already? or should I modify the code to
>> > meet our requirements?
>> >
>>
>> You are phrasing these requirements in terms of a specific
>> implementation. What are your actual consistency goals? If node
>> failure is such a common occurrence in your system, you are going to
>> have _numerous_ problems.
>>
>>
>> b
>>
>
>
>
> --
> Best Regards,
> Chen Xinli
>
--
Best Regards,
Chen Xinli
Re: cassandra for a inbox search with high reading qps
Posted by Chen Xinli <ch...@gmail.com>.
Thanks for your reply.
Cassandra, in our case, is used for searching purposes not for data
storage.
We can build the cassandra keyspace data daily/weekly when system load is
lower.
We have modified the cassandra code to add a value filter which makes the
data-repair not working.
The value filter, as I say, is to filter the columns of a key, and only the
desired column is returned.
The filter is done in local cassandra, not in thrift client; So we have to
disable data-repair.
Cassandra has met most of our needs except that:
if a node fails, after a while, recovers, joins the cluster and doing hinted
handoff, then a reading is forward to this node, the data returned is out of
date.
The node failure is not frequently; if it happens unfortunately, we should
keep the reading consitency.
2010/8/18 Benjamin Black <b...@b3k.us>
> On Tue, Aug 17, 2010 at 7:55 PM, Chen Xinli <ch...@gmail.com> wrote:
> > Hi,
> >
> > We are going to use cassandra for searching purpose like inbox search.
> > The reading qps is very high, we'd like to use ConsitencyLevel.One for
> > reading and disable read-repair at the same time.
> >
>
> In 0.7 you can set a probability for read repair, but disabling it is
> a spectacularly bad idea. Any write problems on a node will result in
> persistent inconsistency.
>
> > For reading consistency in this condition, the writing should use
> > ConsistencyLevel.ALL. But the writing will fail if one node fails.
>
> You are free to read and write with consistency levels where R+W < N,
> it just means you have weaker consistency guarantees.
>
> > We want such a ConsistencyLevel for writing/reading that :
> > 1. writing will success if there is node alive for this key
> > 2. reading will not forward to a node that's just recovered and doing
> hinted
> > handoff
> >
> > So that, if some node fails, others nodes for replica will receive the
> data
> > and surve reading successfully;
> > when the failure node recovers, it will receive hinted handoff from
> other
> > nodes and it'll not surve reading until hinted handoff is done.
> >
> > Does cassandra support the cases already? or should I modify the code to
> > meet our requirements?
> >
>
> You are phrasing these requirements in terms of a specific
> implementation. What are your actual consistency goals? If node
> failure is such a common occurrence in your system, you are going to
> have _numerous_ problems.
>
>
> b
>
--
Best Regards,
Chen Xinli
Re: cassandra for a inbox search with high reading qps
Posted by Benjamin Black <b...@b3k.us>.
On Tue, Aug 17, 2010 at 7:55 PM, Chen Xinli <ch...@gmail.com> wrote:
> Hi,
>
> We are going to use cassandra for searching purpose like inbox search.
> The reading qps is very high, we'd like to use ConsitencyLevel.One for
> reading and disable read-repair at the same time.
>
In 0.7 you can set a probability for read repair, but disabling it is
a spectacularly bad idea. Any write problems on a node will result in
persistent inconsistency.
> For reading consistency in this condition, the writing should use
> ConsistencyLevel.ALL. But the writing will fail if one node fails.
You are free to read and write with consistency levels where R+W < N,
it just means you have weaker consistency guarantees.
> We want such a ConsistencyLevel for writing/reading that :
> 1. writing will success if there is node alive for this key
> 2. reading will not forward to a node that's just recovered and doing hinted
> handoff
>
> So that, if some node fails, others nodes for replica will receive the data
> and surve reading successfully;
> when the failure node recovers, it will receive hinted handoff from other
> nodes and it'll not surve reading until hinted handoff is done.
>
> Does cassandra support the cases already? or should I modify the code to
> meet our requirements?
>
You are phrasing these requirements in terms of a specific
implementation. What are your actual consistency goals? If node
failure is such a common occurrence in your system, you are going to
have _numerous_ problems.
b