You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Johan Oskarsson <jo...@oskarsson.nu> on 2010/09/02 21:01:20 UTC
[DISCUSSION] High-volume counters in Cassandra
In the last few months Digg and Twitter have been using a counter patch that lets Cassandra act as a high-volume realtime counting system. Atomic counters enable new applications that were previously difficult to implement at scale, including realtime analytics and large-scale systems monitoring.
Discussion
There are currently two different suggestions for how to implement counters in Cassandra. The discussion has so far been limited to those following the jiras (CASSANDRA-1072 and CASSANDRA-1421) closely and we don’t seem to be nearing a decision. I want to open it up to the Cassandra community at large to get additional feedback.
Below are very basic and brief introductions to the alternatives. Please help us move forward by reading through the docs and jiras and reply to this thread with your thoughts. Would one or the other, both or neither be suitable for inclusion in Cassandra? Is there a third option? What can we do to reach a decision?
We believe that both options can coexist; their strengths and weaknesses make them suitable for different use cases.
CASSANDRA-1072 + CASSANDRA-1397
https://issues.apache.org/jira/browse/CASSANDRA-1072 (see design doc)
https://issues.apache.org/jira/browse/CASSANDRA-1397
How does it work?
A node is picked as the primary replica for each write. The context byte array for a column contains (primary replica ip, value). Any previous data with the same ip is reconciled with the new increment and put as the column value.
Concerns raised
* an increment in flight will be lost if the wrong node goes down
* if an increment operation times out it’s impossible to know if it has been executed or not
The most recent jira comment proposes a new API method for increments that reflects the different consistency level guarantees.
CASSANDRA-1421
https://issues.apache.org/jira/browse/CASSANDRA-1421
How does it work?
Each increment for a counter is stored as a (UUID, value) tuple. The read operations will read all these increment tuples for a counter, reconcile and return. On a regular interval the values are all read and reconciled into one value to reduce the amount of data required for each read operation.
Concerns raised
* poor read performance, especially for time-series data
* post aggregation reconciliation issues
Again, we feel that both options can co-exist, especially if the 1072 patch uses a new API method that reflects its different consistency level guarantees. Our proposal is to accept 1072 into trunk with the new API method, and when an implementation of 1421 is completed it can be accepted alongside.
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Ben Standefer <be...@gmail.com>.
At SimpleGeo, we're close to just merging 1072 internally. I've talked with
several members of the community who have already done this and are running
1072 in production or quasi-production. It seems like if this isn't merged,
people are going to merge it internally anyways. I think such a widely
desired feature should not be left as a patch for users to merge themselves.
I like the idea of including both approaches and choosing between them
given your requirements.
-Ben Standefer
On Thu, Sep 2, 2010 at 12:01 PM, Johan Oskarsson <jo...@oskarsson.nu> wrote:
> In the last few months Digg and Twitter have been using a counter patch
> that lets Cassandra act as a high-volume realtime counting system. Atomic
> counters enable new applications that were previously difficult to implement
> at scale, including realtime analytics and large-scale systems monitoring.
>
> Discussion
> There are currently two different suggestions for how to implement counters
> in Cassandra. The discussion has so far been limited to those following the
> jiras (CASSANDRA-1072 and CASSANDRA-1421) closely and we don’t seem to be
> nearing a decision. I want to open it up to the Cassandra community at large
> to get additional feedback.
>
> Below are very basic and brief introductions to the alternatives. Please
> help us move forward by reading through the docs and jiras and reply to this
> thread with your thoughts. Would one or the other, both or neither be
> suitable for inclusion in Cassandra? Is there a third option? What can we do
> to reach a decision?
>
> We believe that both options can coexist; their strengths and weaknesses
> make them suitable for different use cases.
>
>
> CASSANDRA-1072 + CASSANDRA-1397
> https://issues.apache.org/jira/browse/CASSANDRA-1072 (see design doc)
> https://issues.apache.org/jira/browse/CASSANDRA-1397
>
> How does it work?
> A node is picked as the primary replica for each write. The context byte
> array for a column contains (primary replica ip, value). Any previous data
> with the same ip is reconciled with the new increment and put as the column
> value.
>
> Concerns raised
> * an increment in flight will be lost if the wrong node goes down
> * if an increment operation times out it’s impossible to know if it has
> been executed or not
>
> The most recent jira comment proposes a new API method for increments that
> reflects the different consistency level guarantees.
>
>
> CASSANDRA-1421
> https://issues.apache.org/jira/browse/CASSANDRA-1421
>
> How does it work?
> Each increment for a counter is stored as a (UUID, value) tuple. The read
> operations will read all these increment tuples for a counter, reconcile and
> return. On a regular interval the values are all read and reconciled into
> one value to reduce the amount of data required for each read operation.
>
> Concerns raised
> * poor read performance, especially for time-series data
> * post aggregation reconciliation issues
>
>
> Again, we feel that both options can co-exist, especially if the 1072 patch
> uses a new API method that reflects its different consistency level
> guarantees. Our proposal is to accept 1072 into trunk with the new API
> method, and when an implementation of 1421 is completed it can be accepted
> alongside.
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Zhu Han <sc...@gmail.com>.
I thought about it again for a while. It might be a good trade-off to just
implement the "CASSANDRA-1421" as a new API and limit the new code only in
StorageProxy level, and never put any dependency on internal memchanism of
Cassandra, e.g. compaction, membership management and other complicated
stuff.
If it does not pollute the core code base, it may be easier to refine it or
remove them when there is a better idea.
If so, the number of writers of a single counter is at most equals to the
number of cassandra node.
If the client takes a simple optimism to raise the thrift request only to
the cassandra node which are the storage node of the counter, the
performance should be almost the same as " CASSANDRA-1072 + CASSANDRA-1397",
as the number of writers is the same as the number of replications. You can
also save an extra round trip time. That's what I did in my project.
best regards,
hanzhu
On Sun, Sep 5, 2010 at 5:24 PM, Zhu Han <sc...@gmail.com> wrote:
> + 1 for Jonathan Ellis.
>
> I might not be on the same page as you active community members. But I'm
> wondering why not put this feature to a popular client library or as a
> contrib package?
>
> In CASSANDRA-1072 + CASSANDRA-1397, the increment of counter is not
> idempotent, so it's difficult to align with the consistency model of
> Cassandra. It's not worth to put a lot of code to the core base to just
> serve a single feature.
>
> In CASSANDRA-1421, the increment is idempotent and is easier to align with
> Cassandra. However, the read performance could be poor because it has to
> reconcile a lot of columns. The memory consumption on cassandra node might
> be much higher than the above approach, if I understood it correctly.
>
> If you decides to put the feature to the client library. The client library
> can take the approach as CASSANDRA-142, and serialize the increment from a
> single writer to limit the columns generated. If the writers of a single
> counter are just hundreds processes, I don't think it is a big deal for
> performance.
>
> If you worry about the performance on the client side because it serialize
> the increment of a single counter, maintain a queue for each counter and
> it's easy to batch multiple updates in the same queue.
>
>
>
> best regards,
> hanzhu
>
>
>
> On Fri, Sep 3, 2010 at 4:55 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> I still have not seen any response to my other misgivings about 1072
>> that I have raised on the ticket. Specifically, the existing patch is
>> based around a Clock structure that, since 580 is a dead end, is no
>> longer necessary.
>>
>> I'm also uneasy about adding 200k of code that meshes as poorly with
>> the rest of Cassandra as this does. The more it can be split off into
>> separate code paths, the better. Adding its own thrift method is a
>> good start, but it should go deeper than that.
>>
>> On Thu, Sep 2, 2010 at 12:01 PM, Johan Oskarsson <jo...@oskarsson.nu>
>> wrote:
>> > In the last few months Digg and Twitter have been using a counter patch
>> that lets Cassandra act as a high-volume realtime counting system. Atomic
>> counters enable new applications that were previously difficult to implement
>> at scale, including realtime analytics and large-scale systems monitoring.
>> >
>> > Discussion
>> > There are currently two different suggestions for how to implement
>> counters in Cassandra. The discussion has so far been limited to those
>> following the jiras (CASSANDRA-1072 and CASSANDRA-1421) closely and we don’t
>> seem to be nearing a decision. I want to open it up to the Cassandra
>> community at large to get additional feedback.
>> >
>> > Below are very basic and brief introductions to the alternatives. Please
>> help us move forward by reading through the docs and jiras and reply to this
>> thread with your thoughts. Would one or the other, both or neither be
>> suitable for inclusion in Cassandra? Is there a third option? What can we do
>> to reach a decision?
>> >
>> > We believe that both options can coexist; their strengths and weaknesses
>> make them suitable for different use cases.
>> >
>> >
>> > CASSANDRA-1072 + CASSANDRA-1397
>> > https://issues.apache.org/jira/browse/CASSANDRA-1072 (see design doc)
>> > https://issues.apache.org/jira/browse/CASSANDRA-1397
>> >
>> > How does it work?
>> > A node is picked as the primary replica for each write. The context byte
>> array for a column contains (primary replica ip, value). Any previous data
>> with the same ip is reconciled with the new increment and put as the column
>> value.
>> >
>> > Concerns raised
>> > * an increment in flight will be lost if the wrong node goes down
>> > * if an increment operation times out it’s impossible to know if it has
>> been executed or not
>> >
>> > The most recent jira comment proposes a new API method for increments
>> that reflects the different consistency level guarantees.
>> >
>> >
>> > CASSANDRA-1421
>> > https://issues.apache.org/jira/browse/CASSANDRA-1421
>> >
>> > How does it work?
>> > Each increment for a counter is stored as a (UUID, value) tuple. The
>> read operations will read all these increment tuples for a counter,
>> reconcile and return. On a regular interval the values are all read and
>> reconciled into one value to reduce the amount of data required for each
>> read operation.
>> >
>> > Concerns raised
>> > * poor read performance, especially for time-series data
>> > * post aggregation reconciliation issues
>> >
>> >
>> > Again, we feel that both options can co-exist, especially if the 1072
>> patch uses a new API method that reflects its different consistency level
>> guarantees. Our proposal is to accept 1072 into trunk with the new API
>> method, and when an implementation of 1421 is completed it can be accepted
>> alongside.
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>>
>
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Zhu Han <sc...@gmail.com>.
+ 1 for Jonathan Ellis.
I might not be on the same page as you active community members. But I'm
wondering why not put this feature to a popular client library or as a
contrib package?
In CASSANDRA-1072 + CASSANDRA-1397, the increment of counter is not
idempotent, so it's difficult to align with the consistency model of
Cassandra. It's not worth to put a lot of code to the core base to just
serve a single feature.
In CASSANDRA-1421, the increment is idempotent and is easier to align with
Cassandra. However, the read performance could be poor because it has to
reconcile a lot of columns. The memory consumption on cassandra node might
be much higher than the above approach, if I understood it correctly.
If you decides to put the feature to the client library. The client library
can take the approach as CASSANDRA-142, and serialize the increment from a
single writer to limit the columns generated. If the writers of a single
counter are just hundreds processes, I don't think it is a big deal for
performance.
If you worry about the performance on the client side because it serialize
the increment of a single counter, maintain a queue for each counter and
it's easy to batch multiple updates in the same queue.
best regards,
hanzhu
On Fri, Sep 3, 2010 at 4:55 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> I still have not seen any response to my other misgivings about 1072
> that I have raised on the ticket. Specifically, the existing patch is
> based around a Clock structure that, since 580 is a dead end, is no
> longer necessary.
>
> I'm also uneasy about adding 200k of code that meshes as poorly with
> the rest of Cassandra as this does. The more it can be split off into
> separate code paths, the better. Adding its own thrift method is a
> good start, but it should go deeper than that.
>
> On Thu, Sep 2, 2010 at 12:01 PM, Johan Oskarsson <jo...@oskarsson.nu>
> wrote:
> > In the last few months Digg and Twitter have been using a counter patch
> that lets Cassandra act as a high-volume realtime counting system. Atomic
> counters enable new applications that were previously difficult to implement
> at scale, including realtime analytics and large-scale systems monitoring.
> >
> > Discussion
> > There are currently two different suggestions for how to implement
> counters in Cassandra. The discussion has so far been limited to those
> following the jiras (CASSANDRA-1072 and CASSANDRA-1421) closely and we don’t
> seem to be nearing a decision. I want to open it up to the Cassandra
> community at large to get additional feedback.
> >
> > Below are very basic and brief introductions to the alternatives. Please
> help us move forward by reading through the docs and jiras and reply to this
> thread with your thoughts. Would one or the other, both or neither be
> suitable for inclusion in Cassandra? Is there a third option? What can we do
> to reach a decision?
> >
> > We believe that both options can coexist; their strengths and weaknesses
> make them suitable for different use cases.
> >
> >
> > CASSANDRA-1072 + CASSANDRA-1397
> > https://issues.apache.org/jira/browse/CASSANDRA-1072 (see design doc)
> > https://issues.apache.org/jira/browse/CASSANDRA-1397
> >
> > How does it work?
> > A node is picked as the primary replica for each write. The context byte
> array for a column contains (primary replica ip, value). Any previous data
> with the same ip is reconciled with the new increment and put as the column
> value.
> >
> > Concerns raised
> > * an increment in flight will be lost if the wrong node goes down
> > * if an increment operation times out it’s impossible to know if it has
> been executed or not
> >
> > The most recent jira comment proposes a new API method for increments
> that reflects the different consistency level guarantees.
> >
> >
> > CASSANDRA-1421
> > https://issues.apache.org/jira/browse/CASSANDRA-1421
> >
> > How does it work?
> > Each increment for a counter is stored as a (UUID, value) tuple. The read
> operations will read all these increment tuples for a counter, reconcile and
> return. On a regular interval the values are all read and reconciled into
> one value to reduce the amount of data required for each read operation.
> >
> > Concerns raised
> > * poor read performance, especially for time-series data
> > * post aggregation reconciliation issues
> >
> >
> > Again, we feel that both options can co-exist, especially if the 1072
> patch uses a new API method that reflects its different consistency level
> guarantees. Our proposal is to accept 1072 into trunk with the new API
> method, and when an implementation of 1421 is completed it can be accepted
> alongside.
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jeremy Hanna <je...@gmail.com>.
So ditch Clocks and refactor to be more cleanly separated and it could go in?
On Sep 2, 2010, at 3:55 PM, Jonathan Ellis wrote:
> I still have not seen any response to my other misgivings about 1072
> that I have raised on the ticket. Specifically, the existing patch is
> based around a Clock structure that, since 580 is a dead end, is no
> longer necessary.
>
> I'm also uneasy about adding 200k of code that meshes as poorly with
> the rest of Cassandra as this does. The more it can be split off into
> separate code paths, the better. Adding its own thrift method is a
> good start, but it should go deeper than that.
>
> On Thu, Sep 2, 2010 at 12:01 PM, Johan Oskarsson <jo...@oskarsson.nu> wrote:
>> In the last few months Digg and Twitter have been using a counter patch that lets Cassandra act as a high-volume realtime counting system. Atomic counters enable new applications that were previously difficult to implement at scale, including realtime analytics and large-scale systems monitoring.
>>
>> Discussion
>> There are currently two different suggestions for how to implement counters in Cassandra. The discussion has so far been limited to those following the jiras (CASSANDRA-1072 and CASSANDRA-1421) closely and we don’t seem to be nearing a decision. I want to open it up to the Cassandra community at large to get additional feedback.
>>
>> Below are very basic and brief introductions to the alternatives. Please help us move forward by reading through the docs and jiras and reply to this thread with your thoughts. Would one or the other, both or neither be suitable for inclusion in Cassandra? Is there a third option? What can we do to reach a decision?
>>
>> We believe that both options can coexist; their strengths and weaknesses make them suitable for different use cases.
>>
>>
>> CASSANDRA-1072 + CASSANDRA-1397
>> https://issues.apache.org/jira/browse/CASSANDRA-1072 (see design doc)
>> https://issues.apache.org/jira/browse/CASSANDRA-1397
>>
>> How does it work?
>> A node is picked as the primary replica for each write. The context byte array for a column contains (primary replica ip, value). Any previous data with the same ip is reconciled with the new increment and put as the column value.
>>
>> Concerns raised
>> * an increment in flight will be lost if the wrong node goes down
>> * if an increment operation times out it’s impossible to know if it has been executed or not
>>
>> The most recent jira comment proposes a new API method for increments that reflects the different consistency level guarantees.
>>
>>
>> CASSANDRA-1421
>> https://issues.apache.org/jira/browse/CASSANDRA-1421
>>
>> How does it work?
>> Each increment for a counter is stored as a (UUID, value) tuple. The read operations will read all these increment tuples for a counter, reconcile and return. On a regular interval the values are all read and reconciled into one value to reduce the amount of data required for each read operation.
>>
>> Concerns raised
>> * poor read performance, especially for time-series data
>> * post aggregation reconciliation issues
>>
>>
>> Again, we feel that both options can co-exist, especially if the 1072 patch uses a new API method that reflects its different consistency level guarantees. Our proposal is to accept 1072 into trunk with the new API method, and when an implementation of 1421 is completed it can be accepted alongside.
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Sylvain Lebresne <sy...@yakaz.com>.
On Fri, Oct 1, 2010 at 5:12 PM, Zhu Han <sc...@gmail.com> wrote:
>> They have however at least one advantage:
>> - your super columns are indexed, you don't have to deserialize them
>> entirely each time.
>>
>
> The size of counter super column is limited to how many replicas propagated
> values as the lead replica. It's size is upper bounded by the number of
> replicas. Even if we support hinted hand off of counters, the size of super
> columns just equals to the size of nodes in the cluster, as the worse case.
I wasn't clear, sorry. What I was saying is that if you encode superColumns
using a standard CF (by projecting), then, when you want to request the
subcolumns of a given super column in the encoding, this amount to requesting
a slice of standard columns, so you only deserialize the requested columns.
While if you use true super columns, not encoded ones, you're subject to the
limitation of super columns not being indexed.
I was not talking about the superColumns holding the counters (my previous
mail is not specific at all to counters).
That being said, everything you said is true. For counters, the super columns
limitations are not a problem at all.
>
>
> IMHO, it's not a big deal to de-serialize them entirely, the can fit into
> memory very easily. Did I miss anything here?
>
> One advantage of implementing counter as a new CF type:
> --you can add some counter specific configuration very easily.
>
> best regards,
> hanzhu
>
> To sum up, I can see the following drawbacks to such encoding:
>> - querying SC by names is less efficient.
>> - it takes more disk space (but that's the cheapest resource we have
>> isn't it).
>>
>> I'd say these are fair compromises.
>>
>> --
>> Sylvain
>>
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Zhu Han <sc...@gmail.com>.
> They have however at least one advantage:
> - your super columns are indexed, you don't have to deserialize them
> entirely each time.
>
The size of counter super column is limited to how many replicas propagated
values as the lead replica. It's size is upper bounded by the number of
replicas. Even if we support hinted hand off of counters, the size of super
columns just equals to the size of nodes in the cluster, as the worse case.
IMHO, it's not a big deal to de-serialize them entirely, the can fit into
memory very easily. Did I miss anything here?
One advantage of implementing counter as a new CF type:
--you can add some counter specific configuration very easily.
best regards,
hanzhu
To sum up, I can see the following drawbacks to such encoding:
> - querying SC by names is less efficient.
> - it takes more disk space (but that's the cheapest resource we have
> isn't it).
>
> I'd say these are fair compromises.
>
> --
> Sylvain
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Sylvain Lebresne <sy...@yakaz.com>.
On Thu, Sep 30, 2010 at 6:29 PM, Ryan King <ry...@twitter.com> wrote:
> On Tue, Sep 28, 2010 at 10:14 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> On Tue, Sep 28, 2010 at 4:00 PM, Sylvain Lebresne <sy...@yakaz.com> wrote:
>>> I agree that it is worth adding a support for counter as supercolumns
>>> in 1546 and that's fairly trivial, so I will add that as soon as possible
>>> (but please understand that I'm working on this for a good part during
>>> my free time).
>>>
>>> As for supercolumns of counters, there is what Jonathan proposes, but
>>> I'll add that encoding a supercolumns CF to a standard column CF is
>>> almost always a fairly trivial encoding. Worst case scenario it requires
>>> you to roll up your own comparator and it's slightly less convenient
>>> for client code.
>>
>> Supporting supercolumns to allow multiple counters per row, but
>> requiring encoding with a custom comparator for deeper nesting, seems
>> like a reasonable compromise to me.
>
> I don't understand how this would work.
This is a general thing, not related to counter. Let's get a little bit
precise.
The idea is that you will encode the following super row:
key: {
scol1 : {
col1 : v1,
col2 : v2,
col3 : v3
},
scol2 : {
col4 : v4,
col5 : v5
}
}
as the standard row:
key : {
scol1|col1 : v1,
scol1|col2 : v2,
scol1|col3 : v3,
scol2|col4 : v4,
scol2|col5 : v5
}
To get slightly more technical, scol1|col1 could be:
[length of scol1][scol1 bytes][0][col1 bytes]
And by that I mean that the bytes of the column in the encoding will start by
4 bytes for the length of scol1 (a byte[]), then scol1, then a 0 byte, then
col1.
The reason for the 0 byte after the super column name is for slice queries, to
express the end of the super column (in the encoding). More precisely, for
slice queries, the start of the scol1 is
[length of scol1][scol1 bytes][0]
and the end of scol1 is
[length of scol1][scol1 bytes][1]
The custom comparator is fairly easy to write. It takes the super column
comparator (comp1) and the column comparator (comp2). To compare two (encoded)
keys, it first read the super column name of the two keys (using the size at
the start to each key) and compare them with comp1. If there are not equal,
return the comparison value. Otherwise, read the next byte of each key. If
unequal, biggest key is the one with the 1. If equal, read the two columns
name and compare using comp2.
Translating slice predicates is fairly trivial. You'll just have to iterate
over the result to regroup the columns into super columns, but no biggy.
The only thing that is less efficient is querying super columns by names
(querying sub columns by name is fine however). For that, you'll have to
issue one slice query for each requested name.
Some remove operation could probably also be slightly less efficient, but in
the end removes is broken with counters (both in 1072 and 1546, I'll refer you
to the comments of this last ticket), so it's not a big deal.
To sum up, I can see the following drawbacks to such encoding:
- querying SC by names is less efficient.
- it takes more disk space (but that's the cheapest resource we have
isn't it).
They have however at least one advantage:
- your super columns are indexed, you don't have to deserialize them
entirely each time.
I'd say these are fair compromises.
--
Sylvain
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Ryan King <ry...@twitter.com>.
On Tue, Sep 28, 2010 at 10:14 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Tue, Sep 28, 2010 at 4:00 PM, Sylvain Lebresne <sy...@yakaz.com> wrote:
>> I agree that it is worth adding a support for counter as supercolumns
>> in 1546 and that's fairly trivial, so I will add that as soon as possible
>> (but please understand that I'm working on this for a good part during
>> my free time).
>>
>> As for supercolumns of counters, there is what Jonathan proposes, but
>> I'll add that encoding a supercolumns CF to a standard column CF is
>> almost always a fairly trivial encoding. Worst case scenario it requires
>> you to roll up your own comparator and it's slightly less convenient
>> for client code.
>
> Supporting supercolumns to allow multiple counters per row, but
> requiring encoding with a custom comparator for deeper nesting, seems
> like a reasonable compromise to me.
I don't understand how this would work.
-ryan
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
On Tue, Sep 28, 2010 at 4:00 PM, Sylvain Lebresne <sy...@yakaz.com> wrote:
> I agree that it is worth adding a support for counter as supercolumns
> in 1546 and that's fairly trivial, so I will add that as soon as possible
> (but please understand that I'm working on this for a good part during
> my free time).
>
> As for supercolumns of counters, there is what Jonathan proposes, but
> I'll add that encoding a supercolumns CF to a standard column CF is
> almost always a fairly trivial encoding. Worst case scenario it requires
> you to roll up your own comparator and it's slightly less convenient
> for client code.
Supporting supercolumns to allow multiple counters per row, but
requiring encoding with a custom comparator for deeper nesting, seems
like a reasonable compromise to me.
Ryan, while this obviously isn't ideal from your perspective of
already having code written against supercolumns-of-counters, is it
something you could live with?
In exchange we get 1072 + 1210 + 1397 + 1375 + ConsistencyLevel
support for substantially less code (= less complexity and special
cases) than 1072 alone.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Sylvain Lebresne <sy...@yakaz.com>.
I agree that it is worth adding a support for counter as supercolumns
in 1546 and that's fairly trivial, so I will add that as soon as possible
(but please understand that I'm working on this for a good part during
my free time).
As for supercolumns of counters, there is what Jonathan proposes, but
I'll add that encoding a supercolumns CF to a standard column CF is
almost always a fairly trivial encoding. Worst case scenario it requires
you to roll up your own comparator and it's slightly less convenient
for client code.
On Tue, Sep 28, 2010 at 10:38 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Tue, Sep 28, 2010 at 3:35 PM, Ryan King <ry...@twitter.com> wrote:
>> On Tue, Sep 28, 2010 at 12:48 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>> On Tue, Sep 28, 2010 at 2:25 PM, Ryan King <ry...@twitter.com> wrote:
>>>> Sorry, been catching up on this.
>>>>
>>>> From Twitter's perspective, 1546 is probably insufficient because it
>>>> doesn't allow one to do time-series data without supercolumns (which
>>>> might work ok, but require a good deal of work). Additionally, one of
>>>> our deployed systems already does supercolumns of counters, which is
>>>> not feasible in this design at all.
>>>
>>> True, although wouldn't rows-of-counters give you basically the same
>>> thing if 1546 kept its counters as subcolumns of a supercolumn?
>>
>> That would solve the first part (single-dimension time-series), but we
>> also do supercolumns of counters, which 1546 doesn't appear to be able
>> to address.
>
> Right, I'm saying that "rows of counters + get_range_slices" looks a
> lot like "supercolumns of counters + get_slice" to me.
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
On Tue, Sep 28, 2010 at 3:35 PM, Ryan King <ry...@twitter.com> wrote:
> On Tue, Sep 28, 2010 at 12:48 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> On Tue, Sep 28, 2010 at 2:25 PM, Ryan King <ry...@twitter.com> wrote:
>>> Sorry, been catching up on this.
>>>
>>> From Twitter's perspective, 1546 is probably insufficient because it
>>> doesn't allow one to do time-series data without supercolumns (which
>>> might work ok, but require a good deal of work). Additionally, one of
>>> our deployed systems already does supercolumns of counters, which is
>>> not feasible in this design at all.
>>
>> True, although wouldn't rows-of-counters give you basically the same
>> thing if 1546 kept its counters as subcolumns of a supercolumn?
>
> That would solve the first part (single-dimension time-series), but we
> also do supercolumns of counters, which 1546 doesn't appear to be able
> to address.
Right, I'm saying that "rows of counters + get_range_slices" looks a
lot like "supercolumns of counters + get_slice" to me.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Ryan King <ry...@twitter.com>.
On Tue, Sep 28, 2010 at 12:48 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Tue, Sep 28, 2010 at 2:25 PM, Ryan King <ry...@twitter.com> wrote:
>> Sorry, been catching up on this.
>>
>> From Twitter's perspective, 1546 is probably insufficient because it
>> doesn't allow one to do time-series data without supercolumns (which
>> might work ok, but require a good deal of work). Additionally, one of
>> our deployed systems already does supercolumns of counters, which is
>> not feasible in this design at all.
>
> True, although wouldn't rows-of-counters give you basically the same
> thing if 1546 kept its counters as subcolumns of a supercolumn?
That would solve the first part (single-dimension time-series), but we
also do supercolumns of counters, which 1546 doesn't appear to be able
to address.
-ryan
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
On Tue, Sep 28, 2010 at 2:25 PM, Ryan King <ry...@twitter.com> wrote:
> Sorry, been catching up on this.
>
> From Twitter's perspective, 1546 is probably insufficient because it
> doesn't allow one to do time-series data without supercolumns (which
> might work ok, but require a good deal of work). Additionally, one of
> our deployed systems already does supercolumns of counters, which is
> not feasible in this design at all.
True, although wouldn't rows-of-counters give you basically the same
thing if 1546 kept its counters as subcolumns of a supercolumn?
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Ryan King <ry...@twitter.com>.
Sorry, been catching up on this.
>From Twitter's perspective, 1546 is probably insufficient because it
doesn't allow one to do time-series data without supercolumns (which
might work ok, but require a good deal of work). Additionally, one of
our deployed systems already does supercolumns of counters, which is
not feasible in this design at all.
-ryan
On Tue, Sep 28, 2010 at 10:12 AM, Jeremy Hanna
<je...@gmail.com> wrote:
> Is there any feedback from Twitter and Digg and perhaps SimpleGeo people about CASSANDRA-1546? Would that work so that you wouldn't have to maintain a fork?
>
> On Sep 27, 2010, at 5:25 AM, Sylvain Lebresne wrote:
>
>> In CASSANDRA-1546, I propose an alternative to #1072. At it's core,
>> it rewrites #1072 without the clocks structure (by splitting the clock into
>> individual columns, not unlike what Zhu Han proposed in his preceding
>> mail, but in a row instead of a super column, for reason explained in the
>> issue).
>>
>> But it is also my belief that it improves on the actual patch of #1072 in
>> the following ways:
>> - it supports increments and decrements
>> - it supports the usual consistency levels
>> - it proposes an (optional) solution to the idempotency problem of
>> increments (it's optional because it has a (fairly slight) performance cost
>> that some may want to remove if they understand the risk).
>>
>> When I say, I propose, I mean that I did wrote the patch (attached to the jira
>> ticket). I've just written it, so it is really under-tested and have a
>> few details here
>> and there to fix, but it should already be fairly functional (it
>> passes basic system
>> tests).
>>
>> I welcome all comments on the patch. It has been written with in mind
>> the goal to
>> address most of the concerns that have been addressed on those counters since a
>> few months (both in terms of performance and implementation). It is my
>> belief that
>> is reaches this goal, hopefully other will agree.
>>
>> --
>> Sylvain
>>
>> On Mon, Sep 27, 2010 at 5:32 AM, Zhu Han <sc...@gmail.com> wrote:
>>> I propose a new way to solve the counter problem in cassandra-1502[1].
>>> Since I do not follow the jira update very carefully, I paste it here and
>>> want to let more people comment it and then to see whether its feasible.
>>>
>>> "Seems like we have not found a solution acceptable to everybody. I tries to
>>> propose a new approach. Let's see whether anybody can shed some light on it
>>> and make it as reality.
>>>
>>> 1) We add a basic data structure, called as counter, which is a special type
>>> of super column.
>>>
>>> 2) The name of each column in the counter super column, is the host name of
>>> a cassandra node. And the value is the calculated result from that node.
>>>
>>> 3) WRITE PATH: Once a node receives the add/dec request of a counter, it
>>> de-serializes its local counter super column, and update the column named by
>>> itself atomically. After that, it propagates the updated column value to
>>> other replicas, just like how the mutation of a normal column is propagated
>>> to other replicas. Different consistency levels can be supported as before.
>>>
>>> 4) READ PATH: Depends on the consistency level, contact several replicas,
>>> read back the counter super column as whole, and get the latest counter
>>> value by summing up all columns in the counter. Read-repair logic can work
>>> as before.
>>>
>>> IMHO, the biggest advantages of this approach, is re-using as many
>>> mechanisms already in the code as possible. So it might not so disruptive.
>>> But adding new thrift API is inevitable. "
>>> NB: If it's feasible, I might not be the right man working on it as I have
>>> not touched the internal of cassandra for more than 1 year. I wants to
>>> contribute something to help us get consensus.
>>>
>>> [1]
>>> https://issues.apache.org/jira/browse/CASSANDRA-1502?focusedCommentId=12915103&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12915103
>>>
>>> best regards,
>>> hanzhu
>>>
>>>
>>> On Sun, Sep 26, 2010 at 9:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>
>>>> you have misunderstood. if we continue the 1072 approach of writing
>>>> counter data to the clock field, this is necessarily incompatible with
>>>> the right way of writing counter data to the value field. it's no
>>>> longer simply a matter of reversing 1070.
>>>>
>>>> On Sat, Sep 25, 2010 at 11:50 PM, Zhu Han <sc...@gmail.com> wrote:
>>>>> Jonathan,
>>>>>
>>>>> This is a personnel email.
>>>>>
>>>>> On Sun, Sep 26, 2010 at 1:27 PM, Jonathan Ellis <jb...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> On Sat, Sep 25, 2010 at 8:57 PM, Zhu Han <sc...@gmail.com> wrote:
>>>>>>> Can we just let the patch committed but mark it as "alpah" or
>>>>>>> "experimental"?
>>>>>>
>>>>>> I explained exactly why that is not a good approach here:
>>>>>> http://www.mail-archive.com/dev@cassandra.apache.org/msg00917.html
>>>>>>
>>>>> Yes, I see. But the clock structure is in truck since Cassandra-1070. We
>>>>> still need to clean them
>>>>> out, whatever. We need somebody to be volunteer to take this work.
>>>>> Considering the complexity
>>>>> of Cassandra-1070, the programmer who has the in depth knowledge of this
>>>>> patch is preferable. And it
>>>>> will take some time to do it.
>>>>>
>>>>> Fortunately, Johan Oskarsson has promised to take it in the comment of
>>>>> Cassandra-1072[1]:
>>>>>
>>>>> "The clock changes would get into trunk quicker if we didn't, avoiding
>>>> the
>>>>> extra overhead of a big patch during reviews, merge with trunk, code
>>>> updates
>>>>> and publication of a new patch.
>>>>> If the concern is that we won't attend to the clocks once this patch is
>>>> in I
>>>>> can promise that we'll look at it straight away. "
>>>>>
>>>>> And if twitter/digg/simplegeo forks their tree of cassandra, this will
>>>> give
>>>>> a big marketing opportunities of other NOSQL system supporters. As you
>>>> know,
>>>>> the competition is quite fierce currently.
>>>>>
>>>>> So, instead of sticking to the embarrassed situation, why not change to
>>>>> another strategy:
>>>>>
>>>>>> "Fork another experimental tree from 0.7 beta 1 and accept
>>>>>> Cassandra-1072. At the same time, start the clean up work on this tree.
>>>>>> Once it's finalized , merge them back to 0.7, no matter it's 0.7.1 or
>>>> 0.7.2.
>>>>>>
>>>>>> Hence, these guys from twitter does not need to maintain a huge
>>>>>> out-of-tree patch, while the quality impact of cassandra-1072 is still
>>>>>> limited.
>>>>>
>>>>> I do know the pain of maintaining a large patch out of the official tree.
>>>>> Once it gets in, everybody will feels much better.
>>>>>
>>>>> If you give some opportunities to this patch, Johan or others can be
>>>> highly
>>>>> motivated because all of the community works together. It's a
>>>> compromise,
>>>>> but it's worth.
>>>>>
>>>>> [1]
>>>>>
>>>> https://issues.apache.org/jira/browse/CASSANDRA-1072?focusedCommentId=12909234&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909234
>>>>>
>>>>>
>>>>>>
>>>>>> --
>>>>>> Jonathan Ellis
>>>>>> Project Chair, Apache Cassandra
>>>>>> co-founder of Riptano, the source for professional Cassandra support
>>>>>> http://riptano.com
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Jonathan Ellis
>>>> Project Chair, Apache Cassandra
>>>> co-founder of Riptano, the source for professional Cassandra support
>>>> http://riptano.com
>>>>
>>>
>
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jeremy Hanna <je...@gmail.com>.
Is there any feedback from Twitter and Digg and perhaps SimpleGeo people about CASSANDRA-1546? Would that work so that you wouldn't have to maintain a fork?
On Sep 27, 2010, at 5:25 AM, Sylvain Lebresne wrote:
> In CASSANDRA-1546, I propose an alternative to #1072. At it's core,
> it rewrites #1072 without the clocks structure (by splitting the clock into
> individual columns, not unlike what Zhu Han proposed in his preceding
> mail, but in a row instead of a super column, for reason explained in the
> issue).
>
> But it is also my belief that it improves on the actual patch of #1072 in
> the following ways:
> - it supports increments and decrements
> - it supports the usual consistency levels
> - it proposes an (optional) solution to the idempotency problem of
> increments (it's optional because it has a (fairly slight) performance cost
> that some may want to remove if they understand the risk).
>
> When I say, I propose, I mean that I did wrote the patch (attached to the jira
> ticket). I've just written it, so it is really under-tested and have a
> few details here
> and there to fix, but it should already be fairly functional (it
> passes basic system
> tests).
>
> I welcome all comments on the patch. It has been written with in mind
> the goal to
> address most of the concerns that have been addressed on those counters since a
> few months (both in terms of performance and implementation). It is my
> belief that
> is reaches this goal, hopefully other will agree.
>
> --
> Sylvain
>
> On Mon, Sep 27, 2010 at 5:32 AM, Zhu Han <sc...@gmail.com> wrote:
>> I propose a new way to solve the counter problem in cassandra-1502[1].
>> Since I do not follow the jira update very carefully, I paste it here and
>> want to let more people comment it and then to see whether its feasible.
>>
>> "Seems like we have not found a solution acceptable to everybody. I tries to
>> propose a new approach. Let's see whether anybody can shed some light on it
>> and make it as reality.
>>
>> 1) We add a basic data structure, called as counter, which is a special type
>> of super column.
>>
>> 2) The name of each column in the counter super column, is the host name of
>> a cassandra node. And the value is the calculated result from that node.
>>
>> 3) WRITE PATH: Once a node receives the add/dec request of a counter, it
>> de-serializes its local counter super column, and update the column named by
>> itself atomically. After that, it propagates the updated column value to
>> other replicas, just like how the mutation of a normal column is propagated
>> to other replicas. Different consistency levels can be supported as before.
>>
>> 4) READ PATH: Depends on the consistency level, contact several replicas,
>> read back the counter super column as whole, and get the latest counter
>> value by summing up all columns in the counter. Read-repair logic can work
>> as before.
>>
>> IMHO, the biggest advantages of this approach, is re-using as many
>> mechanisms already in the code as possible. So it might not so disruptive.
>> But adding new thrift API is inevitable. "
>> NB: If it's feasible, I might not be the right man working on it as I have
>> not touched the internal of cassandra for more than 1 year. I wants to
>> contribute something to help us get consensus.
>>
>> [1]
>> https://issues.apache.org/jira/browse/CASSANDRA-1502?focusedCommentId=12915103&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12915103
>>
>> best regards,
>> hanzhu
>>
>>
>> On Sun, Sep 26, 2010 at 9:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>>> you have misunderstood. if we continue the 1072 approach of writing
>>> counter data to the clock field, this is necessarily incompatible with
>>> the right way of writing counter data to the value field. it's no
>>> longer simply a matter of reversing 1070.
>>>
>>> On Sat, Sep 25, 2010 at 11:50 PM, Zhu Han <sc...@gmail.com> wrote:
>>>> Jonathan,
>>>>
>>>> This is a personnel email.
>>>>
>>>> On Sun, Sep 26, 2010 at 1:27 PM, Jonathan Ellis <jb...@gmail.com>
>>> wrote:
>>>>>
>>>>> On Sat, Sep 25, 2010 at 8:57 PM, Zhu Han <sc...@gmail.com> wrote:
>>>>>> Can we just let the patch committed but mark it as "alpah" or
>>>>>> "experimental"?
>>>>>
>>>>> I explained exactly why that is not a good approach here:
>>>>> http://www.mail-archive.com/dev@cassandra.apache.org/msg00917.html
>>>>>
>>>> Yes, I see. But the clock structure is in truck since Cassandra-1070. We
>>>> still need to clean them
>>>> out, whatever. We need somebody to be volunteer to take this work.
>>>> Considering the complexity
>>>> of Cassandra-1070, the programmer who has the in depth knowledge of this
>>>> patch is preferable. And it
>>>> will take some time to do it.
>>>>
>>>> Fortunately, Johan Oskarsson has promised to take it in the comment of
>>>> Cassandra-1072[1]:
>>>>
>>>> "The clock changes would get into trunk quicker if we didn't, avoiding
>>> the
>>>> extra overhead of a big patch during reviews, merge with trunk, code
>>> updates
>>>> and publication of a new patch.
>>>> If the concern is that we won't attend to the clocks once this patch is
>>> in I
>>>> can promise that we'll look at it straight away. "
>>>>
>>>> And if twitter/digg/simplegeo forks their tree of cassandra, this will
>>> give
>>>> a big marketing opportunities of other NOSQL system supporters. As you
>>> know,
>>>> the competition is quite fierce currently.
>>>>
>>>> So, instead of sticking to the embarrassed situation, why not change to
>>>> another strategy:
>>>>
>>>>> "Fork another experimental tree from 0.7 beta 1 and accept
>>>>> Cassandra-1072. At the same time, start the clean up work on this tree.
>>>>> Once it's finalized , merge them back to 0.7, no matter it's 0.7.1 or
>>> 0.7.2.
>>>>>
>>>>> Hence, these guys from twitter does not need to maintain a huge
>>>>> out-of-tree patch, while the quality impact of cassandra-1072 is still
>>>>> limited.
>>>>
>>>> I do know the pain of maintaining a large patch out of the official tree.
>>>> Once it gets in, everybody will feels much better.
>>>>
>>>> If you give some opportunities to this patch, Johan or others can be
>>> highly
>>>> motivated because all of the community works together. It's a
>>> compromise,
>>>> but it's worth.
>>>>
>>>> [1]
>>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-1072?focusedCommentId=12909234&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909234
>>>>
>>>>
>>>>>
>>>>> --
>>>>> Jonathan Ellis
>>>>> Project Chair, Apache Cassandra
>>>>> co-founder of Riptano, the source for professional Cassandra support
>>>>> http://riptano.com
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of Riptano, the source for professional Cassandra support
>>> http://riptano.com
>>>
>>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Sylvain Lebresne <sy...@yakaz.com>.
In CASSANDRA-1546, I propose an alternative to #1072. At it's core,
it rewrites #1072 without the clocks structure (by splitting the clock into
individual columns, not unlike what Zhu Han proposed in his preceding
mail, but in a row instead of a super column, for reason explained in the
issue).
But it is also my belief that it improves on the actual patch of #1072 in
the following ways:
- it supports increments and decrements
- it supports the usual consistency levels
- it proposes an (optional) solution to the idempotency problem of
increments (it's optional because it has a (fairly slight) performance cost
that some may want to remove if they understand the risk).
When I say, I propose, I mean that I did wrote the patch (attached to the jira
ticket). I've just written it, so it is really under-tested and have a
few details here
and there to fix, but it should already be fairly functional (it
passes basic system
tests).
I welcome all comments on the patch. It has been written with in mind
the goal to
address most of the concerns that have been addressed on those counters since a
few months (both in terms of performance and implementation). It is my
belief that
is reaches this goal, hopefully other will agree.
--
Sylvain
On Mon, Sep 27, 2010 at 5:32 AM, Zhu Han <sc...@gmail.com> wrote:
> I propose a new way to solve the counter problem in cassandra-1502[1].
> Since I do not follow the jira update very carefully, I paste it here and
> want to let more people comment it and then to see whether its feasible.
>
> "Seems like we have not found a solution acceptable to everybody. I tries to
> propose a new approach. Let's see whether anybody can shed some light on it
> and make it as reality.
>
> 1) We add a basic data structure, called as counter, which is a special type
> of super column.
>
> 2) The name of each column in the counter super column, is the host name of
> a cassandra node. And the value is the calculated result from that node.
>
> 3) WRITE PATH: Once a node receives the add/dec request of a counter, it
> de-serializes its local counter super column, and update the column named by
> itself atomically. After that, it propagates the updated column value to
> other replicas, just like how the mutation of a normal column is propagated
> to other replicas. Different consistency levels can be supported as before.
>
> 4) READ PATH: Depends on the consistency level, contact several replicas,
> read back the counter super column as whole, and get the latest counter
> value by summing up all columns in the counter. Read-repair logic can work
> as before.
>
> IMHO, the biggest advantages of this approach, is re-using as many
> mechanisms already in the code as possible. So it might not so disruptive.
> But adding new thrift API is inevitable. "
> NB: If it's feasible, I might not be the right man working on it as I have
> not touched the internal of cassandra for more than 1 year. I wants to
> contribute something to help us get consensus.
>
> [1]
> https://issues.apache.org/jira/browse/CASSANDRA-1502?focusedCommentId=12915103&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12915103
>
> best regards,
> hanzhu
>
>
> On Sun, Sep 26, 2010 at 9:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> you have misunderstood. if we continue the 1072 approach of writing
>> counter data to the clock field, this is necessarily incompatible with
>> the right way of writing counter data to the value field. it's no
>> longer simply a matter of reversing 1070.
>>
>> On Sat, Sep 25, 2010 at 11:50 PM, Zhu Han <sc...@gmail.com> wrote:
>> > Jonathan,
>> >
>> > This is a personnel email.
>> >
>> > On Sun, Sep 26, 2010 at 1:27 PM, Jonathan Ellis <jb...@gmail.com>
>> wrote:
>> >>
>> >> On Sat, Sep 25, 2010 at 8:57 PM, Zhu Han <sc...@gmail.com> wrote:
>> >> > Can we just let the patch committed but mark it as "alpah" or
>> >> > "experimental"?
>> >>
>> >> I explained exactly why that is not a good approach here:
>> >> http://www.mail-archive.com/dev@cassandra.apache.org/msg00917.html
>> >>
>> > Yes, I see. But the clock structure is in truck since Cassandra-1070. We
>> > still need to clean them
>> > out, whatever. We need somebody to be volunteer to take this work.
>> > Considering the complexity
>> > of Cassandra-1070, the programmer who has the in depth knowledge of this
>> > patch is preferable. And it
>> > will take some time to do it.
>> >
>> > Fortunately, Johan Oskarsson has promised to take it in the comment of
>> > Cassandra-1072[1]:
>> >
>> > "The clock changes would get into trunk quicker if we didn't, avoiding
>> the
>> > extra overhead of a big patch during reviews, merge with trunk, code
>> updates
>> > and publication of a new patch.
>> > If the concern is that we won't attend to the clocks once this patch is
>> in I
>> > can promise that we'll look at it straight away. "
>> >
>> > And if twitter/digg/simplegeo forks their tree of cassandra, this will
>> give
>> > a big marketing opportunities of other NOSQL system supporters. As you
>> know,
>> > the competition is quite fierce currently.
>> >
>> > So, instead of sticking to the embarrassed situation, why not change to
>> > another strategy:
>> >
>> >> "Fork another experimental tree from 0.7 beta 1 and accept
>> >> Cassandra-1072. At the same time, start the clean up work on this tree.
>> >> Once it's finalized , merge them back to 0.7, no matter it's 0.7.1 or
>> 0.7.2.
>> >>
>> >> Hence, these guys from twitter does not need to maintain a huge
>> >> out-of-tree patch, while the quality impact of cassandra-1072 is still
>> >> limited.
>> >
>> > I do know the pain of maintaining a large patch out of the official tree.
>> > Once it gets in, everybody will feels much better.
>> >
>> > If you give some opportunities to this patch, Johan or others can be
>> highly
>> > motivated because all of the community works together. It's a
>> compromise,
>> > but it's worth.
>> >
>> > [1]
>> >
>> https://issues.apache.org/jira/browse/CASSANDRA-1072?focusedCommentId=12909234&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909234
>> >
>> >
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of Riptano, the source for professional Cassandra support
>> >> http://riptano.com
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>>
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Zhu Han <sc...@gmail.com>.
I propose a new way to solve the counter problem in cassandra-1502[1].
Since I do not follow the jira update very carefully, I paste it here and
want to let more people comment it and then to see whether its feasible.
"Seems like we have not found a solution acceptable to everybody. I tries to
propose a new approach. Let's see whether anybody can shed some light on it
and make it as reality.
1) We add a basic data structure, called as counter, which is a special type
of super column.
2) The name of each column in the counter super column, is the host name of
a cassandra node. And the value is the calculated result from that node.
3) WRITE PATH: Once a node receives the add/dec request of a counter, it
de-serializes its local counter super column, and update the column named by
itself atomically. After that, it propagates the updated column value to
other replicas, just like how the mutation of a normal column is propagated
to other replicas. Different consistency levels can be supported as before.
4) READ PATH: Depends on the consistency level, contact several replicas,
read back the counter super column as whole, and get the latest counter
value by summing up all columns in the counter. Read-repair logic can work
as before.
IMHO, the biggest advantages of this approach, is re-using as many
mechanisms already in the code as possible. So it might not so disruptive.
But adding new thrift API is inevitable. "
NB: If it's feasible, I might not be the right man working on it as I have
not touched the internal of cassandra for more than 1 year. I wants to
contribute something to help us get consensus.
[1]
https://issues.apache.org/jira/browse/CASSANDRA-1502?focusedCommentId=12915103&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12915103
best regards,
hanzhu
On Sun, Sep 26, 2010 at 9:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> you have misunderstood. if we continue the 1072 approach of writing
> counter data to the clock field, this is necessarily incompatible with
> the right way of writing counter data to the value field. it's no
> longer simply a matter of reversing 1070.
>
> On Sat, Sep 25, 2010 at 11:50 PM, Zhu Han <sc...@gmail.com> wrote:
> > Jonathan,
> >
> > This is a personnel email.
> >
> > On Sun, Sep 26, 2010 at 1:27 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >>
> >> On Sat, Sep 25, 2010 at 8:57 PM, Zhu Han <sc...@gmail.com> wrote:
> >> > Can we just let the patch committed but mark it as "alpah" or
> >> > "experimental"?
> >>
> >> I explained exactly why that is not a good approach here:
> >> http://www.mail-archive.com/dev@cassandra.apache.org/msg00917.html
> >>
> > Yes, I see. But the clock structure is in truck since Cassandra-1070. We
> > still need to clean them
> > out, whatever. We need somebody to be volunteer to take this work.
> > Considering the complexity
> > of Cassandra-1070, the programmer who has the in depth knowledge of this
> > patch is preferable. And it
> > will take some time to do it.
> >
> > Fortunately, Johan Oskarsson has promised to take it in the comment of
> > Cassandra-1072[1]:
> >
> > "The clock changes would get into trunk quicker if we didn't, avoiding
> the
> > extra overhead of a big patch during reviews, merge with trunk, code
> updates
> > and publication of a new patch.
> > If the concern is that we won't attend to the clocks once this patch is
> in I
> > can promise that we'll look at it straight away. "
> >
> > And if twitter/digg/simplegeo forks their tree of cassandra, this will
> give
> > a big marketing opportunities of other NOSQL system supporters. As you
> know,
> > the competition is quite fierce currently.
> >
> > So, instead of sticking to the embarrassed situation, why not change to
> > another strategy:
> >
> >> "Fork another experimental tree from 0.7 beta 1 and accept
> >> Cassandra-1072. At the same time, start the clean up work on this tree.
> >> Once it's finalized , merge them back to 0.7, no matter it's 0.7.1 or
> 0.7.2.
> >>
> >> Hence, these guys from twitter does not need to maintain a huge
> >> out-of-tree patch, while the quality impact of cassandra-1072 is still
> >> limited.
> >
> > I do know the pain of maintaining a large patch out of the official tree.
> > Once it gets in, everybody will feels much better.
> >
> > If you give some opportunities to this patch, Johan or others can be
> highly
> > motivated because all of the community works together. It's a
> compromise,
> > but it's worth.
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/CASSANDRA-1072?focusedCommentId=12909234&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909234
> >
> >
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of Riptano, the source for professional Cassandra support
> >> http://riptano.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
On Sat, Sep 25, 2010 at 8:57 PM, Zhu Han <sc...@gmail.com> wrote:
> Can we just let the patch committed but mark it as "alpah" or
> "experimental"?
I explained exactly why that is not a good approach here:
http://www.mail-archive.com/dev@cassandra.apache.org/msg00917.html
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Zhu Han <sc...@gmail.com>.
>
>
> On the other hand, if the patch authors never bring it up to the
> standards of the rest of the project, well, then it's a good thing we
> didn't commit it under a "commit now, fix later" process.
>
> > Maybe this fork could be prevented if committers could give the guidance?
>
> While it's true -- and unfortunate, mea culpa -- that the rest of us
> weren't involved enough at the beginning of the counter design
> process, that's not the case any more.
>
Can we just let the patch committed but mark it as "alpah" or
"experimental"?
So that we can refine it after commit without breaking any contract?
IIRC, the clock structure was pushed to trunk several months ago
(Cassandra-1070 [1]).
The "commit first, fix later" process is unavoidable even without this
feature. This is my two
cents on it.
[1]: https://issues.apache.org/jira/browse/CASSANDRA-1070
>
> The people most familiar with this patch besides its authors are
> myself and Sylvain, and we have said (starting a month ago) that
> building on the Clock structure looks like the wrong approach. That's
> a big change to the patch, so it's understandable that this is painful
> for the authors.
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Torsten Curdt <tc...@apache.org>.
> 2-4 are bad enough risks that it's worth taking the time to get it
> right before committing it.
So what is the plan for "getting it right" then?
Is it a plan that satisfies all parties?
Can people work together on that plan?
Would be great to discuss the details on the dev list (not in JIRAs)
Just some questions that came to mind when reading the thread.
cheers,
Torsten
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
On Fri, Sep 24, 2010 at 8:45 AM, Chris Goffinet <cg...@chrisgoffinet.com> wrote:
> My two cents on the issue is that, it's an important feature that the community wants, and the work needing to be done to let 1072, isn't worth the amount of effort to delay at this stage. I would much rather get the code in, support the feature and work towards refactoring a better design than completely stalling the feature.
The problem is that 1072 relies on the Clock work that turns out to be
a dead end. That happens sometimes, everyone's written code that
starts off on a path that we later realize doesn't work. But we need
to clean out the deadwood before committing not just because it's the
right thing to do but because storing the counter in the clock
component of a column is dramatically incompatible with storing it in
the value. It's not just a minor tweak. As I said two weeks ago on
the ticket:
----
A major concern is that the clocks are part of the API and schema [now
just schema]. So if we commit this as-is then we have several possible
scenarios:
1. clock change gets in before 0.7.0 and everything is fine
2. clock change isn't ready for 0.7.0 and we block 0.7.0 for it
3. clock change isn't ready for 0.7.0, we release anyway, and break
compatibility w/ the next release
4. clock change isn't ready for 0.7.0 so we postpone it until 0.8
(thus having to maintain both versions until 0.7 is obsolete)
----
2-4 are bad enough risks that it's worth taking the time to get it
right before committing it.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Chris Goffinet <cg...@chrisgoffinet.com>.
I can chime in on that part Joe. The counters weren't the reasons for launch issues. It was just actually the volume of messages we were trying to handle in the cluster, so it didn't matter if it was reads for counts or reads to normal CFs, we just had more on the count side.
It is very unfortunate that it has come down to forking on Github. Digg is in the same position as Twitter on this one. We really do depend on 1072, it's now an integral part of our requirements. We have already felt the pain of trying to keep up with 1072 in our 0.6 branch. I really hope we can come to a conscious on this because I am afraid Digg might have to use Twitter's fork as well.
My two cents on the issue is that, it's an important feature that the community wants, and the work needing to be done to let 1072, isn't worth the amount of effort to delay at this stage. I would much rather get the code in, support the feature and work towards refactoring a better design than completely stalling the feature.
-Chris
On Sep 24, 2010, at 8:11 AM, Joe Stump wrote:
>
> On Sep 24, 2010, at 8:01 AM, Jeremy Hanna wrote:
>
>> Hmmmm... would there be any way that others in the project that are familiar with the design could help the authors to redo some of the elements to remove the internal clock structure and get it to work properly before 0.7.0 is finalized? Not sure if that's feasible, but I would just hate to see *unreconcilable* differences :).
>
> The Digg folks can chime in, but my understanding is a large part of the issues during launch was the vector clock / increment patch they had deployed internally. It fell over. My understanding is it was replaced by Redis.
>
> I don't know enough about the issues around the counters patch, but this is definitely high on SimpleGeo's needs list.
>
> --Joe
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Joe Stump <jo...@joestump.net>.
On Sep 24, 2010, at 8:01 AM, Jeremy Hanna wrote:
> Hmmmm... would there be any way that others in the project that are familiar with the design could help the authors to redo some of the elements to remove the internal clock structure and get it to work properly before 0.7.0 is finalized? Not sure if that's feasible, but I would just hate to see *unreconcilable* differences :).
The Digg folks can chime in, but my understanding is a large part of the issues during launch was the vector clock / increment patch they had deployed internally. It fell over. My understanding is it was replaced by Redis.
I don't know enough about the issues around the counters patch, but this is definitely high on SimpleGeo's needs list.
--Joe
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jeremy Hanna <je...@gmail.com>.
Hmmmm... would there be any way that others in the project that are familiar with the design could help the authors to redo some of the elements to remove the internal clock structure and get it to work properly before 0.7.0 is finalized? Not sure if that's feasible, but I would just hate to see *unreconcilable* differences :).
On Sep 24, 2010, at 9:03 AM, Jonathan Ellis wrote:
> On Fri, Sep 24, 2010 at 5:39 AM, Torsten Curdt <tc...@apache.org> wrote:
>> Cassandra is out of incubation and I am no longer on the PMC ...but this
>> a little concerning.
>>
>> I know this discussion is all about good internal design but - for the sake
>> of the community - isn't there a way this fork could be avoided? I don't have
>> the feeling this is about the work involved implementing it - but
>> rather the lack
>> of direction or consensus how to implement it.
>
> I am disappointed to see this move by an ASF committer [Johann] rather
> than working within the consensus process. But if it's really
> temporary, then I suppose it's not the end of the world.
>
> On the other hand, if the patch authors never bring it up to the
> standards of the rest of the project, well, then it's a good thing we
> didn't commit it under a "commit now, fix later" process.
>
>> Maybe this fork could be prevented if committers could give the guidance?
>
> While it's true -- and unfortunate, mea culpa -- that the rest of us
> weren't involved enough at the beginning of the counter design
> process, that's not the case any more.
>
> The people most familiar with this patch besides its authors are
> myself and Sylvain, and we have said (starting a month ago) that
> building on the Clock structure looks like the wrong approach. That's
> a big change to the patch, so it's understandable that this is painful
> for the authors.
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
On Fri, Sep 24, 2010 at 5:39 AM, Torsten Curdt <tc...@apache.org> wrote:
> Cassandra is out of incubation and I am no longer on the PMC ...but this
> a little concerning.
>
> I know this discussion is all about good internal design but - for the sake
> of the community - isn't there a way this fork could be avoided? I don't have
> the feeling this is about the work involved implementing it - but
> rather the lack
> of direction or consensus how to implement it.
I am disappointed to see this move by an ASF committer [Johann] rather
than working within the consensus process. But if it's really
temporary, then I suppose it's not the end of the world.
On the other hand, if the patch authors never bring it up to the
standards of the rest of the project, well, then it's a good thing we
didn't commit it under a "commit now, fix later" process.
> Maybe this fork could be prevented if committers could give the guidance?
While it's true -- and unfortunate, mea culpa -- that the rest of us
weren't involved enough at the beginning of the counter design
process, that's not the case any more.
The people most familiar with this patch besides its authors are
myself and Sylvain, and we have said (starting a month ago) that
building on the Clock structure looks like the wrong approach. That's
a big change to the patch, so it's understandable that this is painful
for the authors.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Torsten Curdt <tc...@apache.org>.
Cassandra is out of incubation and I am no longer on the PMC ...but this
a little concerning.
I know this discussion is all about good internal design but - for the sake
of the community - isn't there a way this fork could be avoided? I don't have
the feeling this is about the work involved implementing it - but
rather the lack
of direction or consensus how to implement it.
Maybe this fork could be prevented if committers could give the guidance?
Or what am I missing here?
This fork would be bad for the Cassandra community. We already have the
fact that Facebook is running it's own version.
This isn't getting any better like this.
cheers,
Torsten
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Courtney Robinson <sa...@live.co.uk>.
?I've been using Cassandra for a while now and no problems. I have a new
project coming up now that we're penciling out the data structure for.
The best we've come up with has turned into a graph structure i'm just
wanting to know what people think because i
know there are graph db's out there like neo4j etc.
My idea however was to fork hector, remove all the stuff i don't need and
turn it into a graph API sitting on top of Cassandra.
Main reason for this approach is because i am already very familiar with
Cassandra and it will be fast to write a client or modify an existing
one than to learn a new API.
Do you think there are any gotchas in this approach? Any tips, pointers?
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Courtney Robinson <sa...@live.co.uk>.
?Apologies for my last e-mail with the misleading subject i was reading this
thread and mistakenly replied with stuff.
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jeremy Hanna <je...@gmail.com>.
I'm all for some kind of some kind of compromise. It doesn't appear to
be a niche use case for just one company. Twitter, Digg, and SimpleGeo
and several others have said they will be using high volume counters.
They have already cleaned things up and separated it out. I don't know
all the details on the reservations to let it in but it seems like
some compromise could be found so that the feature could make it into
0.7.0.
On Sep 24, 2010, at 7:10 AM, Johan Oskarsson <jo...@oskarsson.nu> wrote:
> Here is an update to where we are at with the counters.
>
> As promised we have published a new patch that adds a separate api method, marked experimental, for the counters in CASSANDRA-1072.
>
> The discussion has now moved to CASSANDRA-1502, a ticket that suggests the internal IClock, reconciler and other classes essential for 1072 should be removed. Doing so would not only severely delay the addition of counters to Cassandra but force a grounds up redesign of 1072 that we believe would not improve the patch. This belief is based on the prototyping Kelvin did for the work that has been refined into 1072.
>
> Since the overwhelming consensus of this discussion thread was that the patch is valuable to the community and worthy of going in, we feel that it should be committed in approximately its current form.
>
> While we work with the Apache community to resolve this issue we’ve decided to put up a temporary version of Cassandra that includes the counters on the Twitter Github account.
> This is a step in releasing some of the systems we’ve built on top of this patch to the open source community in the near future.
>
> On 6 sep 2010, at 21.19, Jonathan Ellis wrote:
>
>> Yes.
>>
>> On Mon, Sep 6, 2010 at 12:11 PM, Johan Oskarsson <jo...@oskarsson.nu> wrote:
>>> The consensus in this thread seems to be moving towards the following todos in order to get 1072 into trunk.
>>>
>>> * create separate api methods for increments
>>> * mark functionality as experimental
>>> * further code cleanup (please comment on jira with specific suggestions)
>>>
>>> Is this a reasonable summary? We'll keep this thread updated as the work progresses.
>>>
>>> /Johan
>>>
>>> On 6 sep 2010, at 16.47, Jonathan Ellis wrote:
>>>
>>>> On Thu, Sep 2, 2010 at 4:10 PM, Torsten Curdt <tc...@apache.org> wrote:
>>>>> The feature could still be marked experimental.
>>>>> That should loosen the contract a little. But at least it would be
>>>>> something to work with.
>>>>
>>>> Maybe this is the best approach, post- code cleanup. Although I'm
>>>> reluctant to add code with known, significant limitations, the problem
>>>> domain seems to be one with no silver bullets.
>>>
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Johan Oskarsson <jo...@oskarsson.nu>.
Here is an update to where we are at with the counters.
As promised we have published a new patch that adds a separate api method, marked experimental, for the counters in CASSANDRA-1072.
The discussion has now moved to CASSANDRA-1502, a ticket that suggests the internal IClock, reconciler and other classes essential for 1072 should be removed. Doing so would not only severely delay the addition of counters to Cassandra but force a grounds up redesign of 1072 that we believe would not improve the patch. This belief is based on the prototyping Kelvin did for the work that has been refined into 1072.
Since the overwhelming consensus of this discussion thread was that the patch is valuable to the community and worthy of going in, we feel that it should be committed in approximately its current form.
While we work with the Apache community to resolve this issue we’ve decided to put up a temporary version of Cassandra that includes the counters on the Twitter Github account.
This is a step in releasing some of the systems we’ve built on top of this patch to the open source community in the near future.
On 6 sep 2010, at 21.19, Jonathan Ellis wrote:
> Yes.
>
> On Mon, Sep 6, 2010 at 12:11 PM, Johan Oskarsson <jo...@oskarsson.nu> wrote:
>> The consensus in this thread seems to be moving towards the following todos in order to get 1072 into trunk.
>>
>> * create separate api methods for increments
>> * mark functionality as experimental
>> * further code cleanup (please comment on jira with specific suggestions)
>>
>> Is this a reasonable summary? We'll keep this thread updated as the work progresses.
>>
>> /Johan
>>
>> On 6 sep 2010, at 16.47, Jonathan Ellis wrote:
>>
>>> On Thu, Sep 2, 2010 at 4:10 PM, Torsten Curdt <tc...@apache.org> wrote:
>>>> The feature could still be marked experimental.
>>>> That should loosen the contract a little. But at least it would be
>>>> something to work with.
>>>
>>> Maybe this is the best approach, post- code cleanup. Although I'm
>>> reluctant to add code with known, significant limitations, the problem
>>> domain seems to be one with no silver bullets.
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
Yes.
On Mon, Sep 6, 2010 at 12:11 PM, Johan Oskarsson <jo...@oskarsson.nu> wrote:
> The consensus in this thread seems to be moving towards the following todos in order to get 1072 into trunk.
>
> * create separate api methods for increments
> * mark functionality as experimental
> * further code cleanup (please comment on jira with specific suggestions)
>
> Is this a reasonable summary? We'll keep this thread updated as the work progresses.
>
> /Johan
>
> On 6 sep 2010, at 16.47, Jonathan Ellis wrote:
>
>> On Thu, Sep 2, 2010 at 4:10 PM, Torsten Curdt <tc...@apache.org> wrote:
>>> The feature could still be marked experimental.
>>> That should loosen the contract a little. But at least it would be
>>> something to work with.
>>
>> Maybe this is the best approach, post- code cleanup. Although I'm
>> reluctant to add code with known, significant limitations, the problem
>> domain seems to be one with no silver bullets.
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jeff Hodges <jh...@twitter.com>.
Sorry, I hit reply thinking that was going to just Johan. My bad. It
is true, though.
On Mon, Sep 6, 2010 at 12:46 PM, Jeff Hodges <jh...@twitter.com> wrote:
> You're a fucking hero.
>
> On Sep 6, 2010 12:12 PM, "Johan Oskarsson" <jo...@oskarsson.nu> wrote:
> The consensus in this thread seems to be moving towards the following todos
> in order to get 1072 into trunk.
>
> * create separate api methods for increments
> * mark functionality as experimental
> * further code cleanup (please comment on jira with specific suggestions)
>
> Is this a reasonable summary? We'll keep this thread updated as the work
> progresses.
>
> /Johan
>
> On 6 sep 2010, at 16.47, Jonathan Ellis wrote:
>
>> On Thu, Sep 2, 2010 at 4:10 PM, Torsten Curdt <tc...
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jeff Hodges <jh...@twitter.com>.
You're a fucking hero.
On Sep 6, 2010 12:12 PM, "Johan Oskarsson" <jo...@oskarsson.nu> wrote:
The consensus in this thread seems to be moving towards the following todos
in order to get 1072 into trunk.
* create separate api methods for increments
* mark functionality as experimental
* further code cleanup (please comment on jira with specific suggestions)
Is this a reasonable summary? We'll keep this thread updated as the work
progresses.
/Johan
On 6 sep 2010, at 16.47, Jonathan Ellis wrote:
> On Thu, Sep 2, 2010 at 4:10 PM, Torsten Curdt <tc...
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Johan Oskarsson <jo...@oskarsson.nu>.
The consensus in this thread seems to be moving towards the following todos in order to get 1072 into trunk.
* create separate api methods for increments
* mark functionality as experimental
* further code cleanup (please comment on jira with specific suggestions)
Is this a reasonable summary? We'll keep this thread updated as the work progresses.
/Johan
On 6 sep 2010, at 16.47, Jonathan Ellis wrote:
> On Thu, Sep 2, 2010 at 4:10 PM, Torsten Curdt <tc...@apache.org> wrote:
>> The feature could still be marked experimental.
>> That should loosen the contract a little. But at least it would be
>> something to work with.
>
> Maybe this is the best approach, post- code cleanup. Although I'm
> reluctant to add code with known, significant limitations, the problem
> domain seems to be one with no silver bullets.
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
On Thu, Sep 2, 2010 at 4:10 PM, Torsten Curdt <tc...@apache.org> wrote:
> The feature could still be marked experimental.
> That should loosen the contract a little. But at least it would be
> something to work with.
Maybe this is the best approach, post- code cleanup. Although I'm
reluctant to add code with known, significant limitations, the problem
domain seems to be one with no silver bullets.
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Torsten Curdt <tc...@apache.org>.
I cannot say anything about the implementation details of the patch or
even the two different approaches. Not sure that even matters that
much at this stage. What can say though is that I got the feeling that
there is a lot of desire and drive in the community to get at least
something in. Ignoring this one has a particular high fork risk IMO.
Especially if the changes are deep and the patch big this sounds like
painful situation for everyone involved.
I understand it adds a contract to maintain and the exact approach is
not set in stone. But I also like the idea of adding both approaches
like Johan suggested. The feature could still be marked experimental.
That should loosen the contract a little. But at least it would be
something to work with.
Another option would be to create a "counter" branch - which of course
also needs to be maintained and merged. But this is still better than
having everyone build and maintain their own set of patches.
Of course the community needs to agree on the general vision first -
that counters would be a great thing to have in Cassandra. If that is
the case ... well ... then let's move into that direction.
My 2 cents
--
Torsten
On Thu, Sep 2, 2010 at 23:46, Adam Samet <ad...@gmail.com> wrote:
> If a new api method is added for counters, the thrift interface Clock
> structure wouldn't be needed, but that's getting to be an
> implementation detail. Whether 1072 is an appropriate step forward is
> tangential to that issue.
>
> The patch has been refactored several times based on JIRA feedback.
> If the objections are code level, perhaps it's time to take this to
> code review to address specific concerns.
>
> On Thu, Sep 2, 2010 at 1:55 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> I still have not seen any response to my other misgivings about 1072
>> that I have raised on the ticket. Specifically, the existing patch is
>> based around a Clock structure that, since 580 is a dead end, is no
>> longer necessary.
>>
>> I'm also uneasy about adding 200k of code that meshes as poorly with
>> the rest of Cassandra as this does. The more it can be split off into
>> separate code paths, the better. Adding its own thrift method is a
>> good start, but it should go deeper than that.
>>
>> On Thu, Sep 2, 2010 at 12:01 PM, Johan Oskarsson <jo...@oskarsson.nu> wrote:
>>> In the last few months Digg and Twitter have been using a counter patch that lets Cassandra act as a high-volume realtime counting system. Atomic counters enable new applications that were previously difficult to implement at scale, including realtime analytics and large-scale systems monitoring.
>>>
>>> Discussion
>>> There are currently two different suggestions for how to implement counters in Cassandra. The discussion has so far been limited to those following the jiras (CASSANDRA-1072 and CASSANDRA-1421) closely and we don’t seem to be nearing a decision. I want to open it up to the Cassandra community at large to get additional feedback.
>>>
>>> Below are very basic and brief introductions to the alternatives. Please help us move forward by reading through the docs and jiras and reply to this thread with your thoughts. Would one or the other, both or neither be suitable for inclusion in Cassandra? Is there a third option? What can we do to reach a decision?
>>>
>>> We believe that both options can coexist; their strengths and weaknesses make them suitable for different use cases.
>>>
>>>
>>> CASSANDRA-1072 + CASSANDRA-1397
>>> https://issues.apache.org/jira/browse/CASSANDRA-1072 (see design doc)
>>> https://issues.apache.org/jira/browse/CASSANDRA-1397
>>>
>>> How does it work?
>>> A node is picked as the primary replica for each write. The context byte array for a column contains (primary replica ip, value). Any previous data with the same ip is reconciled with the new increment and put as the column value.
>>>
>>> Concerns raised
>>> * an increment in flight will be lost if the wrong node goes down
>>> * if an increment operation times out it’s impossible to know if it has been executed or not
>>>
>>> The most recent jira comment proposes a new API method for increments that reflects the different consistency level guarantees.
>>>
>>>
>>> CASSANDRA-1421
>>> https://issues.apache.org/jira/browse/CASSANDRA-1421
>>>
>>> How does it work?
>>> Each increment for a counter is stored as a (UUID, value) tuple. The read operations will read all these increment tuples for a counter, reconcile and return. On a regular interval the values are all read and reconciled into one value to reduce the amount of data required for each read operation.
>>>
>>> Concerns raised
>>> * poor read performance, especially for time-series data
>>> * post aggregation reconciliation issues
>>>
>>>
>>> Again, we feel that both options can co-exist, especially if the 1072 patch uses a new API method that reflects its different consistency level guarantees. Our proposal is to accept 1072 into trunk with the new API method, and when an implementation of 1421 is completed it can be accepted alongside.
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>>
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Adam Samet <ad...@gmail.com>.
If a new api method is added for counters, the thrift interface Clock
structure wouldn't be needed, but that's getting to be an
implementation detail. Whether 1072 is an appropriate step forward is
tangential to that issue.
The patch has been refactored several times based on JIRA feedback.
If the objections are code level, perhaps it's time to take this to
code review to address specific concerns.
On Thu, Sep 2, 2010 at 1:55 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> I still have not seen any response to my other misgivings about 1072
> that I have raised on the ticket. Specifically, the existing patch is
> based around a Clock structure that, since 580 is a dead end, is no
> longer necessary.
>
> I'm also uneasy about adding 200k of code that meshes as poorly with
> the rest of Cassandra as this does. The more it can be split off into
> separate code paths, the better. Adding its own thrift method is a
> good start, but it should go deeper than that.
>
> On Thu, Sep 2, 2010 at 12:01 PM, Johan Oskarsson <jo...@oskarsson.nu> wrote:
>> In the last few months Digg and Twitter have been using a counter patch that lets Cassandra act as a high-volume realtime counting system. Atomic counters enable new applications that were previously difficult to implement at scale, including realtime analytics and large-scale systems monitoring.
>>
>> Discussion
>> There are currently two different suggestions for how to implement counters in Cassandra. The discussion has so far been limited to those following the jiras (CASSANDRA-1072 and CASSANDRA-1421) closely and we don’t seem to be nearing a decision. I want to open it up to the Cassandra community at large to get additional feedback.
>>
>> Below are very basic and brief introductions to the alternatives. Please help us move forward by reading through the docs and jiras and reply to this thread with your thoughts. Would one or the other, both or neither be suitable for inclusion in Cassandra? Is there a third option? What can we do to reach a decision?
>>
>> We believe that both options can coexist; their strengths and weaknesses make them suitable for different use cases.
>>
>>
>> CASSANDRA-1072 + CASSANDRA-1397
>> https://issues.apache.org/jira/browse/CASSANDRA-1072 (see design doc)
>> https://issues.apache.org/jira/browse/CASSANDRA-1397
>>
>> How does it work?
>> A node is picked as the primary replica for each write. The context byte array for a column contains (primary replica ip, value). Any previous data with the same ip is reconciled with the new increment and put as the column value.
>>
>> Concerns raised
>> * an increment in flight will be lost if the wrong node goes down
>> * if an increment operation times out it’s impossible to know if it has been executed or not
>>
>> The most recent jira comment proposes a new API method for increments that reflects the different consistency level guarantees.
>>
>>
>> CASSANDRA-1421
>> https://issues.apache.org/jira/browse/CASSANDRA-1421
>>
>> How does it work?
>> Each increment for a counter is stored as a (UUID, value) tuple. The read operations will read all these increment tuples for a counter, reconcile and return. On a regular interval the values are all read and reconciled into one value to reduce the amount of data required for each read operation.
>>
>> Concerns raised
>> * poor read performance, especially for time-series data
>> * post aggregation reconciliation issues
>>
>>
>> Again, we feel that both options can co-exist, especially if the 1072 patch uses a new API method that reflects its different consistency level guarantees. Our proposal is to accept 1072 into trunk with the new API method, and when an implementation of 1421 is completed it can be accepted alongside.
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: [DISCUSSION] High-volume counters in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
I still have not seen any response to my other misgivings about 1072
that I have raised on the ticket. Specifically, the existing patch is
based around a Clock structure that, since 580 is a dead end, is no
longer necessary.
I'm also uneasy about adding 200k of code that meshes as poorly with
the rest of Cassandra as this does. The more it can be split off into
separate code paths, the better. Adding its own thrift method is a
good start, but it should go deeper than that.
On Thu, Sep 2, 2010 at 12:01 PM, Johan Oskarsson <jo...@oskarsson.nu> wrote:
> In the last few months Digg and Twitter have been using a counter patch that lets Cassandra act as a high-volume realtime counting system. Atomic counters enable new applications that were previously difficult to implement at scale, including realtime analytics and large-scale systems monitoring.
>
> Discussion
> There are currently two different suggestions for how to implement counters in Cassandra. The discussion has so far been limited to those following the jiras (CASSANDRA-1072 and CASSANDRA-1421) closely and we don’t seem to be nearing a decision. I want to open it up to the Cassandra community at large to get additional feedback.
>
> Below are very basic and brief introductions to the alternatives. Please help us move forward by reading through the docs and jiras and reply to this thread with your thoughts. Would one or the other, both or neither be suitable for inclusion in Cassandra? Is there a third option? What can we do to reach a decision?
>
> We believe that both options can coexist; their strengths and weaknesses make them suitable for different use cases.
>
>
> CASSANDRA-1072 + CASSANDRA-1397
> https://issues.apache.org/jira/browse/CASSANDRA-1072 (see design doc)
> https://issues.apache.org/jira/browse/CASSANDRA-1397
>
> How does it work?
> A node is picked as the primary replica for each write. The context byte array for a column contains (primary replica ip, value). Any previous data with the same ip is reconciled with the new increment and put as the column value.
>
> Concerns raised
> * an increment in flight will be lost if the wrong node goes down
> * if an increment operation times out it’s impossible to know if it has been executed or not
>
> The most recent jira comment proposes a new API method for increments that reflects the different consistency level guarantees.
>
>
> CASSANDRA-1421
> https://issues.apache.org/jira/browse/CASSANDRA-1421
>
> How does it work?
> Each increment for a counter is stored as a (UUID, value) tuple. The read operations will read all these increment tuples for a counter, reconcile and return. On a regular interval the values are all read and reconciled into one value to reduce the amount of data required for each read operation.
>
> Concerns raised
> * poor read performance, especially for time-series data
> * post aggregation reconciliation issues
>
>
> Again, we feel that both options can co-exist, especially if the 1072 patch uses a new API method that reflects its different consistency level guarantees. Our proposal is to accept 1072 into trunk with the new API method, and when an implementation of 1421 is completed it can be accepted alongside.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com