You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Saurabh Sehgal <sa...@gmail.com> on 2011/03/25 09:58:16 UTC
data aggregation in Cassandra
Hi All,
I am evaluating Cassandra as a data store for my application.
This is what the format of the data I want to store looks like:
{
timestampuuid: unique time stamp
finite_set_of_values_attribute1: this is a value from a finite set ,
lets say {a,b,c,d}
finite_set_of_values_attribute2: this is a value from a finite set, lets
say {e,f,g}
randomAttribute: xyz
randomAttribute: 123
}
How can I efficiently model this data to answer questions like:
1. From timestampuuid1 to timestampuuid2 I want all values
with finite_set_of_values_attribute1 = "a",
and finite_set_of_values_attribute2 = "e"
2. I want all values from timestamp1 to timestamp2 sorted by
finite_set_of_values_attribute1
Is it possible to do such data aggregations efficiently ? I sense a
map-reduce approach might work better for this. Is there something similar
to aggregate data in such a way in Cassandra ?
Thank you,
Saurabh
Re: data aggregation in Cassandra
Posted by Thomas Bernhardt <be...@yahoo.com>.
May I point to complex event processing, see http://esper.codehaus.org. Esper
provides a SQL-like language for performing the queries, the engine internally
maintains the indexes for answering queries fast. EsperHA runs on Cassandra.
Best regards,
Tom
project lead Esper
________________________________
From: David McNelis <dm...@agentisenergy.com>
To: user@cassandra.apache.org
Sent: Fri, March 25, 2011 7:54:41 AM
Subject: Re: data aggregation in Cassandra
That kind of aggregation is certainly possible today, programmatically... but if
you want to do it in cassandra only, you are out of luck, today. But it sounds
like the project DataStax just announced might help quite a bit with a use case
like that.
David
Sent from my Droid
On Mar 25, 2011 3:58 AM, "Saurabh Sehgal" <sa...@gmail.com> wrote:
> Hi All,
>
> I am evaluating Cassandra as a data store for my application.
>
> This is what the format of the data I want to store looks like:
>
> {
> timestampuuid: unique time stamp
> finite_set_of_values_attribute1: this is a value from a finite set ,
> lets say {a,b,c,d}
> finite_set_of_values_attribute2: this is a value from a finite set, lets
> say {e,f,g}
> randomAttribute: xyz
> randomAttribute: 123
> }
>
> How can I efficiently model this data to answer questions like:
>
> 1. From timestampuuid1 to timestampuuid2 I want all values
> with finite_set_of_values_attribute1 = "a",
> and finite_set_of_values_attribute2 = "e"
> 2. I want all values from timestamp1 to timestamp2 sorted by
> finite_set_of_values_attribute1
>
> Is it possible to do such data aggregations efficiently ? I sense a
> map-reduce approach might work better for this. Is there something similar
> to aggregate data in such a way in Cassandra ?
>
> Thank you,
>
> Saurabh
Re: data aggregation in Cassandra
Posted by David McNelis <dm...@agentisenergy.com>.
That kind of aggregation is certainly possible today, programmatically...
but if you want to do it in cassandra only, you are out of luck, today.
But it sounds like the project DataStax just announced might help quite a
bit with a use case like that.
David
Sent from my Droid
On Mar 25, 2011 3:58 AM, "Saurabh Sehgal" <sa...@gmail.com> wrote:
> Hi All,
>
> I am evaluating Cassandra as a data store for my application.
>
> This is what the format of the data I want to store looks like:
>
> {
> timestampuuid: unique time stamp
> finite_set_of_values_attribute1: this is a value from a finite set ,
> lets say {a,b,c,d}
> finite_set_of_values_attribute2: this is a value from a finite set, lets
> say {e,f,g}
> randomAttribute: xyz
> randomAttribute: 123
> }
>
> How can I efficiently model this data to answer questions like:
>
> 1. From timestampuuid1 to timestampuuid2 I want all values
> with finite_set_of_values_attribute1 = "a",
> and finite_set_of_values_attribute2 = "e"
> 2. I want all values from timestamp1 to timestamp2 sorted by
> finite_set_of_values_attribute1
>
> Is it possible to do such data aggregations efficiently ? I sense a
> map-reduce approach might work better for this. Is there something similar
> to aggregate data in such a way in Cassandra ?
>
> Thank you,
>
> Saurabh
Re: data aggregation in Cassandra
Posted by aaron morton <aa...@thelastpickle.com>.
You can do range based questions inside of one row. For example one row has all of the observations for one day, each observation is represented as a column where (at least the start of the name) is the time of the observation. You can have to 2 billion columns in one row, and the column names are sorted according to the comparator you specify.
If you were to use OOP and say use a time stamp for the key it's going to be difficult to balance the ring. The new writes will happen in the highest range of the ring, so they would be concentrated in the last few nodes in your ring.
A lot depends on your work load. But I would recommend starting with the RP and partitioning the data into rows based on something like a day.
Hope that helps.
Aaron
On 27 Mar 2011, at 15:49, Saurabh Sehgal wrote:
> Thanks for the reply. The reason I want to go with OPP is to do range based queries on time. All queries against the data are going to be time based. With an RPP partitioning scheme, will it be efficient to do range based queries ?
>
> On Mar 26, 2011 9:12 PM, "aaron morton" <aa...@thelastpickle.com> wrote:
> > If you are using OPP you will need to understand how to balance the data around the ring, start with RP until you have an idea why it's now working for you. The RP will transform the key with a hash function, which is then compared to the node tokens to locate the first replica for the data. The OPP uses the raw key. see http://wiki.apache.org/cassandra/Operations#Ring_management and http://ria101.wordpress.com/2010/02/22/cassandra-randompartitioner-vs-orderpreservingpartitioner/
> >
> >
> > Reading 20 to 30 million records will take a while. Perhaps look at http://www.slideshare.net/kevinweil/rainbird-realtime-analytics-at-twitter-strata-2011 and http://www.datastax.com/products/brisk for background.
> >
> > Consider how you can dernormalise to support your queries. e.g. in a CF use keys such as "attr1/value" column name as the time stamp and value as the stuff you need (you could pack all the data you need into a structure like JSON )
> >
> > CF's have a (potentially) large memory overhead. Use fewer and store mixed but related content in them.
> >
> > Hope that helps.
> > Aaron
> >
> >
> > On 26 Mar 2011, at 05:38, Saurabh Sehgal wrote:
> >
> >> Thanks for all the responses.
> >>
> >> My leading questions then are ->
> >>
> >> - Should I go with the OrderPreservingPartitioner based on timestamps so I can do time range queries - is this recommended ? any special cases regarding load balancing I need to keep in mind ? I have read buzz over blogs/forums on how RandomPartitioner yields better load balancing, and it is discouraged to use OrderPreservingPartitioner. Can someone expand/comment on this ?
> >>
> >> - Also, lets say I query all partitioned data between timestampuuid1 and timestampuuid2 (over several weeks) .. this would potentially , in my case, return anywhere to 20 - 30 million records. How would I go about aggregating this data "by hand" ? Will this perform ?
> >>
> >> Since I am only interested in aggregating over a finite set of 10-20 attributes. Does it make more sense to have a column family per finite attribute ? In this case, I do not need to do any aggregation, since all the data for that attribute resides in one column family. Is there an upper bound to the number of column families Cassandra currently supports ?
> >>
> >>
> >>
> >> On Fri, Mar 25, 2011 at 7:31 AM, buddhasystem <po...@bnl.gov> wrote:
> >> Hello Saurabh,
> >>
> >> I have a similar situation, with a more complex data model, and I do an
> >> equivalent of map-reduce "by hand". The redeeming value is that you have
> >> complete freedom in how you hash, and you design the way you store indexes
> >> and similar structures. If there is a pattern in data store, you use it to
> >> your advantage. In the end, you get good performance.
> >>
> >> --
> >> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/data-aggregation-in-Cassandra-tp6206994p6207879.html
> >> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
> >
Re: data aggregation in Cassandra
Posted by Saurabh Sehgal <sa...@gmail.com>.
Thanks for the reply. The reason I want to go with OPP is to do range based
queries on time. All queries against the data are going to be time based.
With an RPP partitioning scheme, will it be efficient to do range based
queries ?
On Mar 26, 2011 9:12 PM, "aaron morton" <aa...@thelastpickle.com> wrote:
> If you are using OPP you will need to understand how to balance the data
around the ring, start with RP until you have an idea why it's now working
for you. The RP will transform the key with a hash function, which is then
compared to the node tokens to locate the first replica for the data. The
OPP uses the raw key. see
http://wiki.apache.org/cassandra/Operations#Ring_management and
http://ria101.wordpress.com/2010/02/22/cassandra-randompartitioner-vs-orderpreservingpartitioner/
>
>
> Reading 20 to 30 million records will take a while. Perhaps look at
http://www.slideshare.net/kevinweil/rainbird-realtime-analytics-at-twitter-strata-2011and
http://www.datastax.com/products/brisk for background.
>
> Consider how you can dernormalise to support your queries. e.g. in a CF
use keys such as "attr1/value" column name as the time stamp and value as
the stuff you need (you could pack all the data you need into a structure
like JSON )
>
> CF's have a (potentially) large memory overhead. Use fewer and store mixed
but related content in them.
>
> Hope that helps.
> Aaron
>
>
> On 26 Mar 2011, at 05:38, Saurabh Sehgal wrote:
>
>> Thanks for all the responses.
>>
>> My leading questions then are ->
>>
>> - Should I go with the OrderPreservingPartitioner based on timestamps so
I can do time range queries - is this recommended ? any special cases
regarding load balancing I need to keep in mind ? I have read buzz over
blogs/forums on how RandomPartitioner yields better load balancing, and it
is discouraged to use OrderPreservingPartitioner. Can someone expand/comment
on this ?
>>
>> - Also, lets say I query all partitioned data between timestampuuid1 and
timestampuuid2 (over several weeks) .. this would potentially , in my case,
return anywhere to 20 - 30 million records. How would I go about aggregating
this data "by hand" ? Will this perform ?
>>
>> Since I am only interested in aggregating over a finite set of 10-20
attributes. Does it make more sense to have a column family per finite
attribute ? In this case, I do not need to do any aggregation, since all the
data for that attribute resides in one column family. Is there an upper
bound to the number of column families Cassandra currently supports ?
>>
>>
>>
>> On Fri, Mar 25, 2011 at 7:31 AM, buddhasystem <po...@bnl.gov> wrote:
>> Hello Saurabh,
>>
>> I have a similar situation, with a more complex data model, and I do an
>> equivalent of map-reduce "by hand". The redeeming value is that you have
>> complete freedom in how you hash, and you design the way you store
indexes
>> and similar structures. If there is a pattern in data store, you use it
to
>> your advantage. In the end, you get good performance.
>>
>> --
>> View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/data-aggregation-in-Cassandra-tp6206994p6207879.html
>> Sent from the cassandra-user@incubator.apache.org mailing list archive at
Nabble.com.
>
Re: data aggregation in Cassandra
Posted by aaron morton <aa...@thelastpickle.com>.
If you are using OPP you will need to understand how to balance the data around the ring, start with RP until you have an idea why it's now working for you. The RP will transform the key with a hash function, which is then compared to the node tokens to locate the first replica for the data. The OPP uses the raw key. see http://wiki.apache.org/cassandra/Operations#Ring_management and http://ria101.wordpress.com/2010/02/22/cassandra-randompartitioner-vs-orderpreservingpartitioner/
Reading 20 to 30 million records will take a while. Perhaps look at http://www.slideshare.net/kevinweil/rainbird-realtime-analytics-at-twitter-strata-2011 and http://www.datastax.com/products/brisk for background.
Consider how you can dernormalise to support your queries. e.g. in a CF use keys such as "attr1/value" column name as the time stamp and value as the stuff you need (you could pack all the data you need into a structure like JSON )
CF's have a (potentially) large memory overhead. Use fewer and store mixed but related content in them.
Hope that helps.
Aaron
On 26 Mar 2011, at 05:38, Saurabh Sehgal wrote:
> Thanks for all the responses.
>
> My leading questions then are ->
>
> - Should I go with the OrderPreservingPartitioner based on timestamps so I can do time range queries - is this recommended ? any special cases regarding load balancing I need to keep in mind ? I have read buzz over blogs/forums on how RandomPartitioner yields better load balancing, and it is discouraged to use OrderPreservingPartitioner. Can someone expand/comment on this ?
>
> - Also, lets say I query all partitioned data between timestampuuid1 and timestampuuid2 (over several weeks) .. this would potentially , in my case, return anywhere to 20 - 30 million records. How would I go about aggregating this data "by hand" ? Will this perform ?
>
> Since I am only interested in aggregating over a finite set of 10-20 attributes. Does it make more sense to have a column family per finite attribute ? In this case, I do not need to do any aggregation, since all the data for that attribute resides in one column family. Is there an upper bound to the number of column families Cassandra currently supports ?
>
>
>
> On Fri, Mar 25, 2011 at 7:31 AM, buddhasystem <po...@bnl.gov> wrote:
> Hello Saurabh,
>
> I have a similar situation, with a more complex data model, and I do an
> equivalent of map-reduce "by hand". The redeeming value is that you have
> complete freedom in how you hash, and you design the way you store indexes
> and similar structures. If there is a pattern in data store, you use it to
> your advantage. In the end, you get good performance.
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/data-aggregation-in-Cassandra-tp6206994p6207879.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: data aggregation in Cassandra
Posted by Saurabh Sehgal <sa...@gmail.com>.
Thanks for all the responses.
My leading questions then are ->
- Should I go with the OrderPreservingPartitioner based on timestamps so I
can do time range queries - is this recommended ? any special cases
regarding load balancing I need to keep in mind ? I have read buzz over
blogs/forums on how RandomPartitioner yields better load balancing, and it
is discouraged to use OrderPreservingPartitioner. Can someone expand/comment
on this ?
- Also, lets say I query all partitioned data between timestampuuid1 and
timestampuuid2 (over several weeks) .. this would potentially , in my case,
return anywhere to 20 - 30 million records. How would I go about aggregating
this data "by hand" ? Will this perform ?
Since I am only interested in aggregating over a finite set of 10-20
attributes. Does it make more sense to have a column family per finite
attribute ? In this case, I do not need to do any aggregation, since all the
data for that attribute resides in one column family. Is there an upper
bound to the number of column families Cassandra currently supports ?
On Fri, Mar 25, 2011 at 7:31 AM, buddhasystem <po...@bnl.gov> wrote:
> Hello Saurabh,
>
> I have a similar situation, with a more complex data model, and I do an
> equivalent of map-reduce "by hand". The redeeming value is that you have
> complete freedom in how you hash, and you design the way you store indexes
> and similar structures. If there is a pattern in data store, you use it to
> your advantage. In the end, you get good performance.
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/data-aggregation-in-Cassandra-tp6206994p6207879.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
Re: data aggregation in Cassandra
Posted by buddhasystem <po...@bnl.gov>.
Hello Saurabh,
I have a similar situation, with a more complex data model, and I do an
equivalent of map-reduce "by hand". The redeeming value is that you have
complete freedom in how you hash, and you design the way you store indexes
and similar structures. If there is a pattern in data store, you use it to
your advantage. In the end, you get good performance.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/data-aggregation-in-Cassandra-tp6206994p6207879.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.