Data model design question

[Nanheng Wu <na...@gmail.com>]

Hi,

 Our team decided to use Cassandra as storage solution to a dataset.
I am very new to the NoSQL world and Cassandra so I am hoping to get
some help from the community: The dataset is pretty simple, we have
for each key a number of columns with values. Each day we compute a
new version of this dataset, the new version will mostly update
existing keys but could also add and delete some keys. (And we'll
build a service that queries Cassandra). A key requirement for us is
we want to keep versions of the dataset and keep N versions around,
this is in case we discover problems in the current version and need
to "roll up" to an older one. I thought about creating a Column Family
per version, this means we will create a new column family every day
and occasionally delete column families according to some truncation
policy. I know Cassandra 0.7 now makes changing schema easier, but is
this a good way to go? I would really like to hear what you guys think
is the better way to handle this. Thank you.

Best,
Alex

Re: Data model design question

[Peter Schuller <pe...@infidyne.com>]

>  Our team decided to use Cassandra as storage solution to a dataset.
> I am very new to the NoSQL world and Cassandra so I am hoping to get
> some help from the community: The dataset is pretty simple, we have
> for each key a number of columns with values. Each day we compute a
> new version of this dataset, the new version will mostly update
> existing keys but could also add and delete some keys. (And we'll
> build a service that queries Cassandra). A key requirement for us is
> we want to keep versions of the dataset and keep N versions around,
> this is in case we discover problems in the current version and need
> to "roll up" to an older one. I thought about creating a Column Family
> per version, this means we will create a new column family every day
> and occasionally delete column families according to some truncation
> policy. I know Cassandra 0.7 now makes changing schema easier, but is
> this a good way to go? I would really like to hear what you guys think
> is the better way to handle this. Thank you.

What you propose would presumably work and in some sense be efficient
if you are intending to re-load the entire data set each day. However,
I wouldn't say column families are intended to be used like this and
it sounds like it would be better to model the data differently.
Column families have implications on e.g. memtables sizes; they are a
bit sensitive in the way you submit them to a cluster (only one schema
change may be propagating at a time, etc). In addition it implies that
you have to keep track of and manage the schema somehow at a
meta-level.

While I don't see an actual unavoidable problem with it (e.g. memtable
sizes is not an issue if you completely stop writing to old memtables,
schema change propagation can of course be solved, etc), my feeling is
that you are likely better off modelling your data differently.

It sounds like a possible way to go  would be a CF with super columns
whose names are your time periods and whose values are your columns.
That assumes a small amount of data per row in each period. This way
you will have access to the latest version of your data by doing a
slice of size 1 (sorting on the period), while older versions will be
remain accessible and individually delete:able. This assumes each
super column is updated completely or not at all, and that you're not
having to do merges across multiple super columns in your application.

Something to keep in mind here is that all insertions of new data and
updates will be to pre-existing rows (unless the row key is completely
new) so you would expect reads to be spread out over sstables on disk.

An alternative is to introduce a level of indirection (if you can live
with that); have a column family that maps rows to columns whose names
are your periods and whose values are row keys for another column
family. Then each row in the pointed-to column family will be your
data for that particular period. This would probably be advantageous
in particular if you intend to have a lot of data associated with each
key, since the data would directly map to a single row in Cassandra
(and large rows are okay and give you efficient access to individual
columns or column ranges).

(But it still assumes you're updating either all the data or no data
for a given row key in a particular time period.)

-- 
/ Peter Schuller

Re: Data model design question

[André Fiedler <fi...@googlemail.com>]

There´s a simple comparison that helped me a lot taking my first steps with
cassandra:

keyspace = database
CF = table

;o)

And this one: http://arin.me/blog/wtf-is-a-supercolumn-cassandra-data-model

Btw,  love the new secondary indexes in cassandra 0.7, this saves a lot of
time and work! But don´t use beta-3! It has some buggy limitations. ;o)
Instead, svn check out the latest branch from here:
https://svn.apache.org/repos/asf/cassandra/branches/cassandra-0.7/
Than make a:

$ ant clean
$ ant jar

in that directory to build the project. And you are good to go. Works for
me. :o)

I hope his helps a bit starting your project. ;o)
greetings André from Germany


2010/11/19 Nanheng Wu <na...@gmail.com>

> Hi,
>
>  Our team decided to use Cassandra as storage solution to a dataset.
> I am very new to the NoSQL world and Cassandra so I am hoping to get
> some help from the community: The dataset is pretty simple, we have
> for each key a number of columns with values. Each day we compute a
> new version of this dataset, the new version will mostly update
> existing keys but could also add and delete some keys. (And we'll
> build a service that queries Cassandra). A key requirement for us is
> we want to keep versions of the dataset and keep N versions around,
> this is in case we discover problems in the current version and need
> to "roll up" to an older one. I thought about creating a Column Family
> per version, this means we will create a new column family every day
> and occasionally delete column families according to some truncation
> policy. I know Cassandra 0.7 now makes changing schema easier, but is
> this a good way to go? I would really like to hear what you guys think
> is the better way to handle this. Thank you.
>
> Best,
> Alex
>