Using Cassandra as cache

[Pavel Kogan <pa...@cortica.com>]

Hi,

In our project, many distributed modules sending each other binary blobs,
up to 100-200kb each in average. Small JSONs are being sent over message
queue, while Cassandra is being used as temporary storage for blobs. We are
using Cassandra instead of in memory distributed cache like Couch due to
following reasons: (1) We don't wan't to be limited by RAM size (2) We are
using intensively ordered composite keys and ranges (it is not simple
key/value cache).

We don't use TTL mechanism for several reasons. Major reason is that we
need to reclaim free disk space immediately and not after 10 days
(gc_grace). We are very limited in disk space cause traffic is intensive
and blobs are big.

So what we did is creating every hour new keyspace named yyyy_MM_dd_HH and
when disk becomes full, script running in crontrab on each node drops
keyspace with "IF EXISTS" flag, and deletes whole keyspace folder. That way
whole process is very clean and no garbage is left on disk.

Keyspace is created by first module in flow on hourly basis and its name is
being sent over message queue, to avoid possible problems. All modules read
and write with consistency ONE and of cause there is no replication.

Actually it works nice but we have several problems:
1) When new keyspace with its columnfamilies is being just created (every
round hour), sometimes other modules failed to read/write data, and we lose
request. Can it be that creation of keyspace and columnfamilies is async
operation or there is propagation time between nodes?

2) We are reading and writing intensively, and usually I don't need the
data for more than 1-2 hours. What optimizations can I do to increase my
small cluster read performance? Cluster configuration - 3 identical nodes:
i7 3GHz, SSD 120Gb, 16Gb RAM, CentOS 6.

Hope not too much text :)

Thanks,
  Pavel

Re: Using Cassandra as cache

[Robert Coli <rc...@eventbrite.com>]

On Fri, Jun 20, 2014 at 3:09 PM, DuyHai Doan <do...@gmail.com> wrote:
> @Robert: do we still need to cleanup manually snapshot when truncating ?
I remembered that on the 1.2 branch, even though the auto_snapshot param
was set to false, truncating leads to snapshot creation that forced us to
manually remove the snapshot folder on disk

Both DROP and TRUNCATE auto_snapshot by default, yes.

> Can you please explain what problems DROP/CREATE keyspace may cause?

Check JIRA, but in general currently DROP/CREATE same entity can cause
bugs. DROP/CREATE with always unique names should be fine.

> Seems like truncate working per column family and I have up to 10.

So TRUNCATE all 10?

> Can working RF=2 and CL=ANY result in any problem with consistency? I am
not sure I can have problems with consistency if I don't do updates, only
writes and reads. Can I?

CL=ANY is a problem with consistency. Wherever you read the instruction to
consider ever using it for any reason, strongly consider never visiting
that site again.

RF=2 means you can't use QUORUM without using CL.ALL. Not being able to use
QUORUM can result in consistency problems. Using CL.ALL results in
availability problems.

=Rob

Re: Using Cassandra as cache

[DuyHai Doan <do...@gmail.com>]

Schema propagation takes times:
https://issues.apache.org/jira/browse/CASSANDRA-5725

@Robert: do we still need to cleanup manually snapshot when truncating ? I
remembered that on the 1.2 branch, even though the auto_snapshot param was
set to false, truncating leads to snapshot creation that forced us to
manually remove the snapshot folder on disk


On Sat, Jun 21, 2014 at 12:01 AM, Robert Stupp <sn...@snazy.de> wrote:

>
> Am 20.06.2014 um 23:48 schrieb Pavel Kogan <pa...@cortica.com>:
>
> > 1) When new keyspace with its columnfamilies is being just created
> (every round hour), sometimes other modules failed to read/write data, and
> we lose request. Can it be that creation of keyspace and columnfamilies is
> async operation or there is propagation time between nodes?
>
> Schema needs to "settle down" (nodes actually agree on a common view) -
> this may take several seconds until all nodes have that common view. Turn
> on DEBUG output in Java driver for example to see these messages.
> CL ONE requires the "one" node to be up and running - if that node's not
> running your request will definitely fail. Maybe you want to try CL ANY or
> increase RF to 2.
>
> > 2) We are reading and writing intensively, and usually I don't need the
> data for more than 1-2 hours. What optimizations can I do to increase my
> small cluster read performance? Cluster configuration - 3 identical nodes:
> i7 3GHz, SSD 120Gb, 16Gb RAM, CentOS 6.
>
> Depending on the data, table layout, access patterns and C* version try
> with various key cache and maybe row cache configurations in both table
> options and cassandra.yaml
>
>

Re: Using Cassandra as cache

[Pavel Kogan <pa...@cortica.com>]

Thank you all,

The issue was resolved (or more exactly bypassed) by adding small python
script running hourly in cron on 1-2 nodes, which pre-provision next hour
keyspace. One hour is definitely enough time for scheme propagation.

Regards,
  Pavel


On Sun, Jun 22, 2014 at 9:35 AM, Robert Stupp <sn...@snazy.de> wrote:

>
> Am 21.06.2014 um 00:37 schrieb Pavel Kogan <pa...@cortica.com>:
>
> > Thanks,
> >
> > Is there any code way to know when the scheme finished to settle down?
>
> Yep - take a look at
> com.datastax.driver.core.ControlConnection#waitForSchemaAgreement in the
> Java Driver source. It basically compares the 'schema_version' column in
> system.peers against the 'schema_version' column in system.local until
> there's only one distinct value.
>
> > Can working RF=2 and CL=ANY result in any problem with consistency? I am
> not sure I can have problems with consistency if I don't do updates, only
> writes and reads. Can I?
>
> Why should it? CL ANY allows you to push updates without the requirement
> that the node(s) that own the key need to be up. Although you do not have
> the guarantee that reads will immediately show the updates. BTW updates =
> insert = upsert ;)
>
> > By the way I am using Cassandra 2.0.8.
>
>

Re: Using Cassandra as cache

[Robert Stupp <sn...@snazy.de>]

Am 21.06.2014 um 00:37 schrieb Pavel Kogan <pa...@cortica.com>:

> Thanks,
> 
> Is there any code way to know when the scheme finished to settle down?

Yep - take a look at com.datastax.driver.core.ControlConnection#waitForSchemaAgreement in the Java Driver source. It basically compares the 'schema_version' column in system.peers against the 'schema_version' column in system.local until there's only one distinct value.

> Can working RF=2 and CL=ANY result in any problem with consistency? I am not sure I can have problems with consistency if I don't do updates, only writes and reads. Can I?

Why should it? CL ANY allows you to push updates without the requirement that the node(s) that own the key need to be up. Although you do not have the guarantee that reads will immediately show the updates. BTW updates = insert = upsert ;)

> By the way I am using Cassandra 2.0.8.

Re: Using Cassandra as cache

[Pavel Kogan <pa...@cortica.com>]

Thanks,

Is there any code way to know when the scheme finished to settle down?
Can working RF=2 and CL=ANY result in any problem with consistency? I am
not sure I can have problems with consistency if I don't do updates, only
writes and reads. Can I?

By the way I am using Cassandra 2.0.8.

Pavel



On Fri, Jun 20, 2014 at 6:01 PM, Robert Stupp <sn...@snazy.de> wrote:

>
> Am 20.06.2014 um 23:48 schrieb Pavel Kogan <pa...@cortica.com>:
>
> > 1) When new keyspace with its columnfamilies is being just created
> (every round hour), sometimes other modules failed to read/write data, and
> we lose request. Can it be that creation of keyspace and columnfamilies is
> async operation or there is propagation time between nodes?
>
> Schema needs to "settle down" (nodes actually agree on a common view) -
> this may take several seconds until all nodes have that common view. Turn
> on DEBUG output in Java driver for example to see these messages.
> CL ONE requires the "one" node to be up and running - if that node's not
> running your request will definitely fail. Maybe you want to try CL ANY or
> increase RF to 2.
>
> > 2) We are reading and writing intensively, and usually I don't need the
> data for more than 1-2 hours. What optimizations can I do to increase my
> small cluster read performance? Cluster configuration - 3 identical nodes:
> i7 3GHz, SSD 120Gb, 16Gb RAM, CentOS 6.
>
> Depending on the data, table layout, access patterns and C* version try
> with various key cache and maybe row cache configurations in both table
> options and cassandra.yaml
>
>

Re: Using Cassandra as cache

[Robert Stupp <sn...@snazy.de>]

Am 20.06.2014 um 23:48 schrieb Pavel Kogan <pa...@cortica.com>:

> 1) When new keyspace with its columnfamilies is being just created (every round hour), sometimes other modules failed to read/write data, and we lose request. Can it be that creation of keyspace and columnfamilies is async operation or there is propagation time between nodes? 

Schema needs to "settle down" (nodes actually agree on a common view) - this may take several seconds until all nodes have that common view. Turn on DEBUG output in Java driver for example to see these messages.
CL ONE requires the "one" node to be up and running - if that node's not running your request will definitely fail. Maybe you want to try CL ANY or increase RF to 2.

> 2) We are reading and writing intensively, and usually I don't need the data for more than 1-2 hours. What optimizations can I do to increase my small cluster read performance? Cluster configuration - 3 identical nodes: i7 3GHz, SSD 120Gb, 16Gb RAM, CentOS 6.

Depending on the data, table layout, access patterns and C* version try with various key cache and maybe row cache configurations in both table options and cassandra.yaml

Re: Using Cassandra as cache

[Pavel Kogan <pa...@cortica.com>]

Thanks Robert,

Can you please explain what problems DROP/CREATE keyspace may cause?
Seems like truncate working per column family and I have up to 10.
What I should I delete from disk in that case? I can't delete whole folder
right? I need to delete all content under each cf folder, but not folders?
Correct?

Pavel



On Fri, Jun 20, 2014 at 6:01 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Fri, Jun 20, 2014 at 2:48 PM, Pavel Kogan <pa...@cortica.com>
> wrote:
>
>> So what we did is creating every hour new keyspace named yyyy_MM_dd_HH
>> and when disk becomes full, script running in crontrab on each node drops
>> keyspace with "IF EXISTS" flag, and deletes whole keyspace folder. That way
>> whole process is very clean and no garbage is left on disk.
>>
>
> I've recommended a similar technique in the past, but with alternating
> between Keyspace_A and Keyspace_B. That way you just TRUNCATE them instead
> of having to DROP.
>
> DROP/CREATE keyspace have problems that TRUNCATE do not. Perhaps use a
> TRUNCATE oriented technique?
>
> =Rob
>
>

Re: Using Cassandra as cache

[Robert Coli <rc...@eventbrite.com>]

On Fri, Jun 20, 2014 at 2:48 PM, Pavel Kogan <pa...@cortica.com>
wrote:

> So what we did is creating every hour new keyspace named yyyy_MM_dd_HH and
> when disk becomes full, script running in crontrab on each node drops
> keyspace with "IF EXISTS" flag, and deletes whole keyspace folder. That way
> whole process is very clean and no garbage is left on disk.
>

I've recommended a similar technique in the past, but with alternating
between Keyspace_A and Keyspace_B. That way you just TRUNCATE them instead
of having to DROP.

DROP/CREATE keyspace have problems that TRUNCATE do not. Perhaps use a
TRUNCATE oriented technique?

=Rob