You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Robert Wille <rw...@fold3.com> on 2014/12/03 18:24:37 UTC
Cassandra taking snapshots automatically?
I built my first multi-node cluster and populated it with a bunch of data, and ran out of space far more quickly than I expected. On one node, I ended up with 76 snapshots, consuming a total of 220 GB of space. I only have 40 GB of data. It took several snapshots per hour, sometimes within a minute of each other. I don’t know why it would have any snapshots at all. I never consciously asked it to take a snapshot. I didn’t truncate or drop any CF’s or keyspaces or make any schema changes, and certainly not 76 times.
Any idea what would have caused Cassandra to take all these snapshots, and how I can make it stop?
Thanks in advance
Robert
Re: Cassandra taking snapshots automatically?
Posted by Robert Wille <rw...@fold3.com>.
No. auto_snapshot is turned on, but not snapshot_before_compaction.
On Dec 3, 2014, at 10:30 AM, Eric Stevens <mi...@gmail.com>> wrote:
Do you have snapshot_before_compaction enabled?
http://datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__snapshot_before_compaction
On Wed Dec 03 2014 at 10:25:12 AM Robert Wille <rw...@fold3.com>> wrote:
I built my first multi-node cluster and populated it with a bunch of data, and ran out of space far more quickly than I expected. On one node, I ended up with 76 snapshots, consuming a total of 220 GB of space. I only have 40 GB of data. It took several snapshots per hour, sometimes within a minute of each other. I don’t know why it would have any snapshots at all. I never consciously asked it to take a snapshot. I didn’t truncate or drop any CF’s or keyspaces or make any schema changes, and certainly not 76 times.
Any idea what would have caused Cassandra to take all these snapshots, and how I can make it stop?
Thanks in advance
Robert
Re: Cassandra taking snapshots automatically?
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Dec 3, 2014 at 10:46 AM, Robert Wille <rw...@fold3.com> wrote:
> No. auto_snapshot is turned on, but snapshot_before_compaction is off.
>
> Maybe this will shed some light on it. I tried running nodetool repair.
> I got several messages saying "Lost notification. You should check server
> log for repair status of keyspace test2_browse”.
>
> I looked in system.log, and I have errors where repair is trying to
> create a snapshot. Not sure why repair is trying to create snapshots, or
> why it is failing. I also now have about 200 snapshots. One table has just
> one. Another table has 124.
>
As mentioned in another thread, this surprising behavior is likely the
result of sequential repair, which involves snapshotting.
I am somewhat redundantly pasting the relevant ticket here for future
googlers.
https://issues.apache.org/jira/browse/CASSANDRA-5950
=Rob
Re: Cassandra taking snapshots automatically?
Posted by Robert Wille <rw...@fold3.com>.
No. auto_snapshot is turned on, but snapshot_before_compaction is off.
Maybe this will shed some light on it. I tried running nodetool repair. I got several messages saying "Lost notification. You should check server log for repair status of keyspace test2_browse”.
I looked in system.log, and I have errors where repair is trying to create a snapshot. Not sure why repair is trying to create snapshots, or why it is failing. I also now have about 200 snapshots. One table has just one. Another table has 124.
There’s so much that’s so odd about this.
I’m running 2.0.11.
Robert
On Dec 3, 2014, at 10:30 AM, Eric Stevens <mi...@gmail.com>> wrote:
Do you have snapshot_before_compaction enabled?
http://datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__snapshot_before_compaction
On Wed Dec 03 2014 at 10:25:12 AM Robert Wille <rw...@fold3.com>> wrote:
I built my first multi-node cluster and populated it with a bunch of data, and ran out of space far more quickly than I expected. On one node, I ended up with 76 snapshots, consuming a total of 220 GB of space. I only have 40 GB of data. It took several snapshots per hour, sometimes within a minute of each other. I don’t know why it would have any snapshots at all. I never consciously asked it to take a snapshot. I didn’t truncate or drop any CF’s or keyspaces or make any schema changes, and certainly not 76 times.
Any idea what would have caused Cassandra to take all these snapshots, and how I can make it stop?
Thanks in advance
Robert
Re: Cassandra taking snapshots automatically?
Posted by Eric Stevens <mi...@gmail.com>.
Do you have snapshot_before_compaction enabled?
http://datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__snapshot_before_compaction
On Wed Dec 03 2014 at 10:25:12 AM Robert Wille <rw...@fold3.com> wrote:
> I built my first multi-node cluster and populated it with a bunch of data,
> and ran out of space far more quickly than I expected. On one node, I ended
> up with 76 snapshots, consuming a total of 220 GB of space. I only have 40
> GB of data. It took several snapshots per hour, sometimes within a minute
> of each other. I don’t know why it would have any snapshots at all. I never
> consciously asked it to take a snapshot. I didn’t truncate or drop any CF’s
> or keyspaces or make any schema changes, and certainly not 76 times.
>
> Any idea what would have caused Cassandra to take all these snapshots, and
> how I can make it stop?
>
> Thanks in advance
>
> Robert
>
>