You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Andrey Ilinykh <ai...@gmail.com> on 2012/09/26 18:30:04 UTC

Why data tripled in size after repair?

Hello everybody!
I have 3 node cluster with replication factor of 3.
each node has 800G disk and it used to have 100G of data.
What is strange every time I run repair data takes almost 3 times more
- 270G, then I run compaction and get 100G back.
Unfortunately, yesterday I forget to compact and run repair again (at
that moment I had around 270G). As result I have 720G on each node.
I run compaction again and get a lot of warnings like this

WARN [CompactionExecutor:732] 2012-09-26 16:13:00,745
CompactionTask.java (line 84) insufficient space to compact all
requested files

which makes sense, because I'm almost out of disk space.

So, I have two questions.

1. Why repair almost triples data size?

2. How to compact my data back to 100G?

Thank you,
  Andrey

Re: Why data tripled in size after repair?

Posted by Andrey Ilinykh <ai...@gmail.com>.
On Tue, Oct 2, 2012 at 12:05 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
>> It's in the 1.1 branch; I don't remember if it went into a release
>> yet. If not, it'll be in the next 1.1.x release.
>
> As the ticket says, this is in since 1.1.1. I don't pretend this is
> well documented, but it's in.
>
Nope. It is in 1.1.1 only. I run 1.1.5, it doesn't have it.

Re: Why data tripled in size after repair?

Posted by Sylvain Lebresne <sy...@datastax.com>.
> It's in the 1.1 branch; I don't remember if it went into a release
> yet. If not, it'll be in the next 1.1.x release.

As the ticket says, this is in since 1.1.1. I don't pretend this is
well documented, but it's in.

--
Sylvain

Re: Why data tripled in size after repair?

Posted by Peter Schuller <pe...@infidyne.com>.
> It looks like what I need. Couple questions.
> Does it work with RandomPartinioner only? I use ByteOrderedPartitioner.

I believe it should work with BOP based on cursory re-examination of
the patch. I could be wrong.

> I don't see it as part of any release. Am I supposed to build my own
> version of cassandra?

It's in the 1.1 branch; I don't remember if it went into a release
yet. If not, it'll be in the next 1.1.x release.

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)

Re: Why data tripled in size after repair?

Posted by Andrey Ilinykh <ai...@gmail.com>.
On Wed, Sep 26, 2012 at 12:36 PM, Peter Schuller
<pe...@infidyne.com> wrote:
>> What is strange every time I run repair data takes almost 3 times more
>> - 270G, then I run compaction and get 100G back.
>
> https://issues.apache.org/jira/browse/CASSANDRA-2699 outlines the
> maion issues with repair. In short - in your case the limited
> granularity of merkle trees is causing too much data to be streamed
> (effectively duplicate data).
> https://issues.apache.org/jira/browse/CASSANDRA-3912 may be a bandaid
> for you in that it allows granularity to be much finer, and the
> process to be more incremental.
>
Thank you, Peter!
It looks like what I need. Couple questions.
Does it work with RandomPartinioner only? I use ByteOrderedPartitioner.
I don't see it as part of any release. Am I supposed to build my own
version of cassandra?

Andrey

Re: Why data tripled in size after repair?

Posted by Peter Schuller <pe...@infidyne.com>.
> What is strange every time I run repair data takes almost 3 times more
> - 270G, then I run compaction and get 100G back.

https://issues.apache.org/jira/browse/CASSANDRA-2699 outlines the
maion issues with repair. In short - in your case the limited
granularity of merkle trees is causing too much data to be streamed
(effectively duplicate data).
https://issues.apache.org/jira/browse/CASSANDRA-3912 may be a bandaid
for you in that it allows granularity to be much finer, and the
process to be more incremental.

A 'nodetool compact' decreases disk space temporarily as you have
noticed, but it may also have a long-term negative effect on steady
state disk space usage depending on your workload. If you've got a
workload that's not limited to insertions only (i.e., you have
overwrites/deletes), a major compaction will tend to push steady state
disk space usage up - because you're creating a single sstable bigger
than what would normally happen, and it takes more total disk space
before it will be part of a compaction again.

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)

Re: Why data tripled in size after repair?

Posted by Sylvain Lebresne <sy...@datastax.com>.
> I see. It explains why I get 85G + 85G instead of 90G. But after next
> repair I have six extra files 75G each,
> how is it possible?

Maybe you've run repair on other nodes? Basically repair is a fairly
blind process. If it consider that a given range (and by range I mean
here the ones that repair hashes, so they're suppose to be small
ranges) is inconsistent between two peers, it doesn't know which peers
is up to date (and in fact it cannot, it is possible that no node is
more up to date than the other on the whole range, even if said range
is small, at least in theory).

> It looks like repair is done per sstable, not CF. Is it possible?

No, repair is done on the CF, not on individual sstables.


> Does Merklee tree calculation algorithm use sstables flushed on hard drive or it uses mem tables also?

It triggers a flush, wait for it, and then use the on-disk sstables.
So in theory, since the order to flush is sent to each replica at the
same time, all the replica will trigger the flush in a very short
interval and thus consider data set that only from that short
interval. So in a cluster with a high write load, it is expected that
a bit of the inconsistence is due to that short interval, but it
should be relatively negligible. That's the theory at least. But
clearly your case (ordered partitioner with intensive update covering
the whole range) does maximize that inconsistency. Still, it shouldn't
be that dramatic.

However your use of the ordered partitionner might not be
insignificant as it's much less used and repair does have a few
specific bits for it. Do you mind opening a ticket on jira with a
summary of your configuration/problem? I'll look if I can spot
something wrong relating to the order partitioner but in any that'll
be simpler to track what's wrong there.

--
Sylvain

Re: Why data tripled in size after repair?

Posted by Andrey Ilinykh <ai...@gmail.com>.
On Thu, Sep 27, 2012 at 9:52 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
>> I don't understand why it copied data twice. In worst case scenario it
>> should copy everything (~90G)
>
> Sadly no, repair is currently peer-to-peer based (there is a ticket to
> fix it: https://issues.apache.org/jira/browse/CASSANDRA-3200, but
> that's not trivial). This mean that you can end up with RF times the
> data after a repair. Obviously that should be a worst case scenario as
> it implies everything is repaired, but at least the triplicate part is
> a problem, but a know and not so easy to fix one.

I see. It explains why I get 85G + 85G instead of 90G. But after next
repair I have six extra files 75G each,
how is it possible? It looks like repair is done per sstable, not CF.
Is it possible?

>
> Is it possible that each time you've ran repair, one of the node in
> the cluster was very out of sync with the other nodes. Maybe a node
> that has crashed for a long time?
>
No, nodes go down time to time (OOM), but I restart them
automatically. But my specific is - I have order preserved partitioner
and update intensively every 5th or 10th row.
As far as I understand, because of that when Merklee tree is
calculated, in every range I have several "hot" rows.  These rows are
good candidates to be inconsistant. There is one thing I don't
understand. Does Merklee tree calculation algorithm use sstables
flushed on hard drive or it uses mem tables also?
Let's say I have "hot" row which sits in memory in one node but
flushed out in another. Is the any difference in Merklee trees?

Thank you,
  Andrey

Re: Why data tripled in size after repair?

Posted by Sylvain Lebresne <sy...@datastax.com>.
> I don't understand why it copied data twice. In worst case scenario it
> should copy everything (~90G)

Sadly no, repair is currently peer-to-peer based (there is a ticket to
fix it: https://issues.apache.org/jira/browse/CASSANDRA-3200, but
that's not trivial). This mean that you can end up with RF times the
data after a repair. Obviously that should be a worst case scenario as
it implies everything is repaired, but at least the triplicate part is
a problem, but a know and not so easy to fix one.

Is it possible that each time you've ran repair, one of the node in
the cluster was very out of sync with the other nodes. Maybe a node
that has crashed for a long time?

--
Sylvain

Re: Why data tripled in size after repair?

Posted by Andrey Ilinykh <ai...@gmail.com>.
On Wed, Sep 26, 2012 at 11:07 AM, Rob Coli <rc...@palominodb.com> wrote:
> On Wed, Sep 26, 2012 at 9:30 AM, Andrey Ilinykh <ai...@gmail.com> wrote:
>> [ repair ballooned my data size ]
>> 1. Why repair almost triples data size?
>
> You didn't mention what version of cassandra you're running. In some
> old versions of cassandra (prior to 1.0), repair often creates even
> more extraneous data than it should by design.
>
Thank you for reply.

I run 1.1.5

Honestly, I don't understand what is going on.

I ran major compaction on Sep 15
as result I had one big sstable and several smalls. This is one biggest:

-rw-rw-r-- 1 ubuntu ubuntu  90G Sep 15 12:56 Bidgely-rawstreams-he-8475-Data.db

On Sep 22 (one week later)I ran repair and get two more sstables:

-rw-rw-r-- 1 ubuntu ubuntu  85G Sep 22 00:41 Bidgely-rawstreams-he-8605-Data.db
-rw-rw-r-- 1 ubuntu ubuntu  86G Sep 22 00:45 Bidgely-rawstreams-he-8606-Data.db

I don't understand why it copied data twice. In worst case scenario it
should copy everything (~90G), but data is triplicates (90G + 85G
+85G).
Yesterday I ran repair one more time, six(!) more big sstables are
added. It does'n make any sense! What do I miss?

-rw-rw-r-- 1 ubuntu ubuntu  75G Sep 26 09:43 Bidgely-rawstreams-he-8785-Data.db
-rw-rw-r-- 1 ubuntu ubuntu  77G Sep 26 09:45 Bidgely-rawstreams-he-8788-Data.db
-rw-rw-r-- 1 ubuntu ubuntu  76G Sep 26 11:54 Bidgely-rawstreams-he-8793-Data.db
-rw-rw-r-- 1 ubuntu ubuntu  75G Sep 26 11:55 Bidgely-rawstreams-he-8797-Data.db
-rw-rw-r-- 1 ubuntu ubuntu  76G Sep 26 14:03 Bidgely-rawstreams-he-8804-Data.db
-rw-rw-r-- 1 ubuntu ubuntu  75G Sep 26 14:03 Bidgely-rawstreams-he-8807-Data.db

Even I somehow compact it back to 100G, I will have the same problem
very soon. What did I do wrong?

Andrey

Re: Why data tripled in size after repair?

Posted by Rob Coli <rc...@palominodb.com>.
On Wed, Sep 26, 2012 at 9:30 AM, Andrey Ilinykh <ai...@gmail.com> wrote:
> [ repair ballooned my data size ]
> 1. Why repair almost triples data size?

You didn't mention what version of cassandra you're running. In some
old versions of cassandra (prior to 1.0), repair often creates even
more extraneous data than it should by design.

However, by design, Repair repairs differing ranges based on merkle
trees. Merkle trees are an optimization, what you trade for the
optimization is over-repair. When you have multiple replicas, each
over-repairs. If you are running repair on your whole cluster, this is
why you should use repair -pr, as it reduces the per-replica
over-repair.

> 2. How to compact my data back to 100G?

1) do a major compaction, one CF at a time. if you only have one CF,
you're out of luck because you don't have enough headroom.
2) then convince someone to write "sstablesplit" so you can turn your
100G sstable into [n] smaller sstables and/or learn to live with your
giant sstable

Or add a new data directory with more space in it, to allow you to
compact. I mention the latter in case it is trivial to attach
additional storage in your env.

The other alternative is to wait. Most space will be reclaimed over
time by minor compaction.

=Rob

-- 
=Robert Coli
AIM&GTALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb