Compact and Repair

[Henrik Schröder <sk...@gmail.com>]

Hi,

We recently ran a major compaction across our cluster, which reduced the
storage used by about 50%. This is fine, since we do a lot of updates to
existing data, so that's the expected result.

The day after, we ran a full repair -pr across the cluster, and when that
finished, each storage node was at about the same size as before the major
compaction. Why does that happen? What gets transferred to other nodes, and
why does it suddenly take up a lot of space again?

We haven't run repair -pr regularly, so is this just something that happens
on the first weekly run, and can we expect a different result next week? Or
does repair always cause the data to grow on each node? To me it just
doesn't seem proportional?


/Henrik

Re: Compact and Repair

[Henrik Schröder <sk...@gmail.com>]

It was the long time since last repair that did it. We've scheduled regular
repairs now, and this time the repairs didn't increase the load very much.
So that was it! :-)


/Henrik


On Thu, Nov 8, 2012 at 7:20 PM, Andrey Ilinykh <ai...@gmail.com> wrote:

> Nothing unusual. When you run repair cassandra streams inconsistent
> regions from all replicas. If you have wide rows or didn't run repair
> regularly it is very easy to get 10-20% of extra data from each replica.
> What probably happens in your case. Theoretically cassandra should compact
> new sstables you get from other nodes. But, by default cassandra compacts
> sstables in the same size tier. Because of major compaction you ran before,
> you have one big sstable and a bunch of small. So, there is nothing to
> compact right now. Eventually cassandra will compact them. But nobody knows
> when it will happen. This is one of problems caused by major compaction.
> For maintenance it is better to have a set of small sstables then one
> big.
>
> Andrey
>
>
> On Thu, Nov 8, 2012 at 2:55 AM, Henrik Schröder <sk...@gmail.com> wrote:
>
>> Hi,
>>
>> We recently ran a major compaction across our cluster, which reduced the
>> storage used by about 50%. This is fine, since we do a lot of updates to
>> existing data, so that's the expected result.
>>
>> The day after, we ran a full repair -pr across the cluster, and when that
>> finished, each storage node was at about the same size as before the major
>> compaction. Why does that happen? What gets transferred to other nodes, and
>> why does it suddenly take up a lot of space again?
>>
>> We haven't run repair -pr regularly, so is this just something that
>> happens on the first weekly run, and can we expect a different result next
>> week? Or does repair always cause the data to grow on each node? To me it
>> just doesn't seem proportional?
>>
>>
>> /Henrik
>>
>
>

Re: Compact and Repair

[Andrey Ilinykh <ai...@gmail.com>]

Nothing unusual. When you run repair cassandra streams inconsistent regions
from all replicas. If you have wide rows or didn't run repair regularly it
is very easy to get 10-20% of extra data from each replica. What probably
happens in your case. Theoretically cassandra should compact new sstables
you get from other nodes. But, by default cassandra compacts sstables in
the same size tier. Because of major compaction you ran before, you have
one big sstable and a bunch of small. So, there is nothing to compact right
now. Eventually cassandra will compact them. But nobody knows when it will
happen. This is one of problems caused by major compaction. For maintenance
it is better to have a set of small sstables then one big.

Andrey


On Thu, Nov 8, 2012 at 2:55 AM, Henrik Schröder <sk...@gmail.com> wrote:

> Hi,
>
> We recently ran a major compaction across our cluster, which reduced the
> storage used by about 50%. This is fine, since we do a lot of updates to
> existing data, so that's the expected result.
>
> The day after, we ran a full repair -pr across the cluster, and when that
> finished, each storage node was at about the same size as before the major
> compaction. Why does that happen? What gets transferred to other nodes, and
> why does it suddenly take up a lot of space again?
>
> We haven't run repair -pr regularly, so is this just something that
> happens on the first weekly run, and can we expect a different result next
> week? Or does repair always cause the data to grow on each node? To me it
> just doesn't seem proportional?
>
>
> /Henrik
>

Re: Compact and Repair

[Henrik Schröder <sk...@gmail.com>]

No, we haven't changed RF, but it's been a very long time since we repaired
last, so we're guessing this is an effect of not running repair regularly,
and that doing it regularly will fix it. It would just be nice to know.

Also, running major compaction after the repair made the data size shrink
back to what it was before, soe clearly a lot of junk data was sent over on
that repair, most probably tombstones of some kind, as discussed in the
other thread.


/Henrik


On Thu, Nov 8, 2012 at 1:53 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:

> Did you change the RF or had a node down since you repaired last time ?
>
>
> 2012/11/8 Henrik Schröder <sk...@gmail.com>
>
>> No, we're not using columns with TTL, and I performed a major compaction
>> before the repair, so there shouldn't be vast amounts of tombstones moving
>> around.
>>
>> And the increase happened during the repair, the nodes gained ~20-30GB
>> each.
>>
>>
>> /Henrik
>>
>>
>>
>> On Thu, Nov 8, 2012 at 12:40 PM, horschi <ho...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> is it possible that your repair is overrepairing due to any of the
>>> issues discussed here:
>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/repair-compaction-and-tombstone-rows-td7583481.html?
>>>
>>>
>>> I've seen repair increasing the load on my cluster, but what you are
>>> describing sounds like a lot to me.
>>>
>>> Does this increase happen due to repair entirely? Or was the load maybe
>>> increasing gradually over the week and you just checked for the first time?
>>>
>>> cheers,
>>> Christian
>>>
>>>
>>>
>>> On Thu, Nov 8, 2012 at 11:55 AM, Henrik Schröder <sk...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> We recently ran a major compaction across our cluster, which reduced
>>>> the storage used by about 50%. This is fine, since we do a lot of updates
>>>> to existing data, so that's the expected result.
>>>>
>>>> The day after, we ran a full repair -pr across the cluster, and when
>>>> that finished, each storage node was at about the same size as before the
>>>> major compaction. Why does that happen? What gets transferred to other
>>>> nodes, and why does it suddenly take up a lot of space again?
>>>>
>>>> We haven't run repair -pr regularly, so is this just something that
>>>> happens on the first weekly run, and can we expect a different result next
>>>> week? Or does repair always cause the data to grow on each node? To me it
>>>> just doesn't seem proportional?
>>>>
>>>>
>>>> /Henrik
>>>>
>>>
>>>
>>
>

Re: Compact and Repair

[Alain RODRIGUEZ <ar...@gmail.com>]

Did you change the RF or had a node down since you repaired last time ?


2012/11/8 Henrik Schröder <sk...@gmail.com>

> No, we're not using columns with TTL, and I performed a major compaction
> before the repair, so there shouldn't be vast amounts of tombstones moving
> around.
>
> And the increase happened during the repair, the nodes gained ~20-30GB
> each.
>
>
> /Henrik
>
>
>
> On Thu, Nov 8, 2012 at 12:40 PM, horschi <ho...@gmail.com> wrote:
>
>> Hi,
>>
>> is it possible that your repair is overrepairing due to any of the issues
>> discussed here:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/repair-compaction-and-tombstone-rows-td7583481.html?
>>
>>
>> I've seen repair increasing the load on my cluster, but what you are
>> describing sounds like a lot to me.
>>
>> Does this increase happen due to repair entirely? Or was the load maybe
>> increasing gradually over the week and you just checked for the first time?
>>
>> cheers,
>> Christian
>>
>>
>>
>> On Thu, Nov 8, 2012 at 11:55 AM, Henrik Schröder <sk...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> We recently ran a major compaction across our cluster, which reduced the
>>> storage used by about 50%. This is fine, since we do a lot of updates to
>>> existing data, so that's the expected result.
>>>
>>> The day after, we ran a full repair -pr across the cluster, and when
>>> that finished, each storage node was at about the same size as before the
>>> major compaction. Why does that happen? What gets transferred to other
>>> nodes, and why does it suddenly take up a lot of space again?
>>>
>>> We haven't run repair -pr regularly, so is this just something that
>>> happens on the first weekly run, and can we expect a different result next
>>> week? Or does repair always cause the data to grow on each node? To me it
>>> just doesn't seem proportional?
>>>
>>>
>>> /Henrik
>>>
>>
>>
>

Re: Compact and Repair

[Henrik Schröder <sk...@gmail.com>]

No, we're not using columns with TTL, and I performed a major compaction
before the repair, so there shouldn't be vast amounts of tombstones moving
around.

And the increase happened during the repair, the nodes gained ~20-30GB each.


/Henrik


On Thu, Nov 8, 2012 at 12:40 PM, horschi <ho...@gmail.com> wrote:

> Hi,
>
> is it possible that your repair is overrepairing due to any of the issues
> discussed here:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/repair-compaction-and-tombstone-rows-td7583481.html?
>
>
> I've seen repair increasing the load on my cluster, but what you are
> describing sounds like a lot to me.
>
> Does this increase happen due to repair entirely? Or was the load maybe
> increasing gradually over the week and you just checked for the first time?
>
> cheers,
> Christian
>
>
>
> On Thu, Nov 8, 2012 at 11:55 AM, Henrik Schröder <sk...@gmail.com>wrote:
>
>> Hi,
>>
>> We recently ran a major compaction across our cluster, which reduced the
>> storage used by about 50%. This is fine, since we do a lot of updates to
>> existing data, so that's the expected result.
>>
>> The day after, we ran a full repair -pr across the cluster, and when that
>> finished, each storage node was at about the same size as before the major
>> compaction. Why does that happen? What gets transferred to other nodes, and
>> why does it suddenly take up a lot of space again?
>>
>> We haven't run repair -pr regularly, so is this just something that
>> happens on the first weekly run, and can we expect a different result next
>> week? Or does repair always cause the data to grow on each node? To me it
>> just doesn't seem proportional?
>>
>>
>> /Henrik
>>
>
>

Re: Compact and Repair

[horschi <ho...@gmail.com>]

Hi,

is it possible that your repair is overrepairing due to any of the issues
discussed here:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/repair-compaction-and-tombstone-rows-td7583481.html?


I've seen repair increasing the load on my cluster, but what you are
describing sounds like a lot to me.

Does this increase happen due to repair entirely? Or was the load maybe
increasing gradually over the week and you just checked for the first time?

cheers,
Christian


On Thu, Nov 8, 2012 at 11:55 AM, Henrik Schröder <sk...@gmail.com> wrote:

> Hi,
>
> We recently ran a major compaction across our cluster, which reduced the
> storage used by about 50%. This is fine, since we do a lot of updates to
> existing data, so that's the expected result.
>
> The day after, we ran a full repair -pr across the cluster, and when that
> finished, each storage node was at about the same size as before the major
> compaction. Why does that happen? What gets transferred to other nodes, and
> why does it suddenly take up a lot of space again?
>
> We haven't run repair -pr regularly, so is this just something that
> happens on the first weekly run, and can we expect a different result next
> week? Or does repair always cause the data to grow on each node? To me it
> just doesn't seem proportional?
>
>
> /Henrik
>