Incremental Repair

[Anuj Wadehra <an...@yahoo.co.in>]

Hi,

Our setup is as follows:
2 DCS with N nodes, RF=DC1:3,DC2:3, Hinted Handoff=3 hours, Incremental Repair scheduled once on every node (ALL DCs) within the gc grace period.

I have following queries regarding incremental repairs:

1. When a node is down for X hours (where x > hinted handoff hours and less than gc grace time), I think incremental repair is sufficient rather than doing the full repair. Is the understanding correct ? 

2. DataStax recommends "Run incremental repair daily, run full repairs weekly to monthly". Does that mean that I have to run full repairs every week to month EVEN IF I do daily incremental repairs? If yes, whats the reasoning of running full repair when inc repair is already run?

Reference: https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesWhen.html

3. We run inc repair at least once in gc grace instead of the general recommendation that inc repair should be run daily. Do you see any problem with the approach? 

As per my understanding, if we run inc repair less frequently, compaction between unrepaired and repaired data wont happen on a node until some node runs inc repair on the unrepaired data range. Thus, there can be some impact on disk space and read performance but immediate compaction is anyways never guaranteed by Cassandra. So, I see minimal impact on performance and that too just on the reads of delta data generated between repairs. 

Thanks
Anuj

Re: Incremental Repair

[Paulo Motta <pa...@gmail.com>]

> there are some nasty edge cases when you mix incremental repair and full
repair ( https://issues.apache.org/jira/browse/CASSANDRA-13153 )

mixing incremental and full repairs will just make that more likely to
happen, but although unlikely it's still possible for a similar condition
to happen even when using incremental repair alone and this is ultimately
fixed by CASSANDRA-9143 on 4.0 so I'd probably stick to Blake's suggestions
on #13153:

> It seems like we should recommend that users who delete data:
> 1. Stop using incremental repair (pre-4.0)
> 2. Run a full repair after upgrading to 4.0 before using incremental
repair again
> We should also recommend that even if users don't delete data, they
should take a look at the amount of streaming their incremental repair is
doing, and decide if it might be less expensive to just do full repairs
instead.

2017-03-13 1:15 GMT-03:00 Jeff Jirsa <jj...@apache.org>:

>
>
> On 2017-03-12 10:44 (-0700), Anuj Wadehra <an...@yahoo.co.in> wrote:
> > Hi,
> >
> > Our setup is as follows:
> > 2 DCS with N nodes, RF=DC1:3,DC2:3, Hinted Handoff=3 hours, Incremental
> Repair scheduled once on every node (ALL DCs) within the gc grace period.
> >
> > I have following queries regarding incremental repairs:
> >
> > 1. When a node is down for X hours (where x > hinted handoff hours and
> less than gc grace time), I think incremental repair is sufficient rather
> than doing the full repair. Is the understanding correct ?
> >
>
> Incremental repair SHOULD provide the same guarantees as regular repair.
>
> > 2. DataStax recommends "Run incremental repair daily, run full repairs
> weekly to monthly". Does that mean that I have to run full repairs every
> week to month EVEN IF I do daily incremental repairs? If yes, whats the
> reasoning of running full repair when inc repair is already run?
> >
> > Reference: https://docs.datastax.com/en/cassandra/3.0/cassandra/
> operations/opsRepairNodesWhen.html
> >
>
> I don't know why datastax suggests this, there are some nasty edge cases
> when you mix incremental repair and full repair (
> https://issues.apache.org/jira/browse/CASSANDRA-13153 )
>
> > 3. We run inc repair at least once in gc grace instead of the general
> recommendation that inc repair should be run daily. Do you see any problem
> with the approach?
> >
> >
>
> The more often you run it, the less data will be transferred, and the less
> painful it will be.  By running it weekly, you're making each run do 7x as
> much as work compared to running it daily, increasing the chance of having
> it impact your typical latencies.
>
>
>

Re: Incremental Repair

[Jeff Jirsa <jj...@apache.org>]

On 2017-03-12 10:44 (-0700), Anuj Wadehra <an...@yahoo.co.in> wrote: 
> Hi,
> 
> Our setup is as follows:
> 2 DCS with N nodes, RF=DC1:3,DC2:3, Hinted Handoff=3 hours, Incremental Repair scheduled once on every node (ALL DCs) within the gc grace period.
> 
> I have following queries regarding incremental repairs:
> 
> 1. When a node is down for X hours (where x > hinted handoff hours and less than gc grace time), I think incremental repair is sufficient rather than doing the full repair. Is the understanding correct ? 
> 

Incremental repair SHOULD provide the same guarantees as regular repair.

> 2. DataStax recommends "Run incremental repair daily, run full repairs weekly to monthly". Does that mean that I have to run full repairs every week to month EVEN IF I do daily incremental repairs? If yes, whats the reasoning of running full repair when inc repair is already run?
> 
> Reference: https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesWhen.html
> 

I don't know why datastax suggests this, there are some nasty edge cases when you mix incremental repair and full repair ( https://issues.apache.org/jira/browse/CASSANDRA-13153 ) 

> 3. We run inc repair at least once in gc grace instead of the general recommendation that inc repair should be run daily. Do you see any problem with the approach? 
> 
>

The more often you run it, the less data will be transferred, and the less painful it will be.  By running it weekly, you're making each run do 7x as much as work compared to running it daily, increasing the chance of having it impact your typical latencies.