Cassandra 3.6 Repair issue with Reaper

[Abhishek Aggarwal <ab...@snapdeal.com>]

Hi All,

we tried sequential repair on very small table having only 20 Rows using
the reaper tool. But the repair got stuck while generating the snapshot.


Same when we tried with Parallel repair then run was working fine in the
begining for few segments but later it got stuck in the compaction and
never got completed and one of the node was shown down to other nodes due
to gossip issue and we had to do rolling restart of all the nodes.


In both the cases 2 nodes out of 6 nodes are getting stuck either in
compaction or in generating snapshot.


Abhishek Aggarwal

*Senior Software Engineer*
*M*: +91 8861212073 , 8588840304
*T*: 0124 6600600 *EXT*: 12128
ASF Center -A, ASF Center Udyog Vihar Phase IV,
Download Our App
[image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image:
A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image:
W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>

Re: Cassandra 3.6 Repair issue with Reaper

[Alexander Dejanovski <al...@thelastpickle.com>]

Hi Abhishek,

Can you check if you're getting the same behavior on this cluster using
nodetool commands to start repair ? (don't forget to add --full in order to
make sure you're not running incremental repair, if that's indeed what
you're doing with reaper).
Could you also open an issue on github for this, with the useful logs you
can get from your Cassandra nodes ?
https://github.com/thelastpickle/cassandra-reaper/issues

Thanks,

On Tue, Nov 15, 2016 at 7:00 AM Abhishek Aggarwal <
abhishek.aggarwal01@snapdeal.com> wrote:

> Hi All,
>
> we tried sequential repair on very small table having only 20 Rows using
> the reaper tool. But the repair got stuck while generating the snapshot.
>
>
> Same when we tried with Parallel repair then run was working fine in the
> begining for few segments but later it got stuck in the compaction and
> never got completed and one of the node was shown down to other nodes due
> to gossip issue and we had to do rolling restart of all the nodes.
>
>
> In both the cases 2 nodes out of 6 nodes are getting stuck either in
> compaction or in generating snapshot.
>
>
> Abhishek Aggarwal
>
> *Senior Software Engineer*
> *M*: +91 8861212073 <+91%2088612%2012073> , 8588840304
> *T*: 0124 6600600 *EXT*: 12128
> ASF Center -A, ASF Center Udyog Vihar Phase IV,
> Download Our App
> [image: A]
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android> [image:
> A]
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios> [image:
> W]
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
-- 
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com