You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Leena Ghatpande <lg...@hotmail.com> on 2018/04/19 13:19:12 UTC
Cassandra 3.7 - Problem with Repairs - all nodes failing
we have 8 node prod cluster running on cassandra 3.7. Our 2 largest tables have around 100M and 30M rows respectively while all others are relatively smaller.
we have been running repairs on alternate days on 2 of our keyspaces.
We run repair on each node in the cluster with the -pr option on every table within each keyspace individually. Repairs are run sequentially on each node
These were working fine, but with no change on the systems, they have started failing since last month.
The repairs have started failing for each table on every node with no specific error.
I have tried running scrub on every table and then running repair , but still the repair fails for all tables.
Our smallest table with only 100 rows also fails on repair.
But if I run the repair with DC option (-dc localdatacenter) for local datacenters, then the repairs are successfully. Is this indication that the repairs are good?
we would still want the repairs to work on individually tables as expected.
Need help trying to get the repairs to work properly as we have a big migration planned for june .
Upgrading cassandra is not an option right now.
Here are some of the errors
INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181 - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for clients from / IP
ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 Validator.java:261 - Failed creating a merkle tree for [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]], /130.5.123.234 (see log for details)
ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 - Exception in thread Thread[ValidationExecutor:213,1,main]
java.lang.NullPointerException: null
INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181 - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for clients from /IP
ERROR [Repair#113:12] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 - Exception in thread Thread[Repair#113:12,5,RMI Runtime]
com.google.common.util.concurrent.UncheckedExecutionException: org.apache.cassandra.exceptions.RepairException: [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]] Validation failed in /130.5.127.60
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]] Validation failed in /130.5.127.60
at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:183) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:439) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:169) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
答复: 答复: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing
Posted by Xiangfei Ni <xi...@cm-dt.com>.
Thank you very much,Nicolas.
Best Regards,
倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516
发件人: Nicolas Guyomar <ni...@gmail.com>
发送时间: 2018年4月20日 16:37
收件人: user@cassandra.apache.org
主题: Re: 答复: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing
Hi,
I believe there is no such official apache documentation, and since Datastax removed the old guide to redirect to the apache one, you are left with Google to look for guidance.
As a general advice (from what I can recall) :
- you might want to run nodetool cleanup before upgrading to free so space if you forget to do it after a node addition
- do not let your repair scheduler launch a repair while upgrading (wait for ongoing compaction to complete if possible)
- check the java version you are using, and update if needed
- upgrade one node at a time and wait for U/N state in nodetool status
- run upgradesstable on each node one at a time once every node is up and running with the new version (make sure you have some free space left on disk because that will rewrite existing sstable in-place)
If you can afford it, check the upgrade procedure on a test cluster as always ;)
- Some default configuration might be changed between version (java GC for instance maybe ? ) => compare your cassandra.yaml and jvm.options with the new one !
Cross check what I wrote with what you can find on the internet !
On 20 April 2018 at 10:21, Xiangfei Ni <xi...@cm-dt.com>> wrote:
Hi Nicolas,
Thanks for your reply.
The doc you sent to me only includes what have been upgrade between every version,but I want the doc of how to do the online upgrade,do you have this kind of doc?
Best Regards,
倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516
发件人: Nicolas Guyomar <ni...@gmail.com>>
发送时间: 2018年4月20日 15:44
收件人: user@cassandra.apache.org<ma...@cassandra.apache.org>
主题: Re: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing
Hi,
You can have a look to https://github.com/apache/cassandra/blob/trunk/NEWS.txt which list every modification / advice for upgrading between each C* version
On 20 April 2018 at 09:25, Xiangfei Ni <xi...@cm-dt.com>> wrote:
By the way,is there official documentation for online upgrade cassandra from 3.9 to 3.11.2 which I can follow?
Best Regards,
倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516
发件人: Anup Shirolkar <an...@instaclustr.com>>
发送时间: 2018年4月20日 11:34
收件人: user@cassandra.apache.org<ma...@cassandra.apache.org>
主题: Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
Contd.
Upgrading from 3.7 to 3.11.1 will not involving any major changes.
It can be achieved without any downtime and it should not impact on Cassandra clients.
You can test the upgrade on a test cluster to be sure if you are considering to upgrade prod.
Thanks,
Anup
On 20 April 2018 at 13:28, Anup Shirolkar <an...@instaclustr.com>> wrote:
Hi Leena,
The repairs are most likely failing because of some bug in Cassandra 3.7. I don't have a JIRA reference handy but there are quite some issues in this version.
Considering your scenario, it is highly recommended that you should upgrade to 3.11.1.
Although, you have mentioned that upgrading is not an option, I would like to tell you that
On 19 April 2018 at 23:19, Leena Ghatpande <lg...@hotmail.com>> wrote:
we have 8 node prod cluster running on cassandra 3.7. Our 2 largest tables have around 100M and 30M rows respectively while all others are relatively smaller.
we have been running repairs on alternate days on 2 of our keyspaces.
We run repair on each node in the cluster with the -pr option on every table within each keyspace individually. Repairs are run sequentially on each node
These were working fine, but with no change on the systems, they have started failing since last month.
The repairs have started failing for each table on every node with no specific error.
I have tried running scrub on every table and then running repair , but still the repair fails for all tables.
Our smallest table with only 100 rows also fails on repair.
But if I run the repair with DC option (-dc localdatacenter) for local datacenters, then the repairs are successfully. Is this indication that the repairs are good?
we would still want the repairs to work on individually tables as expected.
Need help trying to get the repairs to work properly as we have a big migration planned for june .
Upgrading cassandra is not an option right now.
Here are some of the errors
INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181 - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for clients from / IP
ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 Validator.java:261 - Failed creating a merkle tree for [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]], /130.5.123.234<http://130.5.123.234> (see log for details)
ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 - Exception in thread Thread[ValidationExecutor:213,1,main]
java.lang.NullPointerException: null
INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181 - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for clients from /IP
ERROR [Repair#113:12] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 - Exception in thread Thread[Repair#113:12,5,RMI Runtime]
com.google.common.util.concurrent.UncheckedExecutionException: org.apache.cassandra.exceptions.RepairException: [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]] Validation failed in /130.5.127.60<http://130.5.127.60>
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
at org.apache.cassandra.repair.Re<http://org.apache.cassandra.repair.Re>pairJob.run(RepairJob.java:160) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]] Validation failed in /130.5.127.60<http://130.5.127.60>
at org.apache.cassandra.repair.Va<http://org.apache.cassandra.repair.Va>lidationTask.treesReceived(ValidationTask.java:68) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.repair.Re<http://org.apache.cassandra.repair.Re>pairSession.validationComplete(RepairSession.java:183) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:439) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.repair.Re<http://org.apache.cassandra.repair.Re>pairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:169) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.net<http://org.apache.cassandra.net>.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
Re: 答复: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing
Posted by Nicolas Guyomar <ni...@gmail.com>.
Hi,
I believe there is no such official apache documentation, and since
Datastax removed the old guide to redirect to the apache one, you are left
with Google to look for guidance.
As a general advice (from what I can recall) :
- you might want to run nodetool cleanup before upgrading to free so space
if you forget to do it after a node addition
- do not let your repair scheduler launch a repair while upgrading (wait
for ongoing compaction to complete if possible)
- check the java version you are using, and update if needed
- upgrade one node at a time and wait for U/N state in nodetool status
- run upgradesstable on each node one at a time once every node is up and
running with the new version (make sure you have some free space left on
disk because that will rewrite existing sstable in-place)
If you can afford it, check the upgrade procedure on a test cluster as
always ;)
- Some default configuration might be changed between version (java GC for
instance maybe ? ) => compare your cassandra.yaml and jvm.options with the
new one !
Cross check what I wrote with what you can find on the internet !
On 20 April 2018 at 10:21, Xiangfei Ni <xi...@cm-dt.com> wrote:
> Hi Nicolas,
>
> Thanks for your reply.
>
> The doc you sent to me only includes what have been upgrade between every
> version,but I want the doc of how to do the online upgrade,do you have this
> kind of doc?
>
>
>
> Best Regards,
>
>
>
> 倪项菲*/ **David Ni*
>
> 中移德电网络科技有限公司
>
> Virtue Intelligent Network Ltd, co.
>
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>
> Mob: +86 13797007811|Tel: + 86 27 5024 2516
>
>
>
> *发件人:* Nicolas Guyomar <ni...@gmail.com>
> *发送时间:* 2018年4月20日 15:44
> *收件人:* user@cassandra.apache.org
> *主题:* Re: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing
>
>
>
> Hi,
>
>
>
> You can have a look to https://github.com/apache/
> cassandra/blob/trunk/NEWS.txt which list every modification / advice for
> upgrading between each C* version
>
>
>
> On 20 April 2018 at 09:25, Xiangfei Ni <xi...@cm-dt.com> wrote:
>
> By the way,is there official documentation for online upgrade cassandra
> from 3.9 to 3.11.2 which I can follow?
>
>
>
> Best Regards,
>
>
>
> 倪项菲*/ **David Ni*
>
> 中移德电网络科技有限公司
>
> Virtue Intelligent Network Ltd, co.
>
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>
> Mob: +86 13797007811|Tel: + 86 27 5024 2516
>
>
>
> *发件人:* Anup Shirolkar <an...@instaclustr.com>
> *发送时间:* 2018年4月20日 11:34
> *收件人:* user@cassandra.apache.org
> *主题:* Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
>
>
>
> Contd.
>
>
>
> Upgrading from 3.7 to 3.11.1 will not involving any major changes.
>
> It can be achieved without any downtime and it should not impact on
> Cassandra clients.
>
> You can test the upgrade on a test cluster to be sure if you are
> considering to upgrade prod.
>
>
>
> Thanks,
>
> Anup
>
>
>
> On 20 April 2018 at 13:28, Anup Shirolkar <an...@instaclustr.com>
> wrote:
>
> Hi Leena,
>
>
>
> The repairs are most likely failing because of some bug in Cassandra 3.7.
> I don't have a JIRA reference handy but there are quite some issues in this
> version.
>
>
>
> Considering your scenario, it is highly recommended that you should
> upgrade to 3.11.1.
>
> Although, you have mentioned that upgrading is not an option, I would like
> to tell you that
>
>
>
> On 19 April 2018 at 23:19, Leena Ghatpande <lg...@hotmail.com> wrote:
>
> we have 8 node prod cluster running on cassandra 3.7. Our 2 largest tables
> have around 100M and 30M rows respectively while all others are relatively
> smaller.
>
> we have been running repairs on alternate days on 2 of our keyspaces.
> We run repair on each node in the cluster with the -pr option on every
> table within each keyspace individually. Repairs are run sequentially on
> each node
> These were working fine, but with no change on the systems, they have
> started failing since last month.
>
> The repairs have started failing for each table on every node with no
> specific error.
>
> I have tried running scrub on every table and then running repair , but
> still the repair fails for all tables.
>
> Our smallest table with only 100 rows also fails on repair.
>
> But if I run the repair with DC option (-dc localdatacenter) for local
> datacenters, then the repairs are successfully. Is this indication that the
> repairs are good?
> we would still want the repairs to work on individually tables as expected.
>
> Need help trying to get the repairs to work properly as we have a big
> migration planned for june .
>
> Upgrading cassandra is not an option right now.
>
>
> Here are some of the errors
> INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181
> - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for
> clients from / IP
> ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 Validator.java:261
> - Failed creating a merkle tree for [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d
> on secure/clients, [(1849652111528073119,1856811324137977760],
> (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491],
> (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448],
> (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373],
> (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060],
> (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501],
> (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119],
> (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404],
> (7754917992280096132,7754953485457609099]]], /130.5.123.234 (see log for
> details)
> ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461
> CassandraDaemon.java:217 - Exception in thread
> Thread[ValidationExecutor:213,1,main]
> java.lang.NullPointerException: null
> INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181
> - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for
> clients from /IP
> ERROR [Repair#113:12] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 -
> Exception in thread Thread[Repair#113:12,5,RMI Runtime]
> com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.cassandra.exceptions.RepairException: [repair
> #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients,
> [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952],
> (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471],
> (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754],
> (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231],
> (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804],
> (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890],
> (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370],
> (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]]
> Validation failed in /130.5.127.60
> at com.google.common.util.concurrent.Futures.
> wrapAndThrowUnchecked(Futures.java:1525) ~[guava-18.0.jar:na]
> at com.google.common.util.concurrent.Futures.
> getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
> at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160)
> ~[apache-cassandra-3.7.jar:3.7]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
> #223c73c2-4372-11e8-8749-89fc1dde5b7d on clients, [(1849652111528073119,1856811324137977760],
> (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491],
> (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448],
> (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373],
> (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060],
> (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501],
> (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119],
> (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404],
> (7754917992280096132,7754953485457609099]]] Validation failed in /
> 130.5.127.60
> at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:183)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:439)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
> RepairMessageVerbHandler.java:169) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.7.jar:3.7]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
>
>
>
>
>
>
>
>
>
>
答复: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing
Posted by Xiangfei Ni <xi...@cm-dt.com>.
Hi Nicolas,
Thanks for your reply.
The doc you sent to me only includes what have been upgrade between every version,but I want the doc of how to do the online upgrade,do you have this kind of doc?
Best Regards,
倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516
发件人: Nicolas Guyomar <ni...@gmail.com>
发送时间: 2018年4月20日 15:44
收件人: user@cassandra.apache.org
主题: Re: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing
Hi,
You can have a look to https://github.com/apache/cassandra/blob/trunk/NEWS.txt which list every modification / advice for upgrading between each C* version
On 20 April 2018 at 09:25, Xiangfei Ni <xi...@cm-dt.com>> wrote:
By the way,is there official documentation for online upgrade cassandra from 3.9 to 3.11.2 which I can follow?
Best Regards,
倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516
发件人: Anup Shirolkar <an...@instaclustr.com>>
发送时间: 2018年4月20日 11:34
收件人: user@cassandra.apache.org<ma...@cassandra.apache.org>
主题: Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
Contd.
Upgrading from 3.7 to 3.11.1 will not involving any major changes.
It can be achieved without any downtime and it should not impact on Cassandra clients.
You can test the upgrade on a test cluster to be sure if you are considering to upgrade prod.
Thanks,
Anup
On 20 April 2018 at 13:28, Anup Shirolkar <an...@instaclustr.com>> wrote:
Hi Leena,
The repairs are most likely failing because of some bug in Cassandra 3.7. I don't have a JIRA reference handy but there are quite some issues in this version.
Considering your scenario, it is highly recommended that you should upgrade to 3.11.1.
Although, you have mentioned that upgrading is not an option, I would like to tell you that
On 19 April 2018 at 23:19, Leena Ghatpande <lg...@hotmail.com>> wrote:
we have 8 node prod cluster running on cassandra 3.7. Our 2 largest tables have around 100M and 30M rows respectively while all others are relatively smaller.
we have been running repairs on alternate days on 2 of our keyspaces.
We run repair on each node in the cluster with the -pr option on every table within each keyspace individually. Repairs are run sequentially on each node
These were working fine, but with no change on the systems, they have started failing since last month.
The repairs have started failing for each table on every node with no specific error.
I have tried running scrub on every table and then running repair , but still the repair fails for all tables.
Our smallest table with only 100 rows also fails on repair.
But if I run the repair with DC option (-dc localdatacenter) for local datacenters, then the repairs are successfully. Is this indication that the repairs are good?
we would still want the repairs to work on individually tables as expected.
Need help trying to get the repairs to work properly as we have a big migration planned for june .
Upgrading cassandra is not an option right now.
Here are some of the errors
INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181 - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for clients from / IP
ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 Validator.java:261 - Failed creating a merkle tree for [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]], /130.5.123.234<http://130.5.123.234> (see log for details)
ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 - Exception in thread Thread[ValidationExecutor:213,1,main]
java.lang.NullPointerException: null
INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181 - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for clients from /IP
ERROR [Repair#113:12] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 - Exception in thread Thread[Repair#113:12,5,RMI Runtime]
com.google.common.util.concurrent.UncheckedExecutionException: org.apache.cassandra.exceptions.RepairException: [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]] Validation failed in /130.5.127.60<http://130.5.127.60>
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
at org.apache.cassandra.repair.Re<http://org.apache.cassandra.repair.Re>pairJob.run(RepairJob.java:160) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]] Validation failed in /130.5.127.60<http://130.5.127.60>
at org.apache.cassandra.repair.Va<http://org.apache.cassandra.repair.Va>lidationTask.treesReceived(ValidationTask.java:68) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.repair.Re<http://org.apache.cassandra.repair.Re>pairSession.validationComplete(RepairSession.java:183) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:439) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.repair.Re<http://org.apache.cassandra.repair.Re>pairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:169) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.net<http://org.apache.cassandra.net>.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
Re: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing
Posted by Nicolas Guyomar <ni...@gmail.com>.
Hi,
You can have a look to
https://github.com/apache/cassandra/blob/trunk/NEWS.txt which list every
modification / advice for upgrading between each C* version
On 20 April 2018 at 09:25, Xiangfei Ni <xi...@cm-dt.com> wrote:
> By the way,is there official documentation for online upgrade cassandra
> from 3.9 to 3.11.2 which I can follow?
>
>
>
> Best Regards,
>
>
>
> 倪项菲*/ **David Ni*
>
> 中移德电网络科技有限公司
>
> Virtue Intelligent Network Ltd, co.
>
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>
> Mob: +86 13797007811|Tel: + 86 27 5024 2516
>
>
>
> *发件人:* Anup Shirolkar <an...@instaclustr.com>
> *发送时间:* 2018年4月20日 11:34
> *收件人:* user@cassandra.apache.org
> *主题:* Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
>
>
>
> Contd.
>
>
>
> Upgrading from 3.7 to 3.11.1 will not involving any major changes.
>
> It can be achieved without any downtime and it should not impact on
> Cassandra clients.
>
> You can test the upgrade on a test cluster to be sure if you are
> considering to upgrade prod.
>
>
>
> Thanks,
>
> Anup
>
>
>
> On 20 April 2018 at 13:28, Anup Shirolkar <an...@instaclustr.com>
> wrote:
>
> Hi Leena,
>
>
>
> The repairs are most likely failing because of some bug in Cassandra 3.7.
> I don't have a JIRA reference handy but there are quite some issues in this
> version.
>
>
>
> Considering your scenario, it is highly recommended that you should
> upgrade to 3.11.1.
>
> Although, you have mentioned that upgrading is not an option, I would like
> to tell you that
>
>
>
> On 19 April 2018 at 23:19, Leena Ghatpande <lg...@hotmail.com> wrote:
>
> we have 8 node prod cluster running on cassandra 3.7. Our 2 largest tables
> have around 100M and 30M rows respectively while all others are relatively
> smaller.
>
> we have been running repairs on alternate days on 2 of our keyspaces.
> We run repair on each node in the cluster with the -pr option on every
> table within each keyspace individually. Repairs are run sequentially on
> each node
> These were working fine, but with no change on the systems, they have
> started failing since last month.
>
> The repairs have started failing for each table on every node with no
> specific error.
>
> I have tried running scrub on every table and then running repair , but
> still the repair fails for all tables.
>
> Our smallest table with only 100 rows also fails on repair.
>
> But if I run the repair with DC option (-dc localdatacenter) for local
> datacenters, then the repairs are successfully. Is this indication that the
> repairs are good?
> we would still want the repairs to work on individually tables as expected.
>
> Need help trying to get the repairs to work properly as we have a big
> migration planned for june .
>
> Upgrading cassandra is not an option right now.
>
>
> Here are some of the errors
> INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181
> - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for
> clients from / IP
> ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 Validator.java:261
> - Failed creating a merkle tree for [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d
> on secure/clients, [(1849652111528073119,1856811324137977760],
> (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491],
> (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448],
> (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373],
> (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060],
> (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501],
> (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119],
> (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404],
> (7754917992280096132,7754953485457609099]]], /130.5.123.234 (see log for
> details)
> ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461
> CassandraDaemon.java:217 - Exception in thread
> Thread[ValidationExecutor:213,1,main]
> java.lang.NullPointerException: null
> INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181
> - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for
> clients from /IP
> ERROR [Repair#113:12] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 -
> Exception in thread Thread[Repair#113:12,5,RMI Runtime]
> com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.cassandra.exceptions.RepairException: [repair
> #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients,
> [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952],
> (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471],
> (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754],
> (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231],
> (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804],
> (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890],
> (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370],
> (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]]
> Validation failed in /130.5.127.60
> at com.google.common.util.concurrent.Futures.
> wrapAndThrowUnchecked(Futures.java:1525) ~[guava-18.0.jar:na]
> at com.google.common.util.concurrent.Futures.
> getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
> at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160)
> ~[apache-cassandra-3.7.jar:3.7]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
> #223c73c2-4372-11e8-8749-89fc1dde5b7d on clients, [(1849652111528073119,1856811324137977760],
> (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491],
> (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448],
> (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373],
> (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060],
> (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501],
> (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119],
> (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404],
> (7754917992280096132,7754953485457609099]]] Validation failed in /
> 130.5.127.60
> at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:183)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:439)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
> RepairMessageVerbHandler.java:169) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.7.jar:3.7]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
>
>
>
>
>
>
>
>
>
答复: Cassandra 3.7 - Problem with Repairs - all nodes failing
Posted by Xiangfei Ni <xi...@cm-dt.com>.
By the way,is there official documentation for online upgrade cassandra from 3.9 to 3.11.2 which I can follow?
Best Regards,
倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516
发件人: Anup Shirolkar <an...@instaclustr.com>
发送时间: 2018年4月20日 11:34
收件人: user@cassandra.apache.org
主题: Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
Contd.
Upgrading from 3.7 to 3.11.1 will not involving any major changes.
It can be achieved without any downtime and it should not impact on Cassandra clients.
You can test the upgrade on a test cluster to be sure if you are considering to upgrade prod.
Thanks,
Anup
On 20 April 2018 at 13:28, Anup Shirolkar <an...@instaclustr.com>> wrote:
Hi Leena,
The repairs are most likely failing because of some bug in Cassandra 3.7. I don't have a JIRA reference handy but there are quite some issues in this version.
Considering your scenario, it is highly recommended that you should upgrade to 3.11.1.
Although, you have mentioned that upgrading is not an option, I would like to tell you that
On 19 April 2018 at 23:19, Leena Ghatpande <lg...@hotmail.com>> wrote:
we have 8 node prod cluster running on cassandra 3.7. Our 2 largest tables have around 100M and 30M rows respectively while all others are relatively smaller.
we have been running repairs on alternate days on 2 of our keyspaces.
We run repair on each node in the cluster with the -pr option on every table within each keyspace individually. Repairs are run sequentially on each node
These were working fine, but with no change on the systems, they have started failing since last month.
The repairs have started failing for each table on every node with no specific error.
I have tried running scrub on every table and then running repair , but still the repair fails for all tables.
Our smallest table with only 100 rows also fails on repair.
But if I run the repair with DC option (-dc localdatacenter) for local datacenters, then the repairs are successfully. Is this indication that the repairs are good?
we would still want the repairs to work on individually tables as expected.
Need help trying to get the repairs to work properly as we have a big migration planned for june .
Upgrading cassandra is not an option right now.
Here are some of the errors
INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181 - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for clients from / IP
ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 Validator.java:261 - Failed creating a merkle tree for [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]], /130.5.123.234<http://130.5.123.234> (see log for details)
ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 - Exception in thread Thread[ValidationExecutor:213,1,main]
java.lang.NullPointerException: null
INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181 - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for clients from /IP
ERROR [Repair#113:12] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 - Exception in thread Thread[Repair#113:12,5,RMI Runtime]
com.google.common.util.concurrent.UncheckedExecutionException: org.apache.cassandra.exceptions.RepairException: [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]] Validation failed in /130.5.127.60<http://130.5.127.60>
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
at org.apache.cassandra.repair.Re<http://org.apache.cassandra.repair.Re>pairJob.run(RepairJob.java:160) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]] Validation failed in /130.5.127.60<http://130.5.127.60>
at org.apache.cassandra.repair.Va<http://org.apache.cassandra.repair.Va>lidationTask.treesReceived(ValidationTask.java:68) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.repair.Re<http://org.apache.cassandra.repair.Re>pairSession.validationComplete(RepairSession.java:183) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:439) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.repair.Re<http://org.apache.cassandra.repair.Re>pairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:169) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.net<http://org.apache.cassandra.net>.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
Posted by Anup Shirolkar <an...@instaclustr.com>.
Contd.
Upgrading from 3.7 to 3.11.1 will not involving any major changes.
It can be achieved without any downtime and it should not impact on
Cassandra clients.
You can test the upgrade on a test cluster to be sure if you are
considering to upgrade prod.
Thanks,
Anup
On 20 April 2018 at 13:28, Anup Shirolkar <an...@instaclustr.com>
wrote:
> Hi Leena,
>
> The repairs are most likely failing because of some bug in Cassandra 3.7.
> I don't have a JIRA reference handy but there are quite some issues in this
> version.
>
> Considering your scenario, it is highly recommended that you should
> upgrade to 3.11.1.
> Although, you have mentioned that upgrading is not an option, I would like
> to tell you that
>
> On 19 April 2018 at 23:19, Leena Ghatpande <lg...@hotmail.com> wrote:
>
>> we have 8 node prod cluster running on cassandra 3.7. Our 2 largest
>> tables have around 100M and 30M rows respectively while all others are
>> relatively smaller.
>>
>> we have been running repairs on alternate days on 2 of our keyspaces.
>> We run repair on each node in the cluster with the -pr option on every
>> table within each keyspace individually. Repairs are run sequentially on
>> each node
>> These were working fine, but with no change on the systems, they have
>> started failing since last month.
>>
>> The repairs have started failing for each table on every node with no
>> specific error.
>>
>> I have tried running scrub on every table and then running repair , but
>> still the repair fails for all tables.
>>
>> Our smallest table with only 100 rows also fails on repair.
>>
>> But if I run the repair with DC option (-dc localdatacenter) for local
>> datacenters, then the repairs are successfully. Is this indication that the
>> repairs are good?
>> we would still want the repairs to work on individually tables as
>> expected.
>>
>> Need help trying to get the repairs to work properly as we have a big
>> migration planned for june .
>>
>> Upgrading cassandra is not an option right now.
>>
>>
>> Here are some of the errors
>> INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181
>> - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree
>> for clients from / IP
>> ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 Validator.java:261
>> - Failed creating a merkle tree for [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d
>> on secure/clients, [(1849652111528073119,1856811324137977760],
>> (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491],
>> (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448],
>> (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373],
>> (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060],
>> (-4363807597425543488,-4361416517953194804],
>> (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890],
>> (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370],
>> (-5791935107311742541,-5781988493712029404],
>> (7754917992280096132,7754953485457609099]]], /130.5.123.234 (see log for
>> details)
>> ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461
>> CassandraDaemon.java:217 - Exception in thread
>> Thread[ValidationExecutor:213,1,main]
>> java.lang.NullPointerException: null
>> INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181
>> - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree
>> for clients from /IP
>> ERROR [Repair#113:12] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 -
>> Exception in thread Thread[Repair#113:12,5,RMI Runtime]
>> com.google.common.util.concurrent.UncheckedExecutionException:
>> org.apache.cassandra.exceptions.RepairException: [repair
>> #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients,
>> [(1849652111528073119,1856811324137977760],
>> (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491],
>> (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448],
>> (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373],
>> (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060],
>> (-4363807597425543488,-4361416517953194804],
>> (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890],
>> (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370],
>> (-5791935107311742541,-5781988493712029404],
>> (7754917992280096132,7754953485457609099]]] Validation failed in /
>> 130.5.127.60
>> at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525)
>> ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511)
>> ~[guava-18.0.jar:na]
>> at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160)
>> ~[apache-cassandra-3.7.jar:3.7]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[na:1.8.0_45]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> ~[na:1.8.0_45]
>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
>> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
>> #223c73c2-4372-11e8-8749-89fc1dde5b7d on clients,
>> [(1849652111528073119,1856811324137977760],
>> (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491],
>> (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448],
>> (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373],
>> (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060],
>> (-4363807597425543488,-4361416517953194804],
>> (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890],
>> (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370],
>> (-5791935107311742541,-5781988493712029404],
>> (7754917992280096132,7754953485457609099]]] Validation failed in /
>> 130.5.127.60
>> at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
>> ~[apache-cassandra-3.7.jar:3.7]
>> at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:183)
>> ~[apache-cassandra-3.7.jar:3.7]
>> at org.apache.cassandra.service.ActiveRepairService.handleMessa
>> ge(ActiveRepairService.java:439) ~[apache-cassandra-3.7.jar:3.7]
>> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
>> RepairMessageVerbHandler.java:169) ~[apache-cassandra-3.7.jar:3.7]
>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
>> ~[apache-cassandra-3.7.jar:3.7]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[na:1.8.0_45]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[na:1.8.0_45]
>>
>>
>>
>>
>>
>>
>
Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
Posted by Leena Ghatpande <lg...@hotmail.com>.
We dont have enough time before our major infrastructure upgrade to do the cassandra upgrade.
is there any insight on the below
if I run the repair with DC option (-dc localdatacenter) for local
datacenters, then all repairs are successfully. Is this indication that the
repairs are good? can we proceed with with adding new nodes and
decomissioning nodes even when individual repairs fail but the DC repairs work
Is there anything else other than scrub that can be performed to fix the repair issues?
Thanks
________________________________
From: Anup Shirolkar <an...@instaclustr.com>
Sent: Thursday, April 19, 2018 11:28 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
Hi Leena,
The repairs are most likely failing because of some bug in Cassandra 3.7. I don't have a JIRA reference handy but there are quite some issues in this version.
Considering your scenario, it is highly recommended that you should upgrade to 3.11.1.
Although, you have mentioned that upgrading is not an option, I would like to tell you that
On 19 April 2018 at 23:19, Leena Ghatpande <lg...@hotmail.com>> wrote:
we have 8 node prod cluster running on cassandra 3.7. Our 2 largest tables have around 100M and 30M rows respectively while all others are relatively smaller.
we have been running repairs on alternate days on 2 of our keyspaces.
We run repair on each node in the cluster with the -pr option on every table within each keyspace individually. Repairs are run sequentially on each node
These were working fine, but with no change on the systems, they have started failing since last month.
The repairs have started failing for each table on every node with no specific error.
I have tried running scrub on every table and then running repair , but still the repair fails for all tables.
Our smallest table with only 100 rows also fails on repair.
But if I run the repair with DC option (-dc localdatacenter) for local datacenters, then the repairs are successfully. Is this indication that the repairs are good?
we would still want the repairs to work on individually tables as expected.
Need help trying to get the repairs to work properly as we have a big migration planned for june .
Upgrading cassandra is not an option right now.
Here are some of the errors
INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181 - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for clients from / IP
ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 Validator.java:261 - Failed creating a merkle tree for [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]], /130.5.123.234<http://130.5.123.234> (see log for details)
ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 - Exception in thread Thread[ValidationExecutor:213,1,main]
java.lang.NullPointerException: null
INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181 - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for clients from /IP
ERROR [Repair#113:12] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 - Exception in thread Thread[Repair#113:12,5,RMI Runtime]
com.google.common.util.concurrent.UncheckedExecutionException: org.apache.cassandra.exceptions.RepairException: [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]] Validation failed in /130.5.127.60<http://130.5.127.60>
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d on clients, [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]] Validation failed in /130.5.127.60<http://130.5.127.60>
at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:183) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:439) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:169) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.net<http://org.apache.cassandra.net>.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
Posted by Anup Shirolkar <an...@instaclustr.com>.
Hi Leena,
The repairs are most likely failing because of some bug in Cassandra 3.7. I
don't have a JIRA reference handy but there are quite some issues in this
version.
Considering your scenario, it is highly recommended that you should upgrade
to 3.11.1.
Although, you have mentioned that upgrading is not an option, I would like
to tell you that
On 19 April 2018 at 23:19, Leena Ghatpande <lg...@hotmail.com> wrote:
> we have 8 node prod cluster running on cassandra 3.7. Our 2 largest tables
> have around 100M and 30M rows respectively while all others are relatively
> smaller.
>
> we have been running repairs on alternate days on 2 of our keyspaces.
> We run repair on each node in the cluster with the -pr option on every
> table within each keyspace individually. Repairs are run sequentially on
> each node
> These were working fine, but with no change on the systems, they have
> started failing since last month.
>
> The repairs have started failing for each table on every node with no
> specific error.
>
> I have tried running scrub on every table and then running repair , but
> still the repair fails for all tables.
>
> Our smallest table with only 100 rows also fails on repair.
>
> But if I run the repair with DC option (-dc localdatacenter) for local
> datacenters, then the repairs are successfully. Is this indication that the
> repairs are good?
> we would still want the repairs to work on individually tables as expected.
>
> Need help trying to get the repairs to work properly as we have a big
> migration planned for june .
>
> Upgrading cassandra is not an option right now.
>
>
> Here are some of the errors
> INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181
> - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for
> clients from / IP
> ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 Validator.java:261
> - Failed creating a merkle tree for [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d
> on secure/clients, [(1849652111528073119,1856811324137977760],
> (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491],
> (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448],
> (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373],
> (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060],
> (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501],
> (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119],
> (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404],
> (7754917992280096132,7754953485457609099]]], /130.5.123.234 (see log for
> details)
> ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461
> CassandraDaemon.java:217 - Exception in thread
> Thread[ValidationExecutor:213,1,main]
> java.lang.NullPointerException: null
> INFO [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181
> - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for
> clients from /IP
> ERROR [Repair#113:12] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 -
> Exception in thread Thread[Repair#113:12,5,RMI Runtime]
> com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.cassandra.exceptions.RepairException: [repair
> #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients,
> [(1849652111528073119,1856811324137977760], (3733211856223440695,3737790228588239952],
> (-2500456349659149537,-2498953852677197491], (1735271399836012489,1735412813423041471],
> (1871725370007007817,1890457592856328448], (4316163881057906640,4323247409810431754],
> (4286141602946572160,4308169130179803373], (5189663040558066167,5193871822490506231],
> (7160723554094225326,7161133449395023060], (-4363807597425543488,-4361416517953194804],
> (7008956720664744733,7022523551326267501], (-5742986989228874052,-5734436401879059890],
> (1828335330499002859,1849652111528073119], (7072368932695202361,7144087505892848370],
> (-5791935107311742541,-5781988493712029404], (7754917992280096132,7754953485457609099]]]
> Validation failed in /130.5.127.60
> at com.google.common.util.concurrent.Futures.
> wrapAndThrowUnchecked(Futures.java:1525) ~[guava-18.0.jar:na]
> at com.google.common.util.concurrent.Futures.
> getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
> at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160)
> ~[apache-cassandra-3.7.jar:3.7]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
> #223c73c2-4372-11e8-8749-89fc1dde5b7d on clients, [(1849652111528073119,1856811324137977760],
> (3733211856223440695,3737790228588239952], (-2500456349659149537,-2498953852677197491],
> (1735271399836012489,1735412813423041471], (1871725370007007817,1890457592856328448],
> (4316163881057906640,4323247409810431754], (4286141602946572160,4308169130179803373],
> (5189663040558066167,5193871822490506231], (7160723554094225326,7161133449395023060],
> (-4363807597425543488,-4361416517953194804], (7008956720664744733,7022523551326267501],
> (-5742986989228874052,-5734436401879059890], (1828335330499002859,1849652111528073119],
> (7072368932695202361,7144087505892848370], (-5791935107311742541,-5781988493712029404],
> (7754917992280096132,7754953485457609099]]] Validation failed in /
> 130.5.127.60
> at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:183)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:439)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
> RepairMessageVerbHandler.java:169) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.7.jar:3.7]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
>
>
>
>
>
>