You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Meg Mara <mm...@digitalriver.com> on 2017/07/27 18:50:50 UTC
Error Exception in Repair Thread
Hello Cassandra Experts!
I have seen the following errors in the system log when a scheduled nodetool repair operation runs on the cluster. Cassandra Version 3.0.10. Any thoughts or suggestions are welcome!
ERROR [Repair#3794:3] 2017-07-27 17:47:22,000 CassandraDaemon.java:207 - Exception in thread Thread[Repair#3794:3,5,RMI Runtime]
java.lang.AssertionError: java.lang.InterruptedException
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:265) ~[apache-cassandra-3.0.10.jar:3.0.10]
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:225) ~[apache-cassandra-3.0.10.jar:3.0.10]
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:196) ~[apache-cassandra-3.0.10.jar:3.0.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150) ~[na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
Caused by: java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) ~[na:1.8.0_101]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:261) ~[apache-cassandra-3.0.10.jar:3.0.10]
... 5 common frames omitted
Thank you,
Meg Mara
Re: Error Exception in Repair Thread
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi,
java.lang.InterruptedException
>
I am not sure here, but just a silly thought: did you restart a node during
this repair process? A restarting node could probably create an
interruption on a streaming process.
C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2017-07-27 19:50 GMT+01:00 Meg Mara <mm...@digitalriver.com>:
> Hello Cassandra Experts!
>
>
>
> I have seen the following errors in the system log when a scheduled
> nodetool repair operation runs on the cluster. Cassandra Version 3.0.10.
> Any thoughts or suggestions are welcome!
>
>
>
> ERROR [Repair#3794:3] 2017-07-27 17:47:22,000 CassandraDaemon.java:207 -
> Exception in thread Thread[Repair#3794:3,5,RMI Runtime]
>
> java.lang.AssertionError: java.lang.InterruptedException
>
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.
> extractThrowable(DebuggableThreadPoolExecutor.java:265)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
>
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.
> logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:225)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
>
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.
> afterExecute(DebuggableThreadPoolExecutor.java:196)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
> ~[na:1.8.0_101]
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_101]
>
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
>
> Caused by: java.lang.InterruptedException: null
>
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
> ~[na:1.8.0_101]
>
> at com.google.common.util.concurrent.AbstractFuture$
> Sync.get(AbstractFuture.java:285) ~[guava-18.0.jar:na]
>
> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
>
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.
> extractThrowable(DebuggableThreadPoolExecutor.java:261)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
>
> ... 5 common frames omitted
>
>
>
> Thank you,
>
> Meg Mara
>
>
>
Re: Error Exception in Repair Thread
Posted by Erick Ramirez <fl...@gmail.com>.
2 common causes of interrupted streams are (a) network interruptions, or
(b) nodes becoming unresponsive, e.g. GC pause during high loads.
As far as network is concerned, is there a firewall in the middle? If so,
it's quite common for firewalls to close sockets when it thinks the
connection is idle and happens for long-running streams. Set the TCP
keep-alive variables on the nodes. For example:
$ sudo sysctl -w net.ipv4.tcp_keepalive_time=60
net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10
On Fri, Jul 28, 2017 at 4:50 AM, Meg Mara <mm...@digitalriver.com> wrote:
> Hello Cassandra Experts!
>
>
>
> I have seen the following errors in the system log when a scheduled
> nodetool repair operation runs on the cluster. Cassandra Version 3.0.10.
> Any thoughts or suggestions are welcome!
>
>
>
> ERROR [Repair#3794:3] 2017-07-27 17:47:22,000 CassandraDaemon.java:207 -
> Exception in thread Thread[Repair#3794:3,5,RMI Runtime]
>
> java.lang.AssertionError: java.lang.InterruptedException
>
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.
> extractThrowable(DebuggableThreadPoolExecutor.java:265)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
>
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.
> logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:225)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
>
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.
> afterExecute(DebuggableThreadPoolExecutor.java:196)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
> ~[na:1.8.0_101]
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_101]
>
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
>
> Caused by: java.lang.InterruptedException: null
>
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
> ~[na:1.8.0_101]
>
> at com.google.common.util.concurrent.AbstractFuture$
> Sync.get(AbstractFuture.java:285) ~[guava-18.0.jar:na]
>
> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
>
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.
> extractThrowable(DebuggableThreadPoolExecutor.java:261)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
>
> ... 5 common frames omitted
>
>
>
> Thank you,
>
> Meg Mara
>
>
>