You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "J.P. Eiti Kimura (JIRA)" <ji...@apache.org> on 2015/09/30 19:07:04 UTC
[jira] [Comment Edited] (CASSANDRA-10233) IndexOutOfBoundsException
in HintedHandOffManager
[ https://issues.apache.org/jira/browse/CASSANDRA-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937713#comment-14937713 ]
J.P. Eiti Kimura edited comment on CASSANDRA-10233 at 9/30/15 5:06 PM:
-----------------------------------------------------------------------
In fact I don't believe that this problem is related exclusively to versions 2.2.X.
I'm facing the very same problem here. We have upgrade from version 2.1.5 to 2.1.8, after migration we noticed the exception above in the logs.
Looks like something related to hinded hand off, according to the staktrace:
HintedHandOffManager.java:515
Here is the stack for the 2.1.8 version:
{code}
ERROR [HintedHandoff:2] 2015-09-30 16:48:38,729 CassandraDaemon.java:223 - Exception in thread Thread[HintedHandoff:2,1,main]
java.lang.IndexOutOfBoundsException: null
at java.nio.Buffer.checkIndex(Buffer.java:546) ~[na:1.8.0_45]
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:416) ~[na:1.8.0_45]
at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106) ~[apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:514) ~[apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88) ~[apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168) ~[apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-2.1.8.jar:2.1.8]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
{code}
Observations
- the nodes are in UN state
- after a node decommission, looks like the error 'moved' from one node to another
Do have any ideas what are the consequences of this error? Is hinted hand off still working?
was (Author: eitikimura):
In fact I don't believe that this problem is related exclusively to versions 2.2.X.
I'm facing the very same problem here. We have upgrade from version 2.1.5 to 2.1.8, after migration we noticed the exception above in the logs.
Looks like something related to hinded hand off, according to the staktrace:
HintedHandOffManager.java:515
Here is the stack for the 2.1.8 version:
{code}
ERROR [HintedHandoff:2] 2015-09-30 16:48:38,729 CassandraDaemon.java:223 - Exception in thread Thread[HintedHandoff:2,1,main]
java.lang.IndexOutOfBoundsException: null
at java.nio.Buffer.checkIndex(Buffer.java:546) ~[na:1.8.0_45]
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:416) ~[na:1.8.0_45]
at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106) ~[apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:514) ~[apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88) ~[apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168) ~[apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-2.1.8.jar:2.1.8]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
{code}
Do have any ideas what are the consequences of this error? Is hinted hand off still working?
> IndexOutOfBoundsException in HintedHandOffManager
> -------------------------------------------------
>
> Key: CASSANDRA-10233
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10233
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.0
> Reporter: Omri Iluz
> Assignee: Andrew Hust
>
> After upgrading our cluster to 2.2.0, the following error started showing exectly every 10 minutes on every server in the cluster:
> {noformat}
> INFO [CompactionExecutor:1381] 2015-08-31 18:31:55,506 CompactionTask.java:142 - Compacting (8e7e1520-500e-11e5-b1e3-e95897ba4d20) [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-540-big-Data.db:level=0, ]
> INFO [CompactionExecutor:1381] 2015-08-31 18:31:55,599 CompactionTask.java:224 - Compacted (8e7e1520-500e-11e5-b1e3-e95897ba4d20) 1 sstables to [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-541-big,] to level=0. 1,544,495 bytes to 1,544,495 (~100% of original) in 93ms = 15.838121MB/s. 0 total partitions merged to 4. Partition merge counts were {1:4, }
> ERROR [HintedHandoff:1] 2015-08-31 18:31:55,600 CassandraDaemon.java:182 - Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.IndexOutOfBoundsException: null
> at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_79]
> at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:410) ~[na:1.7.0_79]
> at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106) ~[apache-cassandra-2.2.0.jar:2.2.0]
> at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:515) ~[apache-cassandra-2.2.0.jar:2.2.0]
> at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88) ~[apache-cassandra-2.2.0.jar:2.2.0]
> at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168) ~[apache-cassandra-2.2.0.jar:2.2.0]
> at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-2.2.0.jar:2.2.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_79]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_79]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_79]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)