You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Milan Milosevic (JIRA)" <ji...@apache.org> on 2017/06/19 10:33:00 UTC
[jira] [Created] (CASSANDRA-13619)
java.nio.BufferOverflowException: null while flushing hints
Milan Milosevic created CASSANDRA-13619:
-------------------------------------------
Summary: java.nio.BufferOverflowException: null while flushing hints
Key: CASSANDRA-13619
URL: https://issues.apache.org/jira/browse/CASSANDRA-13619
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Milan Milosevic
I'm seeing the following exception running Cassandra 3.0.11 on 21 node cluster in two AWS regions when half of the nodes in one region go down, and the load is high on the rest of the nodes:
WARN [SharedPool-Worker-10] 2017-06-14 12:57:15,017 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-10,5,main]: {}
java.lang.RuntimeException: java.nio.BufferOverflowException
at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2549) ~[apache-cassandra-3.0.11.jar:3.0.11]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0-zing_17.03.1.0]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.11.jar:3.0.11]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0-zing_17.03.1.0]
Caused by: java.nio.BufferOverflowException: null
at org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:195) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:258) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.db.Columns$Serializer.serialize(Columns.java:405) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:407) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:120) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:625) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:305) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.hints.Hint$Serializer.serialize(Hint.java:141) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.hints.HintsBuffer$Allocation.write(HintsBuffer.java:251) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.hints.HintsBuffer$Allocation.write(HintsBuffer.java:230) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.hints.HintsBufferPool.write(HintsBufferPool.java:61) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.hints.HintsService.write(HintsService.java:154) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.service.StorageProxy$11.runMayThrow(StorageProxy.java:2627) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2545) ~[apache-cassandra-3.0.11.jar:3.0.11]
... 5 common frames omitted
Relevant configurations from cassandra.yaml:
-cassandra_hinted_handoff_throttle_in_kb: 1024
cassandra_max_hints_delivery_threads: 4
-cassandra_hints_flush_period_in_ms: 10000
-cassandra_max_hints_file_size_in_mb: 512
When I reduce -cassandra_hints_flush_period_in_ms: 10000 to 5000, the number of exceptions lowers significantly, but they are still present.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org