You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jason Brown (JIRA)" <ji...@apache.org> on 2018/06/13 21:08:00 UTC

[jira] [Commented] (CASSANDRA-14520) ClosedChannelException handled as FSError

    [ https://issues.apache.org/jira/browse/CASSANDRA-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511666#comment-16511666 ] 

Jason Brown commented on CASSANDRA-14520:
-----------------------------------------

Here's a stack trace [~bdeggleston] sent me before with this, unedited from the original:

{noformat}
WARN  [Stream-Deserializer-/17.177.98.2:47610-9785e92c] 2018-02-14 16:46:26,470 CompressedStreamReader.java:111 - [Stream 757ae2c0-11e9-11e8-8fc8-f7aed080d7f2] Error while reading partition DecoratedKey(-7999952965546089109, 2f645c5d1654236063405e7f6c4f273637231a5f037e696a6f3b4e411a18462b0942754651100f381d24757311071a761f46170a46524d62083458313e0442245f6b4046401f46712b1e310456326c245e080d02294d216266252d1635662e093a7f2b027d00451f2c350c1a5e177e574a785d1b702b1b400105281e4c3f6359704875124165705e2c403d0346725454531f076f3d577203233a73213027113520172a7e01467a5c776d4d546d5f57343f64215f4c5b05502922694a5d38621440362d0960005b7d1663565c79501e5c3c7a5032603e6f5e7b19645a2c017b391547212e6a7b516270317a42) from stream on ks='fullks' and table='kvtest'.
ERROR [Stream-Deserializer-/17.177.98.2:47610-9785e92c] 2018-02-14 16:46:26,476 StreamSession.java:617 - [Stream #757ae2c0-11e9-11e8-8fc8-f7aed080d7f2] Streaming error occurred on session with peer 17.177.98.2
org.apache.cassandra.streaming.StreamReceiveException: java.lang.AssertionError: stream can only read forward.
	at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63) ~[main/:na]
	at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41) ~[main/:na]
	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) ~[main/:na]
	at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178) ~[main/:na]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: java.lang.AssertionError: stream can only read forward.
	at org.apache.cassandra.streaming.compress.CompressedInputStream.position(CompressedInputStream.java:108) ~[main/:na]
	at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:94) ~[main/:na]
	at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:58) ~[main/:na]
	... 4 common frames omitted
INFO  [Stream-Deserializer-/17.177.98.2:47610-9785e92c] 2018-02-14 16:46:26,478 StreamResultFuture.java:193 - [Stream #757ae2c0-11e9-11e8-8fc8-f7aed080d7f2] Session with /17.177.98.2 is complete
WARN  [Stream-Deserializer-/17.177.98.2:47610-9785e92c] 2018-02-14 16:46:26,479 StreamResultFuture.java:220 - [Stream #757ae2c0-11e9-11e8-8fc8-f7aed080d7f2] Stream failed
ERROR [NettyStreaming-Outbound-/17.177.98.2:1] 2018-02-14 16:46:26,486 CassandraDaemon.java:211 - Exception in thread Thread[NettyStreaming-Outbound-/17.177.98.2:1,5,main]
org.apache.cassandra.io.FSReadError: java.nio.channels.ClosedByInterruptException
	at org.apache.cassandra.io.util.ChannelProxy.read(ChannelProxy.java:133) ~[main/:na]
	at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:94) ~[main/:na]
	at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:111) ~[main/:na]
	at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:53) ~[main/:na]
	at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42) ~[main/:na]
	at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:41) ~[main/:na]
	at org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:324) ~[main/:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [main/:na]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.nio.channels.ClosedByInterruptException: null
	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[na:1.8.0_131]
	at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:746) ~[na:1.8.0_131]
	at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727) ~[na:1.8.0_131]
	at org.apache.cassandra.io.util.ChannelProxy.read(ChannelProxy.java:129) ~[main/:na]
	... 12 common frames omitted
ERROR [NettyStreaming-Outbound-/17.177.98.2:1] 2018-02-14 16:46:26,488 StorageService.java:393 - Stopping gossiper
WARN  [NettyStreaming-Outbound-/17.177.98.2:1] 2018-02-14 16:46:26,489 StorageService.java:315 - Stopping gossip by operator request
INFO  [NettyStreaming-Outbound-/17.177.98.2:1] 2018-02-14 16:46:26,489 Gossiper.java:1514 - Announcing shutdown
{noformat}

Note: the {{"stream can only read forward"}} error will be resolved by CASSANDRA-13938, which is PATCH_AVAILABLE as of the time I type this.

> ClosedChannelException handled as FSError
> -----------------------------------------
>
>                 Key: CASSANDRA-14520
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14520
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Blake Eggleston
>            Assignee: Jason Brown
>            Priority: Major
>             Fix For: 4.0
>
>
> After the messaging service netty refactor, I’ve seen a few instances where a closed socket causes a ClosedChannelException (an IOException subclass) to be thrown. The exception is caught by ChannelProxy, interpreted as a disk error, and is then re-thrown as an FSError, causing the node to be shutdown.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org