Cancelled Key exception

[ab...@thomsonreuters.com]

Hi Sparklers,

I was wondering if some else has also encountered this... (Actually I am not even sure if this is an issue)...

I have a spark job that reads data from Hbase does a bunch of transformation

sparkContext.newAPIHadoopRDD -> flatMapToPair -> groupByKey -> mapValues

After this I do a take(10) on the result to print it out in the log file.

I always get the results which I am 100% sure are correct. However, every once in a while I get the following in the log file even if the results are correct -

14/09/22 20:16:22 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
14/09/22 20:16:22 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
14/09/22 20:16:22 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
14/09/22 20:16:22 INFO Remoting: Remoting shut down
14/09/22 20:16:22 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
14/09/22 20:16:24 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@a4503d8
14/09/22 20:16:24 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(tr-pan-xxxx-04,55008)
14/09/22 20:16:24 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(tr-pan-xxxx-04,55008)
14/09/22 20:16:24 ERROR ConnectionManager: Corresponding SendingConnectionManagerId not found
14/09/22 20:16:24 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@a4503d8
java.nio.channels.CancelledKeyException
        at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:363)
        at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:116)

The spark dashboard also does not show any error of executors failing.

Would it be possible for someone to throw some light into what this actually means? and whether we should be concerned about it?

I am running a Cloudera CDH 5.1.2 cluster with I believe spark v1.0.0

The spark job is submitted to yarn.

-abe