You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/01/04 15:47:39 UTC
[jira] [Resolved] (SPARK-3549) Yarn client mode does not shutdown
clearnly
[ https://issues.apache.org/jira/browse/SPARK-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-3549.
------------------------------
Resolution: Cannot Reproduce
> Yarn client mode does not shutdown clearnly
> -------------------------------------------
>
> Key: SPARK-3549
> URL: https://issues.apache.org/jira/browse/SPARK-3549
> Project: Spark
> Issue Type: Bug
> Components: PySpark, YARN
> Affects Versions: 1.1.0
> Environment: Cloudera Hadoop 2.3, YARN fair-scheduler, PySpark 1.1.0
> Reporter: Chen Avnery
> Priority: Minor
>
> When I call stop() function on a instance variable of SparkContext at the end of my pyspark client mode application, I get a barrage of exceptions:
> {quote}
> 14/09/16 17:44:37 INFO SparkContext: Job finished: foreach at /home/chena/projects/sparkapps /dummy.py:34, took 24.597733296 s
> 14/09/16 17:44:37 INFO SparkUI: Stopped Spark web UI at http://workhorse.dummycorp.local:4040
> 14/09/16 17:44:37 INFO DAGScheduler: Stopping DAGScheduler
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Shutting down all executors
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Asking each executor to shut down
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 26 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 26 on g012.dummycorp.local: rem ote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 90 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 90 on g017.dummycorp.local: rem ote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 70 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 70 on g019.dummycorp.local: rem ote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 152 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 152 on g027.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 140 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 140 on g019.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 18 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 18 on g021.dummycorp.local: rem ote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 145 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 145 on g027.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 166 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 166 on g027.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 9 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 9 on g027.dummycorp.local: remo te Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 32 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 32 on g021.dummycorp.local: rem ote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 3 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 3 on g029.dummycorp.local: remo te Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 163 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 163 on g029.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 149 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 149 on g029.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 17 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 17 on g029.dummycorp.local: rem ote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 89 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 89 on g007.dummycorp.local: rem ote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 128 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 128 on g029.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 80 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 80 on g023.dummycorp.local: rem ote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 5 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 5 on g007.dummycorp.local: remo te Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 57 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 57 on g015.dummycorp.local: rem ote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 124 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 124 on g027.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 157 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 157 on g021.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 150 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 150 on g021.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 105 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 105 on g024.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 51 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 51 on g027.dummycorp.local: rem ote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 159 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 159 on g027.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 107 disconnected, so removing it
> 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 107 on g027.dummycorp.local: re mote Akka client disassociated
> 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Stopped
> 14/09/16 17:44:37 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,39146)
> 14/09/16 17:44:37 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,39146)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48080)
> 14/09/16 17:44:37 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,58341)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,58341)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,58341) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,35349)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,35349)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,57831)
> 14/09/16 17:44:37 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,48080)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,48080) not found
> 14/09/16 17:44:37 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,39146)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,57831)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,57831) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,35349)
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@2eb3ec34
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@2eb3ec 34
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@1d321764
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@1d3217 64
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@1ee061c2
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@1ee061 c2
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,39250)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,39250)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,37654)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54108)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,55074)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55074)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,55515)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,56089)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,48853)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,36788)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36788)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,57170)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,57170)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,60009)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,60881)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,41322)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,36171)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,41093)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g021.dummycorp.local,53156)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60519)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,44689)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44689)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,33434)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,33434)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,39250)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,39250)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,45596)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,32898)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,54108)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,54108) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,46744)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48837)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,56582)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,53880)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,53880)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,33434)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,33434)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,33434)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,33434)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36358)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,60519)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,60519) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,54831)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,60881)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,54400)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54400)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,56582)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,56582) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,59023)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,36131)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36131)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,36358)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,36358) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,48517)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48517)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,48898)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48898)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,34401)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,46744)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,46744) not found
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,36171)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,36171)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36171)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36171)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,48898)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,48898)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48898)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48898)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,49829)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g021.dummycorp.local,40583)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,48440)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54647)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,54647)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,54647) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,45596)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,45596) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,41759)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,48837)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,48837) not found
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,60881) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g021.dummycorp.local,47580)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g021.dummycorp.local,47580)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 21.dummycorp.local,47580) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,36748)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,39250)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,39250)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,59257)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44584)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,45485)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,59023)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,59023) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,35412)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,54817)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54817)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g021.dummycorp.local,40583)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 21.dummycorp.local,40583) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,46499)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,46499)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,36788)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,36788)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36788)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36788)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,37654)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,37654) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36171)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,49829)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,49829) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,33175)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,33175)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,45485)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,45485) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,50451)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,50451)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,39260)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,39260)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,54217)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,34576)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,60009)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,60009) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,44584)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,44584) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,54214)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54214)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,47763)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,35412)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,35412) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,59836)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55705)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,59836)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,59836) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,32898)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,32898) not found
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,44689)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,44689)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44689)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44689)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,50848)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,50848)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,48853)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,48853) not found
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,54817)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,54817)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54817)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,59257)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,59257) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,52380)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,36748)
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@5259e682
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@5259e6 82
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@bf7e726
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@bf7e72 6
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@2444569
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@244456 9
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@63f0beab
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@63f0be ab
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@3743a434
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@3743a4 34
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@406875ed
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@406875 ed
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@1ab75e8d
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@1ab75e 8d
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,60207)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,50848)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,50848)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,50848)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,50848)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,51799)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,51799)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,55705)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,55705) not found
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,51799)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,51799)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,51799)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,51799)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54817)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,50451)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,50451)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,50451)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,50451)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,34576)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,34576) not found
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,46499)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,46499)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,46499)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,46499)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,33175)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,33175)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,33175)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,33175)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,54214)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,54214)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,60303)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60303)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,47763)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,47763) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,41759)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,41759) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,60207)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,60207) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,48440)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,48440) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,34401)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,34401) not found
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,39260)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,39260)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,39260)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,39260)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,55806)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,57170)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,57170)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,57170)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,57170)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,46077)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,36748)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,36748)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,36748)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,36748)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,53880)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,53880)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,53880)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,53880)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,60303)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,60303)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,41093)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,41093) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,41322)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,41322) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,46077)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,46077) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g021.dummycorp.local,53156)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 21.dummycorp.local,53156) not found
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,48517)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,48517)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48517)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48517)
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,36131)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,36131)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36131)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36131)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,54217)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,54217) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,55515)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,55515) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,54831)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,54831) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,56089)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,56089) not found
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,54400)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,54400)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54400)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54400)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,52380)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,52380) not found
> 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,55074)
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,55074)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55074)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55074)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60303)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60303)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54214)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54214)
> 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,55806)
> 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,55806) not found
> 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@653f8d49
> 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@653f8d 49
> java.nio.channels.CancelledKeyException
> at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/09/16 17:44:38 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
> 14/09/16 17:44:38 INFO ConnectionManager: Selector thread was interrupted!
> 14/09/16 17:44:38 INFO ConnectionManager: ConnectionManager stopped
> 14/09/16 17:44:38 INFO MemoryStore: MemoryStore cleared
> 14/09/16 17:44:38 INFO BlockManager: BlockManager stopped
> 14/09/16 17:44:38 INFO BlockManagerMaster: BlockManagerMaster stopped
> 14/09/16 17:44:38 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
> 14/09/16 17:44:38 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceedin g with flushing remote transports.
> 14/09/16 17:44:39 INFO SparkContext: Successfully stopped SparkContext
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org