You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chen Avnery (JIRA)" <ji...@apache.org> on 2014/09/16 17:05:33 UTC
[jira] [Created] (SPARK-3549) Yarn client mode does not shutdown
clearnly
Chen Avnery created SPARK-3549:
----------------------------------
Summary: Yarn client mode does not shutdown clearnly
Key: SPARK-3549
URL: https://issues.apache.org/jira/browse/SPARK-3549
Project: Spark
Issue Type: Bug
Components: PySpark, YARN
Affects Versions: 1.1.0
Environment: Cloudera Hadoop 2.3, YARN fair-scheduler, PySpark 1.1.0
Reporter: Chen Avnery
Priority: Minor
When I call SparkContext.stop() at the end of my pyspark client mode application, I get a barrage of exceptions:
{quote}
14/09/16 17:44:37 INFO SparkContext: Job finished: foreach at /home/chena/projects/sparkapps /dummy.py:34, took 24.597733296 s
14/09/16 17:44:37 INFO SparkUI: Stopped Spark web UI at http://workhorse.dummycorp.local:4040
14/09/16 17:44:37 INFO DAGScheduler: Stopping DAGScheduler
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Shutting down all executors
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Asking each executor to shut down
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 26 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 26 on g012.dummycorp.local: rem ote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 90 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 90 on g017.dummycorp.local: rem ote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 70 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 70 on g019.dummycorp.local: rem ote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 152 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 152 on g027.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 140 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 140 on g019.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 18 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 18 on g021.dummycorp.local: rem ote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 145 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 145 on g027.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 166 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 166 on g027.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 9 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 9 on g027.dummycorp.local: remo te Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 32 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 32 on g021.dummycorp.local: rem ote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 3 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 3 on g029.dummycorp.local: remo te Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 163 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 163 on g029.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 149 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 149 on g029.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 17 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 17 on g029.dummycorp.local: rem ote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 89 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 89 on g007.dummycorp.local: rem ote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 128 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 128 on g029.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 80 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 80 on g023.dummycorp.local: rem ote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 5 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 5 on g007.dummycorp.local: remo te Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 57 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 57 on g015.dummycorp.local: rem ote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 124 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 124 on g027.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 157 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 157 on g021.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 150 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 150 on g021.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 105 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 105 on g024.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 51 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 51 on g027.dummycorp.local: rem ote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 159 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 159 on g027.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 107 disconnected, so removing it
14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 107 on g027.dummycorp.local: re mote Akka client disassociated
14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Stopped
14/09/16 17:44:37 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,39146)
14/09/16 17:44:37 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,39146)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48080)
14/09/16 17:44:37 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,58341)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,58341)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,58341) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,35349)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,35349)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,57831)
14/09/16 17:44:37 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,48080)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,48080) not found
14/09/16 17:44:37 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,39146)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,57831)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,57831) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,35349)
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@2eb3ec34
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@2eb3ec 34
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@1d321764
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@1d3217 64
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@1ee061c2
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@1ee061 c2
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,39250)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,39250)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,37654)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54108)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,55074)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55074)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,55515)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,56089)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,48853)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,36788)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36788)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,57170)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,57170)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,60009)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,60881)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,41322)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,36171)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,41093)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g021.dummycorp.local,53156)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60519)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,44689)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44689)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,33434)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,33434)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,39250)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,39250)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,45596)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,32898)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,54108)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,54108) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,46744)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48837)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,56582)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,53880)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,53880)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,33434)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,33434)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,33434)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,33434)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36358)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,60519)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,60519) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,54831)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,60881)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,54400)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54400)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,56582)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,56582) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,59023)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,36131)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36131)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,36358)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,36358) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,48517)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48517)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,48898)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48898)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,34401)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,46744)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,46744) not found
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,36171)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,36171)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36171)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36171)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,48898)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,48898)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48898)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48898)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,49829)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g021.dummycorp.local,40583)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,48440)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54647)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,54647)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,54647) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,45596)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,45596) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,41759)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,48837)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,48837) not found
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,60881) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g021.dummycorp.local,47580)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g021.dummycorp.local,47580)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 21.dummycorp.local,47580) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,36748)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,39250)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,39250)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,59257)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44584)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,45485)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,59023)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,59023) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,35412)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,54817)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54817)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g021.dummycorp.local,40583)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 21.dummycorp.local,40583) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,46499)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,46499)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,36788)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,36788)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36788)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36788)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,37654)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,37654) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36171)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,49829)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,49829) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,33175)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,33175)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,45485)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,45485) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,50451)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,50451)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,39260)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,39260)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,54217)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,34576)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,60009)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,60009) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,44584)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,44584) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,54214)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54214)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,47763)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,35412)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,35412) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,59836)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55705)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,59836)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,59836) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,32898)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,32898) not found
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,44689)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,44689)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44689)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44689)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,50848)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,50848)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,48853)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,48853) not found
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,54817)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,54817)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54817)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,59257)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,59257) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,52380)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,36748)
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@5259e682
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@5259e6 82
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@bf7e726
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@bf7e72 6
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@2444569
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@244456 9
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@63f0beab
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@63f0be ab
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@3743a434
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@3743a4 34
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@406875ed
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@406875 ed
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@1ab75e8d
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@1ab75e 8d
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,60207)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,50848)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,50848)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,50848)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,50848)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,51799)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,51799)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,55705)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,55705) not found
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,51799)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,51799)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,51799)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,51799)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54817)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,50451)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,50451)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,50451)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,50451)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,34576)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,34576) not found
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,46499)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,46499)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,46499)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,46499)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,33175)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,33175)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,33175)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,33175)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,54214)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,54214)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,60303)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60303)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,47763)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,47763) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,41759)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,41759) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,60207)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,60207) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,48440)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,48440) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,34401)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,34401) not found
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,39260)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,39260)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,39260)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,39260)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,55806)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,57170)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,57170)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,57170)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,57170)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,46077)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,36748)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,36748)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,36748)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,36748)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,53880)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,53880)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,53880)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,53880)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,60303)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,60303)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,41093)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,41093) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,41322)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,41322) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,46077)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,46077) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g021.dummycorp.local,53156)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 21.dummycorp.local,53156) not found
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,48517)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,48517)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48517)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48517)
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,36131)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,36131)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36131)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36131)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,54217)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,54217) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,55515)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,55515) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,54831)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,54831) not found
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,56089)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,56089) not found
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,54400)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,54400)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54400)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54400)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,52380)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,52380) not found
14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,55074)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,55074)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55074)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55074)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60303)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60303)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54214)
14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54214)
14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,55806)
14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,55806) not found
14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@653f8d49
14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@653f8d 49
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/09/16 17:44:38 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
14/09/16 17:44:38 INFO ConnectionManager: Selector thread was interrupted!
14/09/16 17:44:38 INFO ConnectionManager: ConnectionManager stopped
14/09/16 17:44:38 INFO MemoryStore: MemoryStore cleared
14/09/16 17:44:38 INFO BlockManager: BlockManager stopped
14/09/16 17:44:38 INFO BlockManagerMaster: BlockManagerMaster stopped
14/09/16 17:44:38 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
14/09/16 17:44:38 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceedin g with flushing remote transports.
14/09/16 17:44:39 INFO SparkContext: Successfully stopped SparkContext
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org