You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by akshaym <ak...@hashmapinc.com> on 2018/02/27 06:55:23 UTC
Spark 'close' API call hangs within ignite service grid
Hi,
I am running a sprak job within ignite service grid. After job is done I am
calling close() on spark session. This is working for stable version of
ignite i.e. 2.3.0. However, its not working with 2.5.0-SNAPSHOT version.
Here are the detailed logs :
/
Closing spark context
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][G] Shutdown hook is removed.
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][G] Unregistered MBean:
org.apache:clsLdr=764c12b6,group=Kernal,name=Ignition
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][IgniteKernal] Notifying
lifecycle beans.
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
org.spark_project.jetty.server.Server@19480852
[27-02-2018 12:12:18][DEBUG][svc-#88][Server] doStop
org.spark_project.jetty.server.Server@19480852
[27-02-2018 12:12:18][DEBUG][SparkUI-161][QueuedThreadPool] ran
SparkUI-161-acceptor-0@1e34fb5-ServerConnector@a75cb0a{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
[27-02-2018 12:12:18][DEBUG][svc-#88][Server] Graceful shutdown
org.spark_project.jetty.server.Server@19480852 by
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
Spark@a75cb0a{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
org.spark_project.jetty.server.ServerConnector$ServerConnectorManager@7313f6d3
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=0 selected=0
[27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Stopping
org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=0 selected=0
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridIoManager] Removed
message listener [topic=TOPIC_DATASTREAM,
lsnr=org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1@378382bf]
[27-02-2018
12:12:18][DEBUG][grid-data-loader-flusher-#40][DataStreamProcessor] Caught
interrupted exception: java.lang.InterruptedException
[27-02-2018
12:12:18][DEBUG][grid-data-loader-flusher-#40][DataStreamProcessor] Grid
runnable finished due to interruption without cancellation:
grid-data-loader-flusher
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][DataStreamProcessor] Stopped
data streamer processor.
[27-02-2018 12:12:18][DEBUG][session-timeout-worker-#34][GridRestProcessor]
Caught interrupted exception: java.lang.InterruptedException: sleep
interrupted
[27-02-2018 12:12:18][DEBUG][session-timeout-worker-#34][GridRestProcessor]
Grid runnable finished due to interruption without cancellation:
session-timeout-worker
[27-02-2018 12:12:18][DEBUG][nio-acceptor-#35][GridTcpRestProtocol]
Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
[27-02-2018 12:12:18][DEBUG][nio-acceptor-#35][GridTcpRestProtocol] Closing
all listening sockets.
[27-02-2018 12:12:18][DEBUG][nio-acceptor-#35][GridTcpRestProtocol] Closing
NIO selector.
[27-02-2018 12:12:18][DEBUG][nio-acceptor-#35][GridTcpRestProtocol] Grid
runnable finished due to interruption without cancellation: nio-acceptor
[27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Queued change
org.spark_project.jetty.io.ManagedSelector$CloseEndPoints@28c2db1e on
org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=0 selected=0
[27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Selector loop
woken up from select, 0/0 selected
[27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Running change
org.spark_project.jetty.io.ManagedSelector$CloseEndPoints@28c2db1e
[27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Closing 0
endPoints on org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=0
selected=0
[27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Closed 0
endPoints on org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=0
selected=0
[27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Selector loop
waiting on select
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol]
Cancelling grid runnable: ByteBufferNioClientWorker
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0,
bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-rest-0, igniteInstanceName=null, finished=false,
hashCode=2091822083, interrupted=false,
runner=grid-nio-worker-tcp-rest-0-#36]]]
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-0-#36][GridTcpRestProtocol]
Closing all connected client sockets.
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol]
Cancelling grid runnable: ByteBufferNioClientWorker
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0,
bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null, finished=false,
hashCode=629894509, interrupted=false,
runner=grid-nio-worker-tcp-rest-1-#37]]]
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-0-#36][GridTcpRestProtocol]
Closing NIO selector.
[27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Queued change
org.spark_project.jetty.io.ManagedSelector$CloseSelector@b08d809 on
org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=0 selected=0
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-0-#36][GridTcpRestProtocol] Grid
runnable finished due to cancellation: grid-nio-worker-tcp-rest-0
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol]
Closing all connected client sockets.
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol]
Closing NIO selector.
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol] Grid
runnable finished due to cancellation: grid-nio-worker-tcp-rest-1
[27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Selector loop
woken up from select, 0/0 selected
[27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Running change
org.spark_project.jetty.io.ManagedSelector$CloseSelector@b08d809
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol]
Cancelling grid runnable: ByteBufferNioClientWorker
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
super=AbstractNioClientWorker [idx=2, bytesRcvd=0, bytesSent=0,
bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-rest-2, igniteInstanceName=null, finished=false,
hashCode=266454991, interrupted=false,
runner=grid-nio-worker-tcp-rest-2-#38]]]
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-2-#38][GridTcpRestProtocol]
Closing all connected client sockets.
[27-02-2018 12:12:18][DEBUG][SparkUI-160][ExecuteProduceConsume] EPC
Prod/org.spark_project.jetty.io.ManagedSelector$SelectorProducer@2a26278b
produced null
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol]
Cancelling grid runnable: ByteBufferNioClientWorker
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
super=AbstractNioClientWorker [idx=3, bytesRcvd=0, bytesSent=0,
bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-rest-3, igniteInstanceName=null, finished=false,
hashCode=1884453621, interrupted=false,
runner=grid-nio-worker-tcp-rest-3-#39]]]
[27-02-2018 12:12:18][DEBUG][SparkUI-160][ExecuteProduceConsume] EPC
Idle/org.spark_project.jetty.io.ManagedSelector$SelectorProducer@2a26278b
produce exit
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol] Joining
grid runnable: ByteBufferNioClientWorker
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0,
bytesRcvd0=0, bytesSent0=0, select=false, super=GridWorker
[name=grid-nio-worker-tcp-rest-0, igniteInstanceName=null, finished=true,
hashCode=2091822083, interrupted=unknown, runner=null]]]
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-2-#38][GridTcpRestProtocol]
Closing NIO selector.
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol] Joining
grid runnable: ByteBufferNioClientWorker
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0,
bytesRcvd0=0, bytesSent0=0, select=false, super=GridWorker
[name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null, finished=true,
hashCode=629894509, interrupted=unknown, runner=null]]]
[27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Stopped
org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=-1 selected=-1
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol] Joining
grid runnable: ByteBufferNioClientWorker
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
super=AbstractNioClientWorker [idx=2, bytesRcvd=0, bytesSent=0,
bytesRcvd0=0, bytesSent0=0, select=false, super=GridWorker
[name=grid-nio-worker-tcp-rest-2, igniteInstanceName=null, finished=true,
hashCode=266454991, interrupted=true,
runner=grid-nio-worker-tcp-rest-2-#38]]]
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=-1 selected=-1
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol] Joining
grid runnable: ByteBufferNioClientWorker
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
super=AbstractNioClientWorker [idx=3, bytesRcvd=0, bytesSent=0,
bytesRcvd0=0, bytesSent0=0, select=false, super=GridWorker
[name=grid-nio-worker-tcp-rest-3, igniteInstanceName=null, finished=false,
hashCode=1884453621, interrupted=true,
runner=grid-nio-worker-tcp-rest-3-#39]]]
[27-02-2018 12:12:18][DEBUG][SparkUI-160][QueuedThreadPool] ran
org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=-1 selected=-1
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-3-#39][GridTcpRestProtocol]
Closing all connected client sockets.
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-3-#39][GridTcpRestProtocol]
Closing NIO selector.
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=0 selected=0
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-3-#39][GridTcpRestProtocol] Grid
runnable finished due to cancellation: grid-nio-worker-tcp-rest-3
[27-02-2018 12:12:18][INFO ][SparkListenerBus][GridTcpRestProtocol] Command
protocol successfully stopped: TCP binary
[27-02-2018
12:12:18][DEBUG][grid-nio-worker-tcp-rest-2-#38][GridTcpRestProtocol] Grid
runnable finished due to cancellation: grid-nio-worker-tcp-rest-2
[27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Stopping
org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=0 selected=0
[27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Queued change
org.spark_project.jetty.io.ManagedSelector$CloseEndPoints@66931c58 on
org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=0 selected=0
[27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Selector loop
woken up from select, 0/0 selected
[27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Running change
org.spark_project.jetty.io.ManagedSelector$CloseEndPoints@66931c58
[27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Closing 0
endPoints on org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=0
selected=0
[27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Closed 0
endPoints on org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=0
selected=0
[27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Selector loop
waiting on select
[27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Queued change
org.spark_project.jetty.io.ManagedSelector$CloseSelector@1e8d41a on
org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=0 selected=0
[27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Selector loop
woken up from select, 0/0 selected
[27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Running change
org.spark_project.jetty.io.ManagedSelector$CloseSelector@1e8d41a
[27-02-2018 12:12:18][DEBUG][SparkUI-159][ExecuteProduceConsume] EPC
Prod/org.spark_project.jetty.io.ManagedSelector$SelectorProducer@7c6b8701
produced null
[27-02-2018 12:12:18][DEBUG][SparkUI-159][ExecuteProduceConsume] EPC
Idle/org.spark_project.jetty.io.ManagedSelector$SelectorProducer@7c6b8701
produce exit
[27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Stopped
org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=-1 selected=-1
[27-02-2018 12:12:18][DEBUG][SparkUI-159][QueuedThreadPool] ran
org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=-1 selected=-1
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=-1 selected=-1
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
org.spark_project.jetty.server.ServerConnector$ServerConnectorManager@7313f6d3
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
HttpConnectionFactory@181ade16[HTTP/1.1]
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
HttpConnectionFactory@181ade16[HTTP/1.1]
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
org.spark_project.jetty.util.thread.ScheduledExecutorScheduler@4ced9cbe
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
org.spark_project.jetty.util.thread.ScheduledExecutorScheduler@4ced9cbe
[27-02-2018 12:12:18][INFO ][svc-#88][AbstractConnector] Stopped
Spark@a75cb0a{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
Spark@a75cb0a{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractHandler] stopping
org.spark_project.jetty.server.Server@19480852
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
org.spark_project.jetty.server.handler.ContextHandlerCollection@31522a7a[org.spark_project.jetty.server.handler.gzip.GzipHandler@61de4808,
org.spark_project.jetty.server.handler.gzip.GzipHandler@55fef7fb,
org.spark_project.jetty.server.handler.gzip.GzipHandler@3c790dd4,
org.spark_project.jetty.server.handler.gzip.GzipHandler@1cd8e1db,
org.spark_project.jetty.server.handler.gzip.GzipHandler@f557440,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2db61b7e,
org.spark_project.jetty.server.handler.gzip.GzipHandler@59d5b4b1,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2d84069,
org.spark_project.jetty.server.handler.gzip.GzipHandler@268bc306,
org.spark_project.jetty.server.handler.gzip.GzipHandler@3bfb5142,
org.spark_project.jetty.server.handler.gzip.GzipHandler@4fff0ab0,
org.spark_project.jetty.server.handler.gzip.GzipHandler@fe4c02e,
org.spark_project.jetty.server.handler.gzip.GzipHandler@778fa127,
org.spark_project.jetty.server.handler.gzip.GzipHandler@169f8fd5,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2102bd3c,
org.spark_project.jetty.server.handler.gzip.GzipHandler@8cefc82,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2c2f225,
org.spark_project.jetty.server.handler.gzip.GzipHandler@6849348f,
org.spark_project.jetty.server.handler.gzip.GzipHandler@5c168fe7,
org.spark_project.jetty.server.handler.gzip.GzipHandler@23c3520b,
org.spark_project.jetty.server.handler.gzip.GzipHandler@3685df8c,
org.spark_project.jetty.server.handler.gzip.GzipHandler@1749789f,
org.spark_project.jetty.server.handler.gzip.GzipHandler@7f48a426,
org.spark_project.jetty.server.handler.gzip.GzipHandler@4622f2ce,
org.spark_project.jetty.server.handler.gzip.GzipHandler@5e433c39,
o.s.j.s.ServletContextHandler@5a643879{/metrics/json,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@6f8e11a2{/SQL,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@299f3860{/SQL/json,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@45005fd3{/SQL/execution,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@6dfd7426{/SQL/execution/json,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@10407136{/static/sql,null,SHUTDOWN,@Spark}]
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractHandler] stopping
org.spark_project.jetty.server.handler.ContextHandlerCollection@31522a7a[org.spark_project.jetty.server.handler.gzip.GzipHandler@61de4808,
org.spark_project.jetty.server.handler.gzip.GzipHandler@55fef7fb,
org.spark_project.jetty.server.handler.gzip.GzipHandler@3c790dd4,
org.spark_project.jetty.server.handler.gzip.GzipHandler@1cd8e1db,
org.spark_project.jetty.server.handler.gzip.GzipHandler@f557440,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2db61b7e,
org.spark_project.jetty.server.handler.gzip.GzipHandler@59d5b4b1,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2d84069,
org.spark_project.jetty.server.handler.gzip.GzipHandler@268bc306,
org.spark_project.jetty.server.handler.gzip.GzipHandler@3bfb5142,
org.spark_project.jetty.server.handler.gzip.GzipHandler@4fff0ab0,
org.spark_project.jetty.server.handler.gzip.GzipHandler@fe4c02e,
org.spark_project.jetty.server.handler.gzip.GzipHandler@778fa127,
org.spark_project.jetty.server.handler.gzip.GzipHandler@169f8fd5,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2102bd3c,
org.spark_project.jetty.server.handler.gzip.GzipHandler@8cefc82,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2c2f225,
org.spark_project.jetty.server.handler.gzip.GzipHandler@6849348f,
org.spark_project.jetty.server.handler.gzip.GzipHandler@5c168fe7,
org.spark_project.jetty.server.handler.gzip.GzipHandler@23c3520b,
org.spark_project.jetty.server.handler.gzip.GzipHandler@3685df8c,
org.spark_project.jetty.server.handler.gzip.GzipHandler@1749789f,
org.spark_project.jetty.server.handler.gzip.GzipHandler@7f48a426,
org.spark_project.jetty.server.handler.gzip.GzipHandler@4622f2ce,
org.spark_project.jetty.server.handler.gzip.GzipHandler@5e433c39,
o.s.j.s.ServletContextHandler@5a643879{/metrics/json,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@6f8e11a2{/SQL,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@299f3860{/SQL/json,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@45005fd3{/SQL/execution,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@6dfd7426{/SQL/execution/json,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@10407136{/static/sql,null,SHUTDOWN,@Spark}]
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
org.spark_project.jetty.server.handler.ContextHandlerCollection@31522a7a[org.spark_project.jetty.server.handler.gzip.GzipHandler@61de4808,
org.spark_project.jetty.server.handler.gzip.GzipHandler@55fef7fb,
org.spark_project.jetty.server.handler.gzip.GzipHandler@3c790dd4,
org.spark_project.jetty.server.handler.gzip.GzipHandler@1cd8e1db,
org.spark_project.jetty.server.handler.gzip.GzipHandler@f557440,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2db61b7e,
org.spark_project.jetty.server.handler.gzip.GzipHandler@59d5b4b1,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2d84069,
org.spark_project.jetty.server.handler.gzip.GzipHandler@268bc306,
org.spark_project.jetty.server.handler.gzip.GzipHandler@3bfb5142,
org.spark_project.jetty.server.handler.gzip.GzipHandler@4fff0ab0,
org.spark_project.jetty.server.handler.gzip.GzipHandler@fe4c02e,
org.spark_project.jetty.server.handler.gzip.GzipHandler@778fa127,
org.spark_project.jetty.server.handler.gzip.GzipHandler@169f8fd5,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2102bd3c,
org.spark_project.jetty.server.handler.gzip.GzipHandler@8cefc82,
org.spark_project.jetty.server.handler.gzip.GzipHandler@2c2f225,
org.spark_project.jetty.server.handler.gzip.GzipHandler@6849348f,
org.spark_project.jetty.server.handler.gzip.GzipHandler@5c168fe7,
org.spark_project.jetty.server.handler.gzip.GzipHandler@23c3520b,
org.spark_project.jetty.server.handler.gzip.GzipHandler@3685df8c,
org.spark_project.jetty.server.handler.gzip.GzipHandler@1749789f,
org.spark_project.jetty.server.handler.gzip.GzipHandler@7f48a426,
org.spark_project.jetty.server.handler.gzip.GzipHandler@4622f2ce,
org.spark_project.jetty.server.handler.gzip.GzipHandler@5e433c39,
o.s.j.s.ServletContextHandler@5a643879{/metrics/json,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@6f8e11a2{/SQL,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@299f3860{/SQL/json,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@45005fd3{/SQL/execution,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@6dfd7426{/SQL/execution/json,null,SHUTDOWN,@Spark},
o.s.j.s.ServletContextHandler@10407136{/static/sql,null,SHUTDOWN,@Spark}]
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
org.spark_project.jetty.server.handler.ErrorHandler@1ed1eb6e
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractHandler] stopping
org.spark_project.jetty.server.handler.ErrorHandler@1ed1eb6e
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
org.spark_project.jetty.server.handler.ErrorHandler@1ed1eb6e
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
SparkUI{STARTED,8<=8<=200,i=8,q=0}
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
SparkUI{STOPPED,8<=8<=200,i=0,q=0}
[27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
org.spark_project.jetty.server.Server@19480852
[27-02-2018 12:12:18][INFO ][svc-#88][SparkUI] Stopped Spark web UI at
http://192.168.1.98:4040
[27-02-2018 12:12:18][INFO ][SparkListenerBus][GridJettyRestProtocol]
Command protocol successfully stopped: Jetty REST
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridRestProcessor] REST
processor stopped.
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridIoManager] Removed
message listener [topic=TOPIC_JOB_SIBLINGS,
lsnr=org.apache.ignite.internal.processors.task.GridTaskProcessor$JobSiblingsMessageListener@32f763ec]
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridIoManager] Removed
message listener [topic=TOPIC_TASK_CANCEL,
lsnr=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskCancelMessageListener@6ea394a1]
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTaskProcessor] Finished
executing task processor onKernalStop() callback.
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridIoManager] Removed
message listener [topic=TOPIC_JOB,
lsnr=org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener@16cae9c4]
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridIoManager] Removed
message listener [topic=TOPIC_JOB_CANCEL,
lsnr=org.apache.ignite.internal.processors.job.GridJobProcessor$JobCancelListener@1114a9fb]
[27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridJobWorker] Joining grid
runnable: GridJobWorker [createTime=1519713725262, startTime=1519713725292,
finishTime=0, taskNode=TcpDiscoveryNode
[id=1b035a31-cd11-4a73-a753-8f30824ed940, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 172.17.0.1, 192.168.1.98], sockAddrs=[/172.17.0.1:0,
/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /192.168.1.98:0], discPort=0, order=3,
intOrder=3, lastExchangeTime=1519713675143, loc=false,
ver=2.5.0#20180213-sha1:c6619637, isClient=true], internal=false,
marsh=org.apache.ignite.internal.binary.BinaryMarshaller@7dc19a70,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=org.apache.ignite.internal.processors.service.GridServiceProxy$ServiceProxyCallable,
dep=GridDeployment [ts=1519713501172, depMode=SHARED,
clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6,
clsLdrId=6429bf5d161-a88e471d-c381-40b7-8665-048d5789cad6, userVer=0,
loc=true,
sampleClsName=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
pendingUndeploy=false, undeployed=false, usage=1],
taskClsName=org.apache.ignite.internal.processors.service.GridServiceProxy$ServiceProxyCallable,
sesId=fb35ef5d161-1b035a31-cd11-4a73-a753-8f30824ed940,
startTime=1519713725229, endTime=9223372036854775807,
taskNodeId=1b035a31-cd11-4a73-a753-8f30824ed940,
clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=1b035a31-cd11-4a73-a753-8f30824ed940,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=531343034]], execName=null],
jobId=0c35ef5d161-1b035a31-cd11-4a73-a753-8f30824ed940],
jobCtx=GridJobContextImpl
[jobId=0c35ef5d161-1b035a31-cd11-4a73-a753-8f30824ed940, timeoutObj=null,
attrs={}], dep=GridDeployment [ts=1519713501172, depMode=SHARED,
clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6,
clsLdrId=6429bf5d161-a88e471d-c381-40b7-8665-048d5789cad6, userVer=0,
loc=true,
sampleClsName=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
pendingUndeploy=false, undeployed=false, usage=1], finishing=false,
masterLeaveGuard=false, timedOut=false, sysCancelled=false,
sysStopping=false, isStarted=true, job=C2 [c=ServiceProxyCallable
[mtdName=run, svcName=jdbcReaderService, ignite=IgniteKernal
[cfg=IgniteConfiguration [igniteInstanceName=null, pubPoolSize=8,
svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8,
mgmtPoolSize=4, igfsPoolSize=4, dataStreamerPoolSize=8,
utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2,
qryPoolSize=8, igniteHome=/home/akshay/Downloads/igniteSource/ignite,
igniteWorkDir=/home/akshay/Downloads/igniteSource/ignite/work,
mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e,
nodeId=a88e471d-c381-40b7-8665-048d5789cad6,
marsh=org.apache.ignite.internal.binary.BinaryMarshaller@7dc19a70,
marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000,
sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000,
metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000,
marsh=JdkMarshaller
[clsFilter=org.apache.ignite.internal.IgniteKernal$5@44bc3a8e], reconCnt=10,
reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false,
clientReconnectDisabled=false], segPlc=STOP, segResolveAttempts=2,
waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000,
commSpi=TcpCommunicationSpi
[connectGate=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$ConnectGateway@dd71cc5,
connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$6@4149c59c,
enableForcibleNodeKill=false, enableTroubleshootingLog=false,
srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@2d29b4ee,
locAddr=null, locHost=0.0.0.0/0.0.0.0, locPort=47100, locPortRange=100,
shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000,
connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768,
sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0,
nioSrvr=GridNioServer [selectorSpins=0,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=org.apache.ignite.internal.util.nio.GridDirectParser@38e7ba9e,
directMode=true], GridConnectionBytesVerifyFilter], closed=false,
directBuf=true, tcpNoDelay=true, sockSndBuf=32768, sockRcvBuf=32768,
writeTimeout=2000, idleTimeout=600000, skipWrite=false, skipRead=false,
locAddr=0.0.0.0/0.0.0.0:47100, order=LITTLE_ENDIAN, sndQueueLimit=0,
directMode=true,
metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@6e2aa843,
sslFilter=null, msgQueueLsnr=null, readerMoveCnt=0, writerMoveCnt=0,
readWriteSelectorsAssign=false], shmemSrv=null, usePairedConnections=false,
connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false,
ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000,
lsnr=org.apache.ignite.internal.managers.communication.GridIoManager$2@378dfd3b,
boundTcpPort=47100, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0,
addrRslvr=null,
ctxInitLatch=java.util.concurrent.CountDownLatch@69ee81fc[Count = 0],
stopping=false,
metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@6e2aa843],
evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@6f36c2f0,
colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi
[lsnr=org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore$LocalDeploymentListener@6b3784c1],
indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@3b35a229,
addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=1,
txCfg=org.apache.ignite.configuration.TransactionConfiguration@9816741,
cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED,
p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100,
timeSrvPortRange=100, failureDetectionTimeout=10000,
clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null,
connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@1e16c0aa,
odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration
[seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null,
grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null,
binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration
[sysRegionInitSize=41943040, sysCacheMaxSize=104857600, pageSize=4096,
concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default,
maxSize=3340354355, initSize=268435456, swapPath=null,
pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100,
metricsEnabled=false, metricsSubIntervalCount=5,
metricsRateTimeInterval=60000, persistenceEnabled=false,
checkpointPageBufSize=0], storagePath=null, checkpointFreq=180000,
lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL,
walHistSize=20, walSegments=10, walSegmentSize=67108864, walPath=db/wal,
walArchivePath=db/wal/archive, metricsEnabled=false, walMode=DEFAULT,
walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000,
walRecordIterBuffSize=67108864, alwaysWriteFullPages=false,
fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@2f67a4d3,
metricsSubIntervalCnt=5, metricsRateTimeInterval=60000,
walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false,
walCompactionEnabled=false], activeOnStart=true, autoActivation=true,
longQryWarnTimeout=3000, sqlConnCfg=null,
cliConnCfg=ClientConnectorConfiguration [host=null, port=10800,
portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true,
maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0,
jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false,
useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null]],
igniteInstanceName=null, startTime=1519713491243,
rsrcCtx=org.apache.ignite.internal.processors.resource.GridSpringResourceContextImpl@7b32aa52,
reconnectState=org.apache.ignite.internal.IgniteKernal$ReconnectState@7e04cce0]]],
held=0, partsReservation=null, reqTopVer=null, execName=null]
[27-02-2018 12:12:18][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=1e69bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713738695,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7019c482],
process=true]
[27-02-2018 12:12:18][DEBUG][nio-acceptor-#29][ClientListenerProcessor]
Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
[27-02-2018 12:12:18][DEBUG][nio-acceptor-#24][TcpCommunicationSpi]
Balancing data [min0=0, minIdx=2, max0=-1, maxIdx=-1]
[27-02-2018 12:12:19][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=d379bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713739141,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7049b82b],
process=true]
[27-02-2018 12:12:19][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713739160,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
[27-02-2018
12:12:19][DEBUG][exchange-worker-#42][GridCachePartitionExchangeManager]
Before waiting for exchange futures [futs[], worker=GridWorker
[name=partition-exchanger, igniteInstanceName=null, finished=false,
hashCode=569919564, interrupted=false, runner=exchange-worker-#42]]
[27-02-2018 12:12:19][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=9679bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713739639,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@9a27600],
process=true]
[27-02-2018
12:12:19][DEBUG][heartbeat-receiver-event-loop-thread][LiveListenerBus]
SparkListenerBus has already stopped! Dropping event
SparkListenerExecutorMetricsUpdate(driver,WrappedArray())
[27-02-2018 12:12:19][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713739897,
period=3000, cancel=false,
task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862],
process=true]
[27-02-2018 12:12:20][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713740230,
period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
prevCpuTime=5575,
super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80]],
process=true]
[27-02-2018 12:12:20][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=e229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713740641,
period=10000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$12@1df0989f],
process=true]
[27-02-2018 12:12:21][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713741160,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
[27-02-2018 12:12:21][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=7329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713741181,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@20c5f3ea],
process=true]
[27-02-2018 12:12:21][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=8329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713741312,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@14d3a934],
process=true]
[27-02-2018 12:12:21][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred
[obj=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask@745208a,
process=true]
[27-02-2018
12:12:21][DEBUG][grid-timeout-worker-#23][GridDeploymentLocalStore]
Deployment meta for local deployment: GridDeploymentMetadata
[depMode=SHARED,
alias=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
clsName=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
userVer=null, sndNodeId=a88e471d-c381-40b7-8665-048d5789cad6, clsLdrId=null,
clsLdr=null, participants=null, parentLdr=null, record=true,
nodeFilter=null, seqNum=n/a]
[27-02-2018
12:12:21][DEBUG][grid-timeout-worker-#23][GridDeploymentLocalStore] Acquired
deployment class from local cache: GridDeployment [ts=1519713501172,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6,
clsLdrId=6429bf5d161-a88e471d-c381-40b7-8665-048d5789cad6, userVer=0,
loc=true,
sampleClsName=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
pendingUndeploy=false, undeployed=false, usage=1]
[27-02-2018 12:12:21][DEBUG][grid-timeout-worker-#23][GridResourceProcessor]
Injecting resources
[target=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1@48d652b2]
[27-02-2018 12:12:21][DEBUG][sys-#71][GridClosureProcessor] Grid runnable
started: closure-proc-worker
[27-02-2018 12:12:21][DEBUG][sys-#71][GridClosureProcessor] Grid runnable
finished normally: closure-proc-worker
[27-02-2018 12:12:22][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713742897,
period=3000, cancel=false,
task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862],
process=true]
[27-02-2018 12:12:23][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713743167,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
[27-02-2018 12:12:23][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713743230,
period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
prevCpuTime=6177,
super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80]],
process=true]
[27-02-2018 12:12:23][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=c389bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713743242,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@58c4c5e9],
process=true]
[27-02-2018 12:12:23][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=1e69bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713743695,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7019c482],
process=true]
[27-02-2018 12:12:24][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=d379bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713744150,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7049b82b],
process=true]
[27-02-2018
12:12:24][DEBUG][exchange-worker-#42][GridCachePartitionExchangeManager]
Before waiting for exchange futures [futs[], worker=GridWorker
[name=partition-exchanger, igniteInstanceName=null, finished=false,
hashCode=569919564, interrupted=false, runner=exchange-worker-#42]]
[27-02-2018 12:12:24][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=9679bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713744645,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@9a27600],
process=true]
[27-02-2018 12:12:24][DEBUG][nio-acceptor-#29][ClientListenerProcessor]
Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
[27-02-2018 12:12:24][DEBUG][nio-acceptor-#24][TcpCommunicationSpi]
Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
[27-02-2018 12:12:25][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713745175,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
[27-02-2018 12:12:25][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713745901,
period=3000, cancel=false,
task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862],
process=true]
[27-02-2018 12:12:26][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=7329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713746187,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@20c5f3ea],
process=true]
[27-02-2018 12:12:26][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713746236,
period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
prevCpuTime=6185,
super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80]],
process=true]
[27-02-2018 12:12:26][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=8329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713746318,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@14d3a934],
process=true]
[27-02-2018 12:12:27][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=GridCommunicationMessageSet
[nodeId=d81e21d6-bd30-4626-a0e2-3cca9f4318e1, endTime=1519713747069,
timeoutId=7729bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, topic=T6
[topic=TOPIC_CACHE, id1=83e8ca36-2305-3266-8e65-1463be879baa, id2=0], plc=5,
msgs=[], reserved=false, timeout=10000, skipOnTimeout=false,
lastTs=1519713516928], process=true]
[27-02-2018 12:12:27][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713747178,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
[27-02-2018 12:12:28][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=c389bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713748246,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@58c4c5e9],
process=true]
[27-02-2018 12:12:28][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=1e69bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713748695,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7019c482],
process=true]
[27-02-2018 12:12:28][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713748906,
period=3000, cancel=false,
task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862],
process=true]
[27-02-2018 12:12:29][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=d379bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713749159,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7049b82b],
process=true]
[27-02-2018 12:12:29][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713749186,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
[27-02-2018 12:12:29][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713749238,
period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
prevCpuTime=6195,
super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80]],
process=true]
[27-02-2018
12:12:29][DEBUG][exchange-worker-#42][GridCachePartitionExchangeManager]
Before waiting for exchange futures [futs[], worker=GridWorker
[name=partition-exchanger, igniteInstanceName=null, finished=false,
hashCode=569919564, interrupted=false, runner=exchange-worker-#42]]
[27-02-2018 12:12:29][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=9679bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713749645,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@9a27600],
process=true]
[27-02-2018
12:12:29][DEBUG][heartbeat-receiver-event-loop-thread][LiveListenerBus]
SparkListenerBus has already stopped! Dropping event
SparkListenerExecutorMetricsUpdate(driver,WrappedArray())
[27-02-2018 12:12:30][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=e229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713750643,
period=10000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$12@1df0989f],
process=true]
[27-02-2018 12:12:30][DEBUG][nio-acceptor-#29][ClientListenerProcessor]
Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
[27-02-2018 12:12:30][DEBUG][nio-acceptor-#24][TcpCommunicationSpi]
Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
[27-02-2018 12:12:31][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713751188,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
[27-02-2018 12:12:31][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=7329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713751188,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@20c5f3ea],
process=true]
[27-02-2018 12:12:31][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=8329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713751319,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@14d3a934],
process=true]
[27-02-2018 12:12:31][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713751914,
period=3000, cancel=false,
task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862],
process=true]
[27-02-2018 12:12:32][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713752239,
period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
prevCpuTime=6205,
super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80]],
process=true]
[27-02-2018 12:12:33][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713753195,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
[27-02-2018 12:12:33][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=c389bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713753254,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@58c4c5e9],
process=true]
[27-02-2018 12:12:33][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=1e69bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713753701,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7019c482],
process=true]
[27-02-2018 12:12:34][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=d379bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713754168,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7049b82b],
process=true]
[27-02-2018
12:12:34][DEBUG][exchange-worker-#42][GridCachePartitionExchangeManager]
Before waiting for exchange futures [futs[], worker=GridWorker
[name=partition-exchanger, igniteInstanceName=null, finished=false,
hashCode=569919564, interrupted=false, runner=exchange-worker-#42]]
[27-02-2018 12:12:34][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=9679bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713754653,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@9a27600],
process=true]
[27-02-2018 12:12:34][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713754922,
period=3000, cancel=false,
task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862],
process=true]
[27-02-2018 12:12:35][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713755203,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
[27-02-2018 12:12:35][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713755245,
period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
prevCpuTime=6215,
super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80]],
process=true]
[27-02-2018 12:12:36][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=7329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713756195,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@20c5f3ea],
process=true]
[27-02-2018 12:12:36][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=8329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713756327,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@14d3a934],
process=true]
[27-02-2018 12:12:36][DEBUG][nio-acceptor-#29][ClientListenerProcessor]
Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
[27-02-2018 12:12:36][DEBUG][nio-acceptor-#24][TcpCommunicationSpi]
Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
[27-02-2018 12:12:37][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=GridCommunicationMessageSet
[nodeId=d81e21d6-bd30-4626-a0e2-3cca9f4318e1, endTime=1519713757075,
timeoutId=7729bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, topic=T6
[topic=TOPIC_CACHE, id1=83e8ca36-2305-3266-8e65-1463be879baa, id2=0], plc=5,
msgs=[], reserved=false, timeout=10000, skipOnTimeout=false,
lastTs=1519713516928], process=true]
[27-02-2018 12:12:37][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713757209,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
[27-02-2018 12:12:37][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713757927,
period=3000, cancel=false,
task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862],
process=true]
[27-02-2018 12:12:38][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713758249,
period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
prevCpuTime=6230,
super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80]],
process=true]
[27-02-2018 12:12:38][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=c389bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713758254,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@58c4c5e9],
process=true]
[27-02-2018 12:12:38][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=1e69bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713758708,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7019c482],
process=true]
[27-02-2018 12:12:39][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=d379bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713759176,
period=5000, cancel=false,
task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7049b82b],
process=true]
[27-02-2018 12:12:39][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
Timeout has occurred [obj=CancelableTask
[id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, endTime=1519713759216,
period=2000, cancel=false,
task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5],
process=true]
/
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Spark 'close' API call hangs within ignite service grid
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
Socket exception can be caused by wrong network configuration or firewall
configuration.
If node will not be able to send an response to other node then it can
cause grid hanging.
(Un)marshall exceptions (that is not caused by network exceptions) is a
signal that smth goes wrong.
On Fri, Mar 23, 2018 at 8:48 AM, akshaym <ak...@hashmapinc.com>
wrote:
> I have pushed the sample application to github
> <https://github.com/akshaymhetre/SparkJobAsIgniteService> . Please check
> it
> once.
>
> Also, I am able to get rid of the hang issue with spark.close API call by
> adding "igniteInstanceName" property. Not sure if its a right approach
> though.
> I came up with this solution, while debugging this issue. What I observed
> is
> that during saving dataframe to ignite, it needs Ignite context. It first
> checks if the context is already there, if it is exists it uses that
> context
> to save dataframe in ignite and on spark close API call it tries to close
> the same context.
> As I am trying to run this spark job as an ignite service, I wanted it to
> run continuously. So closing the ignite context was causing this issue. So
> to make dataframe APIs to create new context everytime, I added
> "igniteInstanceName" property to config which I am apsing to new ignite DF
> APIs.
>
> Though it resolves the hang issue it is still showing some socket
> connection
> and unmarshalling exceptions. Do I need to worry about it? How can I get
> rid
> of those?
>
> Also, Any trade-offs if we use Spark As Ignite Service when executed with
> Yarn?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: Spark 'close' API call hangs within ignite service grid
Posted by akshaym <ak...@hashmapinc.com>.
I have pushed the sample application to github
<https://github.com/akshaymhetre/SparkJobAsIgniteService> . Please check it
once.
Also, I am able to get rid of the hang issue with spark.close API call by
adding "igniteInstanceName" property. Not sure if its a right approach
though.
I came up with this solution, while debugging this issue. What I observed is
that during saving dataframe to ignite, it needs Ignite context. It first
checks if the context is already there, if it is exists it uses that context
to save dataframe in ignite and on spark close API call it tries to close
the same context.
As I am trying to run this spark job as an ignite service, I wanted it to
run continuously. So closing the ignite context was causing this issue. So
to make dataframe APIs to create new context everytime, I added
"igniteInstanceName" property to config which I am apsing to new ignite DF
APIs.
Though it resolves the hang issue it is still showing some socket connection
and unmarshalling exceptions. Do I need to worry about it? How can I get rid
of those?
Also, Any trade-offs if we use Spark As Ignite Service when executed with
Yarn?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Spark 'close' API call hangs within ignite service grid
Posted by Denis Mekhanikov <dm...@gmail.com>.
I don't see anything suspicious in the log.
Can you submit a project with a reproducer to GitHub or attach to this
thread?
Denis
ср, 28 февр. 2018 г. в 8:52, akshaym <ak...@hashmapinc.com>:
> Hi, I have updated my post. In short when I call spark.close() inside
> ignite
> service it hangs and call never returns.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Spark 'close' API call hangs within ignite service grid
Posted by akshaym <ak...@hashmapinc.com>.
Hi, I have updated my post. In short when I call spark.close() inside ignite
service it hangs and call never returns.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Spark 'close' API call hangs within ignite service grid
Posted by Denis Mekhanikov <dm...@gmail.com>.
Hi!
Please attach the log as a file. Right now it's impossible to read.
Also please specify, what exactly doesn't work on the master version. I
couldn't figure it out from the log.
Denis
вт, 27 февр. 2018 г. в 9:55, akshaym <ak...@hashmapinc.com>:
> Hi,
>
> I am running a sprak job within ignite service grid. After job is done I am
> calling close() on spark session. This is working for stable version of
> ignite i.e. 2.3.0. However, its not working with 2.5.0-SNAPSHOT version.
> Here are the detailed logs :
>
> /
> Closing spark context
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][G] Shutdown hook is removed.
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][G] Unregistered MBean:
> org.apache:clsLdr=764c12b6,group=Kernal,name=Ignition
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][IgniteKernal] Notifying
> lifecycle beans.
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
> org.spark_project.jetty.server.Server@19480852
> [27-02-2018 12:12:18][DEBUG][svc-#88][Server] doStop
> org.spark_project.jetty.server.Server@19480852
> [27-02-2018 12:12:18][DEBUG][SparkUI-161][QueuedThreadPool] ran
> SparkUI-161-acceptor-0@1e34fb5-ServerConnector
> @a75cb0a{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
> [27-02-2018 12:12:18][DEBUG][svc-#88][Server] Graceful shutdown
> org.spark_project.jetty.server.Server@19480852 by
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
> Spark@a75cb0a{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
>
> org.spark_project.jetty.server.ServerConnector$ServerConnectorManager@7313f6d3
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
> org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=0 selected=0
> [27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Stopping
> org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=0 selected=0
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridIoManager] Removed
> message listener [topic=TOPIC_DATASTREAM,
>
> lsnr=org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1@378382bf
> ]
> [27-02-2018
> 12:12:18][DEBUG][grid-data-loader-flusher-#40][DataStreamProcessor] Caught
> interrupted exception: java.lang.InterruptedException
> [27-02-2018
> 12:12:18][DEBUG][grid-data-loader-flusher-#40][DataStreamProcessor] Grid
> runnable finished due to interruption without cancellation:
> grid-data-loader-flusher
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][DataStreamProcessor] Stopped
> data streamer processor.
> [27-02-2018 12:12:18][DEBUG][session-timeout-worker-#34][GridRestProcessor]
> Caught interrupted exception: java.lang.InterruptedException: sleep
> interrupted
> [27-02-2018 12:12:18][DEBUG][session-timeout-worker-#34][GridRestProcessor]
> Grid runnable finished due to interruption without cancellation:
> session-timeout-worker
> [27-02-2018 12:12:18][DEBUG][nio-acceptor-#35][GridTcpRestProtocol]
> Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
> [27-02-2018 12:12:18][DEBUG][nio-acceptor-#35][GridTcpRestProtocol] Closing
> all listening sockets.
> [27-02-2018 12:12:18][DEBUG][nio-acceptor-#35][GridTcpRestProtocol] Closing
> NIO selector.
> [27-02-2018 12:12:18][DEBUG][nio-acceptor-#35][GridTcpRestProtocol] Grid
> runnable finished due to interruption without cancellation: nio-acceptor
> [27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Queued change
> org.spark_project.jetty.io.ManagedSelector$CloseEndPoints@28c2db1e on
> org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=0 selected=0
> [27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Selector loop
> woken up from select, 0/0 selected
> [27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Running change
> org.spark_project.jetty.io.ManagedSelector$CloseEndPoints@28c2db1e
> [27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Closing 0
> endPoints on org.spark_project.jetty.io.ManagedSelector@375067e7 id=1
> keys=0
> selected=0
> [27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Closed 0
> endPoints on org.spark_project.jetty.io.ManagedSelector@375067e7 id=1
> keys=0
> selected=0
> [27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Selector loop
> waiting on select
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol]
> Cancelling grid runnable: ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
> super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0,
> bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
> [name=grid-nio-worker-tcp-rest-0, igniteInstanceName=null, finished=false,
> hashCode=2091822083, interrupted=false,
> runner=grid-nio-worker-tcp-rest-0-#36]]]
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-0-#36][GridTcpRestProtocol]
> Closing all connected client sockets.
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol]
> Cancelling grid runnable: ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
> super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0,
> bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
> [name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null, finished=false,
> hashCode=629894509, interrupted=false,
> runner=grid-nio-worker-tcp-rest-1-#37]]]
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-0-#36][GridTcpRestProtocol]
> Closing NIO selector.
> [27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Queued change
> org.spark_project.jetty.io.ManagedSelector$CloseSelector@b08d809 on
> org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=0 selected=0
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-0-#36][GridTcpRestProtocol] Grid
> runnable finished due to cancellation: grid-nio-worker-tcp-rest-0
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol]
> Closing all connected client sockets.
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol]
> Closing NIO selector.
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-1-#37][GridTcpRestProtocol] Grid
> runnable finished due to cancellation: grid-nio-worker-tcp-rest-1
> [27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Selector loop
> woken up from select, 0/0 selected
> [27-02-2018 12:12:18][DEBUG][SparkUI-160][ManagedSelector] Running change
> org.spark_project.jetty.io.ManagedSelector$CloseSelector@b08d809
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol]
> Cancelling grid runnable: ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
> super=AbstractNioClientWorker [idx=2, bytesRcvd=0, bytesSent=0,
> bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
> [name=grid-nio-worker-tcp-rest-2, igniteInstanceName=null, finished=false,
> hashCode=266454991, interrupted=false,
> runner=grid-nio-worker-tcp-rest-2-#38]]]
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-2-#38][GridTcpRestProtocol]
> Closing all connected client sockets.
> [27-02-2018 12:12:18][DEBUG][SparkUI-160][ExecuteProduceConsume] EPC
> Prod/org.spark_project.jetty.io.ManagedSelector$SelectorProducer@2a26278b
> produced null
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol]
> Cancelling grid runnable: ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
> super=AbstractNioClientWorker [idx=3, bytesRcvd=0, bytesSent=0,
> bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
> [name=grid-nio-worker-tcp-rest-3, igniteInstanceName=null, finished=false,
> hashCode=1884453621, interrupted=false,
> runner=grid-nio-worker-tcp-rest-3-#39]]]
> [27-02-2018 12:12:18][DEBUG][SparkUI-160][ExecuteProduceConsume] EPC
> Idle/org.spark_project.jetty.io.ManagedSelector$SelectorProducer@2a26278b
> produce exit
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol] Joining
> grid runnable: ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
> super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0,
> bytesRcvd0=0, bytesSent0=0, select=false, super=GridWorker
> [name=grid-nio-worker-tcp-rest-0, igniteInstanceName=null, finished=true,
> hashCode=2091822083, interrupted=unknown, runner=null]]]
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-2-#38][GridTcpRestProtocol]
> Closing NIO selector.
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol] Joining
> grid runnable: ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
> super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0,
> bytesRcvd0=0, bytesSent0=0, select=false, super=GridWorker
> [name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null, finished=true,
> hashCode=629894509, interrupted=unknown, runner=null]]]
> [27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Stopped
> org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=-1
> selected=-1
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol] Joining
> grid runnable: ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
> super=AbstractNioClientWorker [idx=2, bytesRcvd=0, bytesSent=0,
> bytesRcvd0=0, bytesSent0=0, select=false, super=GridWorker
> [name=grid-nio-worker-tcp-rest-2, igniteInstanceName=null, finished=true,
> hashCode=266454991, interrupted=true,
> runner=grid-nio-worker-tcp-rest-2-#38]]]
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
> org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=-1
> selected=-1
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTcpRestProtocol] Joining
> grid runnable: ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192],
> super=AbstractNioClientWorker [idx=3, bytesRcvd=0, bytesSent=0,
> bytesRcvd0=0, bytesSent0=0, select=false, super=GridWorker
> [name=grid-nio-worker-tcp-rest-3, igniteInstanceName=null, finished=false,
> hashCode=1884453621, interrupted=true,
> runner=grid-nio-worker-tcp-rest-3-#39]]]
> [27-02-2018 12:12:18][DEBUG][SparkUI-160][QueuedThreadPool] ran
> org.spark_project.jetty.io.ManagedSelector@375067e7 id=1 keys=-1
> selected=-1
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-3-#39][GridTcpRestProtocol]
> Closing all connected client sockets.
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-3-#39][GridTcpRestProtocol]
> Closing NIO selector.
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
> org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=0 selected=0
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-3-#39][GridTcpRestProtocol] Grid
> runnable finished due to cancellation: grid-nio-worker-tcp-rest-3
> [27-02-2018 12:12:18][INFO ][SparkListenerBus][GridTcpRestProtocol] Command
> protocol successfully stopped: TCP binary
> [27-02-2018
> 12:12:18][DEBUG][grid-nio-worker-tcp-rest-2-#38][GridTcpRestProtocol] Grid
> runnable finished due to cancellation: grid-nio-worker-tcp-rest-2
> [27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Stopping
> org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=0 selected=0
> [27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Queued change
> org.spark_project.jetty.io.ManagedSelector$CloseEndPoints@66931c58 on
> org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=0 selected=0
> [27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Selector loop
> woken up from select, 0/0 selected
> [27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Running change
> org.spark_project.jetty.io.ManagedSelector$CloseEndPoints@66931c58
> [27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Closing 0
> endPoints on org.spark_project.jetty.io.ManagedSelector@35acf85d id=0
> keys=0
> selected=0
> [27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Closed 0
> endPoints on org.spark_project.jetty.io.ManagedSelector@35acf85d id=0
> keys=0
> selected=0
> [27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Selector loop
> waiting on select
> [27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Queued change
> org.spark_project.jetty.io.ManagedSelector$CloseSelector@1e8d41a on
> org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=0 selected=0
> [27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Selector loop
> woken up from select, 0/0 selected
> [27-02-2018 12:12:18][DEBUG][SparkUI-159][ManagedSelector] Running change
> org.spark_project.jetty.io.ManagedSelector$CloseSelector@1e8d41a
> [27-02-2018 12:12:18][DEBUG][SparkUI-159][ExecuteProduceConsume] EPC
> Prod/org.spark_project.jetty.io.ManagedSelector$SelectorProducer@7c6b8701
> produced null
> [27-02-2018 12:12:18][DEBUG][SparkUI-159][ExecuteProduceConsume] EPC
> Idle/org.spark_project.jetty.io.ManagedSelector$SelectorProducer@7c6b8701
> produce exit
> [27-02-2018 12:12:18][DEBUG][svc-#88][ManagedSelector] Stopped
> org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=-1
> selected=-1
> [27-02-2018 12:12:18][DEBUG][SparkUI-159][QueuedThreadPool] ran
> org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=-1
> selected=-1
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
> org.spark_project.jetty.io.ManagedSelector@35acf85d id=0 keys=-1
> selected=-1
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
>
> org.spark_project.jetty.server.ServerConnector$ServerConnectorManager@7313f6d3
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
> HttpConnectionFactory@181ade16[HTTP/1.1]
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
> HttpConnectionFactory@181ade16[HTTP/1.1]
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
> org.spark_project.jetty.util.thread.ScheduledExecutorScheduler@4ced9cbe
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
> org.spark_project.jetty.util.thread.ScheduledExecutorScheduler@4ced9cbe
> [27-02-2018 12:12:18][INFO ][svc-#88][AbstractConnector] Stopped
> Spark@a75cb0a{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
> Spark@a75cb0a{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractHandler] stopping
> org.spark_project.jetty.server.Server@19480852
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
> org.spark_project.jetty.server.handler.ContextHandlerCollection@31522a7a
> [org.spark_project.jetty.server.handler.gzip.GzipHandler@61de4808,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@55fef7fb,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@3c790dd4,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@1cd8e1db,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@f557440,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2db61b7e,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@59d5b4b1,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2d84069,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@268bc306,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@3bfb5142,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@4fff0ab0,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@fe4c02e,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@778fa127,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@169f8fd5,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2102bd3c,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@8cefc82,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2c2f225,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@6849348f,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@5c168fe7,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@23c3520b,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@3685df8c,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@1749789f,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@7f48a426,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@4622f2ce,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@5e433c39,
> o.s.j.s.ServletContextHandler@5a643879
> {/metrics/json,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@6f8e11a2{/SQL,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@299f3860{/SQL/json,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@45005fd3
> {/SQL/execution,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@6dfd7426
> {/SQL/execution/json,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@10407136{/static/sql,null,SHUTDOWN,@Spark}]
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractHandler] stopping
> org.spark_project.jetty.server.handler.ContextHandlerCollection@31522a7a
> [org.spark_project.jetty.server.handler.gzip.GzipHandler@61de4808,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@55fef7fb,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@3c790dd4,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@1cd8e1db,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@f557440,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2db61b7e,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@59d5b4b1,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2d84069,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@268bc306,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@3bfb5142,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@4fff0ab0,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@fe4c02e,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@778fa127,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@169f8fd5,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2102bd3c,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@8cefc82,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2c2f225,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@6849348f,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@5c168fe7,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@23c3520b,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@3685df8c,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@1749789f,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@7f48a426,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@4622f2ce,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@5e433c39,
> o.s.j.s.ServletContextHandler@5a643879
> {/metrics/json,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@6f8e11a2{/SQL,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@299f3860{/SQL/json,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@45005fd3
> {/SQL/execution,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@6dfd7426
> {/SQL/execution/json,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@10407136{/static/sql,null,SHUTDOWN,@Spark}]
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
> org.spark_project.jetty.server.handler.ContextHandlerCollection@31522a7a
> [org.spark_project.jetty.server.handler.gzip.GzipHandler@61de4808,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@55fef7fb,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@3c790dd4,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@1cd8e1db,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@f557440,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2db61b7e,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@59d5b4b1,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2d84069,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@268bc306,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@3bfb5142,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@4fff0ab0,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@fe4c02e,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@778fa127,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@169f8fd5,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2102bd3c,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@8cefc82,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@2c2f225,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@6849348f,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@5c168fe7,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@23c3520b,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@3685df8c,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@1749789f,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@7f48a426,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@4622f2ce,
> org.spark_project.jetty.server.handler.gzip.GzipHandler@5e433c39,
> o.s.j.s.ServletContextHandler@5a643879
> {/metrics/json,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@6f8e11a2{/SQL,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@299f3860{/SQL/json,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@45005fd3
> {/SQL/execution,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@6dfd7426
> {/SQL/execution/json,null,SHUTDOWN,@Spark},
> o.s.j.s.ServletContextHandler@10407136{/static/sql,null,SHUTDOWN,@Spark}]
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
> org.spark_project.jetty.server.handler.ErrorHandler@1ed1eb6e
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractHandler] stopping
> org.spark_project.jetty.server.handler.ErrorHandler@1ed1eb6e
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
> org.spark_project.jetty.server.handler.ErrorHandler@1ed1eb6e
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] stopping
> SparkUI{STARTED,8<=8<=200,i=8,q=0}
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
> SparkUI{STOPPED,8<=8<=200,i=0,q=0}
> [27-02-2018 12:12:18][DEBUG][svc-#88][AbstractLifeCycle] STOPPED
> org.spark_project.jetty.server.Server@19480852
> [27-02-2018 12:12:18][INFO ][svc-#88][SparkUI] Stopped Spark web UI at
> http://192.168.1.98:4040
> [27-02-2018 12:12:18][INFO ][SparkListenerBus][GridJettyRestProtocol]
> Command protocol successfully stopped: Jetty REST
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridRestProcessor] REST
> processor stopped.
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridIoManager] Removed
> message listener [topic=TOPIC_JOB_SIBLINGS,
>
> lsnr=org.apache.ignite.internal.processors.task.GridTaskProcessor$JobSiblingsMessageListener@32f763ec
> ]
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridIoManager] Removed
> message listener [topic=TOPIC_TASK_CANCEL,
>
> lsnr=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskCancelMessageListener@6ea394a1
> ]
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridTaskProcessor] Finished
> executing task processor onKernalStop() callback.
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridIoManager] Removed
> message listener [topic=TOPIC_JOB,
>
> lsnr=org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener@16cae9c4
> ]
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridIoManager] Removed
> message listener [topic=TOPIC_JOB_CANCEL,
>
> lsnr=org.apache.ignite.internal.processors.job.GridJobProcessor$JobCancelListener@1114a9fb
> ]
> [27-02-2018 12:12:18][DEBUG][SparkListenerBus][GridJobWorker] Joining grid
> runnable: GridJobWorker [createTime=1519713725262, startTime=1519713725292,
> finishTime=0, taskNode=TcpDiscoveryNode
> [id=1b035a31-cd11-4a73-a753-8f30824ed940, addrs=[0:0:0:0:0:0:0:1%lo,
> 127.0.0.1, 172.17.0.1, 192.168.1.98], sockAddrs=[/172.17.0.1:0,
> /0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /192.168.1.98:0], discPort=0,
> order=3,
> intOrder=3, lastExchangeTime=1519713675143, loc=false,
> ver=2.5.0#20180213-sha1:c6619637, isClient=true], internal=false,
> marsh=org.apache.ignite.internal.binary.BinaryMarshaller@7dc19a70,
> ses=GridJobSessionImpl [ses=GridTaskSessionImpl
>
> [taskName=org.apache.ignite.internal.processors.service.GridServiceProxy$ServiceProxyCallable,
> dep=GridDeployment [ts=1519713501172, depMode=SHARED,
> clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6,
> clsLdrId=6429bf5d161-a88e471d-c381-40b7-8665-048d5789cad6, userVer=0,
> loc=true,
>
> sampleClsName=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
> pendingUndeploy=false, undeployed=false, usage=1],
>
> taskClsName=org.apache.ignite.internal.processors.service.GridServiceProxy$ServiceProxyCallable,
> sesId=fb35ef5d161-1b035a31-cd11-4a73-a753-8f30824ed940,
> startTime=1519713725229, endTime=9223372036854775807,
> taskNodeId=1b035a31-cd11-4a73-a753-8f30824ed940,
> clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, closed=false,
> cpSpi=null,
> failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
> topPred=null, subjId=1b035a31-cd11-4a73-a753-8f30824ed940,
> mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
> state=INIT, res=null, hash=531343034]], execName=null],
> jobId=0c35ef5d161-1b035a31-cd11-4a73-a753-8f30824ed940],
> jobCtx=GridJobContextImpl
> [jobId=0c35ef5d161-1b035a31-cd11-4a73-a753-8f30824ed940, timeoutObj=null,
> attrs={}], dep=GridDeployment [ts=1519713501172, depMode=SHARED,
> clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6,
> clsLdrId=6429bf5d161-a88e471d-c381-40b7-8665-048d5789cad6, userVer=0,
> loc=true,
>
> sampleClsName=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
> pendingUndeploy=false, undeployed=false, usage=1], finishing=false,
> masterLeaveGuard=false, timedOut=false, sysCancelled=false,
> sysStopping=false, isStarted=true, job=C2 [c=ServiceProxyCallable
> [mtdName=run, svcName=jdbcReaderService, ignite=IgniteKernal
> [cfg=IgniteConfiguration [igniteInstanceName=null, pubPoolSize=8,
> svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8,
> mgmtPoolSize=4, igfsPoolSize=4, dataStreamerPoolSize=8,
> utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2,
> qryPoolSize=8, igniteHome=/home/akshay/Downloads/igniteSource/ignite,
> igniteWorkDir=/home/akshay/Downloads/igniteSource/ignite/work,
> mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e,
> nodeId=a88e471d-c381-40b7-8665-048d5789cad6,
> marsh=org.apache.ignite.internal.binary.BinaryMarshaller@7dc19a70,
> marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000,
> sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000,
> metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
> discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000,
> ackTimeout=5000,
> marsh=JdkMarshaller
> [clsFilter=org.apache.ignite.internal.IgniteKernal$5@44bc3a8e],
> reconCnt=10,
> reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false,
> clientReconnectDisabled=false], segPlc=STOP, segResolveAttempts=2,
> waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000,
> commSpi=TcpCommunicationSpi
>
> [connectGate=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$ConnectGateway@dd71cc5
> ,
>
> connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$6@4149c59c
> ,
> enableForcibleNodeKill=false, enableTroubleshootingLog=false,
>
> srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@2d29b4ee
> ,
> locAddr=null, locHost=0.0.0.0/0.0.0.0, locPort=47100, locPortRange=100,
> shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000,
> connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768,
> sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0,
> nioSrvr=GridNioServer [selectorSpins=0,
> filterChain=FilterChain[filters=[GridNioCodecFilter
> [parser=org.apache.ignite.internal.util.nio.GridDirectParser@38e7ba9e,
> directMode=true], GridConnectionBytesVerifyFilter], closed=false,
> directBuf=true, tcpNoDelay=true, sockSndBuf=32768, sockRcvBuf=32768,
> writeTimeout=2000, idleTimeout=600000, skipWrite=false, skipRead=false,
> locAddr=0.0.0.0/0.0.0.0:47100, order=LITTLE_ENDIAN, sndQueueLimit=0,
> directMode=true,
>
> metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@6e2aa843
> ,
> sslFilter=null, msgQueueLsnr=null, readerMoveCnt=0, writerMoveCnt=0,
> readWriteSelectorsAssign=false], shmemSrv=null, usePairedConnections=false,
> connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false,
> ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000,
>
> lsnr=org.apache.ignite.internal.managers.communication.GridIoManager$2@378dfd3b
> ,
> boundTcpPort=47100, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0,
> addrRslvr=null,
> ctxInitLatch=java.util.concurrent.CountDownLatch@69ee81fc[Count = 0],
> stopping=false,
>
> metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@6e2aa843
> ],
> evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@6f36c2f0,
> colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi
>
> [lsnr=org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore$LocalDeploymentListener@6b3784c1
> ],
> indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@3b35a229,
> addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=1,
> txCfg=org.apache.ignite.configuration.TransactionConfiguration@9816741,
> cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED,
> p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100,
> timeSrvPortRange=100, failureDetectionTimeout=10000,
> clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null,
>
> connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@1e16c0aa
> ,
> odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration
> [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null,
> grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null,
> binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration
> [sysRegionInitSize=41943040, sysCacheMaxSize=104857600, pageSize=4096,
> concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default,
> maxSize=3340354355, initSize=268435456, swapPath=null,
> pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100,
> metricsEnabled=false, metricsSubIntervalCount=5,
> metricsRateTimeInterval=60000, persistenceEnabled=false,
> checkpointPageBufSize=0], storagePath=null, checkpointFreq=180000,
> lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL,
> walHistSize=20, walSegments=10, walSegmentSize=67108864, walPath=db/wal,
> walArchivePath=db/wal/archive, metricsEnabled=false, walMode=DEFAULT,
> walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000,
> walRecordIterBuffSize=67108864, alwaysWriteFullPages=false,
>
> fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@2f67a4d3
> ,
> metricsSubIntervalCnt=5, metricsRateTimeInterval=60000,
> walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false,
> walCompactionEnabled=false], activeOnStart=true, autoActivation=true,
> longQryWarnTimeout=3000, sqlConnCfg=null,
> cliConnCfg=ClientConnectorConfiguration [host=null, port=10800,
> portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true,
> maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0,
> jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false,
> useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null]],
> igniteInstanceName=null, startTime=1519713491243,
>
> rsrcCtx=org.apache.ignite.internal.processors.resource.GridSpringResourceContextImpl@7b32aa52
> ,
>
> reconnectState=org.apache.ignite.internal.IgniteKernal$ReconnectState@7e04cce0
> ]]],
> held=0, partsReservation=null, reqTopVer=null, execName=null]
> [27-02-2018 12:12:18][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=1e69bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713738695,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7019c482
> ],
> process=true]
> [27-02-2018 12:12:18][DEBUG][nio-acceptor-#29][ClientListenerProcessor]
> Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
> [27-02-2018 12:12:18][DEBUG][nio-acceptor-#24][TcpCommunicationSpi]
> Balancing data [min0=0, minIdx=2, max0=-1, maxIdx=-1]
> [27-02-2018 12:12:19][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=d379bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713739141,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7049b82b
> ],
> process=true]
> [27-02-2018 12:12:19][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713739160,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
> [27-02-2018
> 12:12:19][DEBUG][exchange-worker-#42][GridCachePartitionExchangeManager]
> Before waiting for exchange futures [futs[], worker=GridWorker
> [name=partition-exchanger, igniteInstanceName=null, finished=false,
> hashCode=569919564, interrupted=false, runner=exchange-worker-#42]]
> [27-02-2018 12:12:19][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=9679bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713739639,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@9a27600
> ],
> process=true]
> [27-02-2018
> 12:12:19][DEBUG][heartbeat-receiver-event-loop-thread][LiveListenerBus]
> SparkListenerBus has already stopped! Dropping event
> SparkListenerExecutorMetricsUpdate(driver,WrappedArray())
> [27-02-2018 12:12:19][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713739897,
> period=3000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862
> ],
> process=true]
> [27-02-2018 12:12:20][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713740230,
> period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
> prevCpuTime=5575,
>
> super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80
> ]],
> process=true]
> [27-02-2018 12:12:20][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=e229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713740641,
> period=10000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$12@1df0989f
> ],
> process=true]
> [27-02-2018 12:12:21][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713741160,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
> [27-02-2018 12:12:21][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=7329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713741181,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@20c5f3ea
> ],
> process=true]
> [27-02-2018 12:12:21][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=8329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713741312,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@14d3a934
> ],
> process=true]
> [27-02-2018 12:12:21][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred
>
> [obj=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask@745208a
> ,
> process=true]
> [27-02-2018
> 12:12:21][DEBUG][grid-timeout-worker-#23][GridDeploymentLocalStore]
> Deployment meta for local deployment: GridDeploymentMetadata
> [depMode=SHARED,
>
> alias=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
>
> clsName=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
> userVer=null, sndNodeId=a88e471d-c381-40b7-8665-048d5789cad6,
> clsLdrId=null,
> clsLdr=null, participants=null, parentLdr=null, record=true,
> nodeFilter=null, seqNum=n/a]
> [27-02-2018
> 12:12:21][DEBUG][grid-timeout-worker-#23][GridDeploymentLocalStore]
> Acquired
> deployment class from local cache: GridDeployment [ts=1519713501172,
> depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6,
> clsLdrId=6429bf5d161-a88e471d-c381-40b7-8665-048d5789cad6, userVer=0,
> loc=true,
>
> sampleClsName=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
> pendingUndeploy=false, undeployed=false, usage=1]
> [27-02-2018
> 12:12:21][DEBUG][grid-timeout-worker-#23][GridResourceProcessor]
> Injecting resources
>
> [target=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1@48d652b2
> ]
> [27-02-2018 12:12:21][DEBUG][sys-#71][GridClosureProcessor] Grid runnable
> started: closure-proc-worker
> [27-02-2018 12:12:21][DEBUG][sys-#71][GridClosureProcessor] Grid runnable
> finished normally: closure-proc-worker
> [27-02-2018 12:12:22][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713742897,
> period=3000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862
> ],
> process=true]
> [27-02-2018 12:12:23][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713743167,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
> [27-02-2018 12:12:23][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713743230,
> period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
> prevCpuTime=6177,
>
> super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80
> ]],
> process=true]
> [27-02-2018 12:12:23][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=c389bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713743242,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@58c4c5e9
> ],
> process=true]
> [27-02-2018 12:12:23][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=1e69bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713743695,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7019c482
> ],
> process=true]
> [27-02-2018 12:12:24][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=d379bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713744150,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7049b82b
> ],
> process=true]
> [27-02-2018
> 12:12:24][DEBUG][exchange-worker-#42][GridCachePartitionExchangeManager]
> Before waiting for exchange futures [futs[], worker=GridWorker
> [name=partition-exchanger, igniteInstanceName=null, finished=false,
> hashCode=569919564, interrupted=false, runner=exchange-worker-#42]]
> [27-02-2018 12:12:24][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=9679bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713744645,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@9a27600
> ],
> process=true]
> [27-02-2018 12:12:24][DEBUG][nio-acceptor-#29][ClientListenerProcessor]
> Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
> [27-02-2018 12:12:24][DEBUG][nio-acceptor-#24][TcpCommunicationSpi]
> Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
> [27-02-2018 12:12:25][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713745175,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
> [27-02-2018 12:12:25][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713745901,
> period=3000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862
> ],
> process=true]
> [27-02-2018 12:12:26][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=7329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713746187,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@20c5f3ea
> ],
> process=true]
> [27-02-2018 12:12:26][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713746236,
> period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
> prevCpuTime=6185,
>
> super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80
> ]],
> process=true]
> [27-02-2018 12:12:26][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=8329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713746318,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@14d3a934
> ],
> process=true]
> [27-02-2018 12:12:27][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=GridCommunicationMessageSet
> [nodeId=d81e21d6-bd30-4626-a0e2-3cca9f4318e1, endTime=1519713747069,
> timeoutId=7729bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, topic=T6
> [topic=TOPIC_CACHE, id1=83e8ca36-2305-3266-8e65-1463be879baa, id2=0],
> plc=5,
> msgs=[], reserved=false, timeout=10000, skipOnTimeout=false,
> lastTs=1519713516928], process=true]
> [27-02-2018 12:12:27][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713747178,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
> [27-02-2018 12:12:28][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=c389bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713748246,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@58c4c5e9
> ],
> process=true]
> [27-02-2018 12:12:28][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=1e69bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713748695,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7019c482
> ],
> process=true]
> [27-02-2018 12:12:28][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713748906,
> period=3000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862
> ],
> process=true]
> [27-02-2018 12:12:29][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=d379bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713749159,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7049b82b
> ],
> process=true]
> [27-02-2018 12:12:29][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713749186,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
> [27-02-2018 12:12:29][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713749238,
> period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
> prevCpuTime=6195,
>
> super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80
> ]],
> process=true]
> [27-02-2018
> 12:12:29][DEBUG][exchange-worker-#42][GridCachePartitionExchangeManager]
> Before waiting for exchange futures [futs[], worker=GridWorker
> [name=partition-exchanger, igniteInstanceName=null, finished=false,
> hashCode=569919564, interrupted=false, runner=exchange-worker-#42]]
> [27-02-2018 12:12:29][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=9679bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713749645,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@9a27600
> ],
> process=true]
> [27-02-2018
> 12:12:29][DEBUG][heartbeat-receiver-event-loop-thread][LiveListenerBus]
> SparkListenerBus has already stopped! Dropping event
> SparkListenerExecutorMetricsUpdate(driver,WrappedArray())
> [27-02-2018 12:12:30][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=e229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713750643,
> period=10000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$12@1df0989f
> ],
> process=true]
> [27-02-2018 12:12:30][DEBUG][nio-acceptor-#29][ClientListenerProcessor]
> Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
> [27-02-2018 12:12:30][DEBUG][nio-acceptor-#24][TcpCommunicationSpi]
> Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
> [27-02-2018 12:12:31][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713751188,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
> [27-02-2018 12:12:31][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=7329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713751188,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@20c5f3ea
> ],
> process=true]
> [27-02-2018 12:12:31][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=8329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713751319,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@14d3a934
> ],
> process=true]
> [27-02-2018 12:12:31][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713751914,
> period=3000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862
> ],
> process=true]
> [27-02-2018 12:12:32][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713752239,
> period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
> prevCpuTime=6205,
>
> super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80
> ]],
> process=true]
> [27-02-2018 12:12:33][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713753195,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
> [27-02-2018 12:12:33][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=c389bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713753254,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@58c4c5e9
> ],
> process=true]
> [27-02-2018 12:12:33][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=1e69bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713753701,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7019c482
> ],
> process=true]
> [27-02-2018 12:12:34][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=d379bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713754168,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7049b82b
> ],
> process=true]
> [27-02-2018
> 12:12:34][DEBUG][exchange-worker-#42][GridCachePartitionExchangeManager]
> Before waiting for exchange futures [futs[], worker=GridWorker
> [name=partition-exchanger, igniteInstanceName=null, finished=false,
> hashCode=569919564, interrupted=false, runner=exchange-worker-#42]]
> [27-02-2018 12:12:34][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=9679bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713754653,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@9a27600
> ],
> process=true]
> [27-02-2018 12:12:34][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713754922,
> period=3000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862
> ],
> process=true]
> [27-02-2018 12:12:35][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713755203,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
> [27-02-2018 12:12:35][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713755245,
> period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
> prevCpuTime=6215,
>
> super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80
> ]],
> process=true]
> [27-02-2018 12:12:36][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=7329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713756195,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@20c5f3ea
> ],
> process=true]
> [27-02-2018 12:12:36][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=8329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713756327,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@14d3a934
> ],
> process=true]
> [27-02-2018 12:12:36][DEBUG][nio-acceptor-#29][ClientListenerProcessor]
> Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
> [27-02-2018 12:12:36][DEBUG][nio-acceptor-#24][TcpCommunicationSpi]
> Balancing data [min0=0, minIdx=0, max0=-1, maxIdx=-1]
> [27-02-2018 12:12:37][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=GridCommunicationMessageSet
> [nodeId=d81e21d6-bd30-4626-a0e2-3cca9f4318e1, endTime=1519713757075,
> timeoutId=7729bf5d161-0dab547c-f627-4116-844e-27bcf79977ff, topic=T6
> [topic=TOPIC_CACHE, id1=83e8ca36-2305-3266-8e65-1463be879baa, id2=0],
> plc=5,
> msgs=[], reserved=false, timeout=10000, skipOnTimeout=false,
> lastTs=1519713516928], process=true]
> [27-02-2018 12:12:37][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713757209,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
> [27-02-2018 12:12:37][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=0329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713757927,
> period=3000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@2cae0862
> ],
> process=true]
> [27-02-2018 12:12:38][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=2329bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713758249,
> period=3000, cancel=false, task=MetricsUpdater [prevGcTime=81,
> prevCpuTime=6230,
>
> super=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$MetricsUpdater@4b828e80
> ]],
> process=true]
> [27-02-2018 12:12:38][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=c389bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713758254,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@58c4c5e9
> ],
> process=true]
> [27-02-2018 12:12:38][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=1e69bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713758708,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7019c482
> ],
> process=true]
> [27-02-2018 12:12:39][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=d379bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713759176,
> period=5000, cancel=false,
>
> task=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@7049b82b
> ],
> process=true]
> [27-02-2018 12:12:39][DEBUG][grid-timeout-worker-#23][GridTimeoutProcessor]
> Timeout has occurred [obj=CancelableTask
> [id=f229bf5d161-0dab547c-f627-4116-844e-27bcf79977ff,
> endTime=1519713759216,
> period=2000, cancel=false,
>
> task=org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$13@307c8da5
> ],
> process=true]
>
>
>
> /
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>