You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Biren <bi...@servicenow.com> on 2017/06/28 23:33:12 UTC
Server node fails to start with exception
IllegalThreadStateException
Hi,
I am trying to embed Ignite server node into my application. I want to
integration ignite such that i can start/stop ignite node at anytime while
application is running. For that i have application level APIs to start or
stop the ignite node. For most part the ignite node start and stops without
any issue. But at time the application/ignite gets into state where It can
not start the ignite node anymore.
For my testing i am running multiple instances of the application on my
machine. The goal is to have multiple instances of application running with
ignite. That will form a cluster where work will be distributed.
Start API:
a. Initialize an instance of IgniteConfiguration. Things that i set in
configuration are instance name, static IP based discovery and few user
attributed
b. Start the node with Ingnition.getOrStart(config)
c. Instance of config and node are store in the application. Both instances
are singleton instance
Stop API:
a. Stop the ignite node by calling igniteInstance.close or
Ignition.stop(instanceName, true)
b. Set config to null
I get below error after i start/stop the node multiple times while
application is running.
java.lang.IllegalThreadStateException
2017/06/28 15:25:49 | at
java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867)
2017/06/28 15:25:49 | at java.lang.Thread.init(Thread.java:402)
2017/06/28 15:25:49 | at java.lang.Thread.init(Thread.java:349)
2017/06/28 15:25:49 | at java.lang.Thread.<init>(Thread.java:596)
2017/06/28 15:25:49 | at
org.apache.ignite.thread.IgniteThread.<init>(IgniteThread.java:100)
2017/06/28 15:25:49 | at
org.apache.ignite.thread.IgniteThread.<init>(IgniteThread.java:85)
2017/06/28 15:25:49 | at
org.apache.ignite.internal.util.StripedExecutor$Stripe.start(StripedExecutor.java:438)
2017/06/28 15:25:49 | at
org.apache.ignite.internal.util.StripedExecutor.<init>(StripedExecutor.java:84)
2017/06/28 15:25:49 | at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1729)
2017/06/28 15:25:49 | at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1647)
2017/06/28 15:25:49 | at
org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1075)
2017/06/28 15:25:49 | at
org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:595)
2017/06/28 15:25:49 | at
org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:535)
2017/06/28 15:25:49 | at
org.apache.ignite.Ignition.getOrStart(Ignition.java:414)
Looks like the node/grid is not shutting down correctly. Few threads are
left behind and when i try to start the node again, it errors out. When i
have multiple instances of application running, this error occurs randomly
on any instance. Not on all instances at the same time. Error occurs even
when i have single instance.
I'm not sure if there is anything wrong with config or they way i start/stop
the node. Is there any other API to stop the node?
Thanks!
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Server-node-fails-to-start-with-exception-IllegalThreadStateException-tp14150.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Server node fails to start with exception
IllegalThreadStateException
Posted by Biren <bi...@servicenow.com>.
Well when i look at active threads after i stop the node, i don't see any
active thread which was started by ignite. But i do see threads started by
Ignite in finished threads list on yourkit. I think that is fine.
When i try to start again, i get the IllegalThreadStateException exception.
Here is the thread dump after the exception.
2017-06-29 17:22:28
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.66-b17 mixed mode):
"Worker-Interactive:Idle" #114 daemon prio=10 os_prio=31
tid=0x00007fcb18bff800 nid=0x9707 waiting on condition [0x0000000123357000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a075b1f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Worker-Interactive:Idle" #112 daemon prio=10 os_prio=31
tid=0x00007fcb1906f000 nid=0x9607 waiting on condition [0x0000000123ff4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a075b1f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"YJPAgent-OOMESnapshotDetector" #110 daemon prio=10 os_prio=31
tid=0x00007fcb19012800 nid=0x5f0b runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"YJPAgent-CPUSampler" #109 daemon prio=10 os_prio=31 tid=0x00007fcb16cd4800
nid=0x9007 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"YJPAgent-RequestListener" #108 daemon prio=1 os_prio=31
tid=0x00007fcb19979800 nid=0x5e0b runnable [0x00000001286d5000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at com.yourkit.runtime.Core$4.run(Core.java:753)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"YJPAgent-Telemetry" #107 daemon prio=5 os_prio=31 tid=0x00007fcb16cda800
nid=0x920b in Object.wait() [0x0000000125137000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.yourkit.runtime.TelemetryThread.run(TelemetryThread.java:627)
- locked <0x00000007a3f58000> (a java.lang.Object)
Locked ownable synchronizers:
- None
"Attach Listener" #106 daemon prio=9 os_prio=31 tid=0x00007fcb19083000
nid=0xcc07 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Worker-Interactive:Idle" #105 daemon prio=10 os_prio=31
tid=0x00007fcb18806000 nid=0xce07 waiting on condition [0x000000012322c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a075b1f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:173)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"Worker-Interactive:Idle" #103 daemon prio=10 os_prio=31
tid=0x00007fcb16937000 nid=0x610b waiting on condition [0x00000001205ee000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a075b1f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:173)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"Worker-Interactive:Idle" #101 daemon prio=10 os_prio=31
tid=0x00007fcb198c9800 nid=0xd803 waiting on condition [0x0000000139823000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a075b1f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:173)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- <0x00000007a38a4198> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x00000007a3c1fb50> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
"Worker-Interactive:Idle" #52 daemon prio=10 os_prio=31
tid=0x00007fcb1800a000 nid=0x400f waiting on condition [0x0000000122746000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a075b1f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:173)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"IdleConnectionMonitor.5" #50 daemon prio=5 os_prio=31
tid=0x00007fcb1544d000 nid=0x8b03 in Object.wait() [0x0000000122abf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007a2233138> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Collector" #49 daemon prio=5 os_prio=31 tid=0x00007fcb144a0000 nid=0x8903
in Object.wait() [0x0000000121427000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007a22335f8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"StatusMonitor.600" #48 daemon prio=5 os_prio=31 tid=0x00007fcb159f7000
nid=0x8703 in Object.wait() [0x000000012437c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007a07da550> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"LogStatusMonitor.60" #47 daemon prio=5 os_prio=31 tid=0x00007fcb1880b800
nid=0x8503 in Object.wait() [0x0000000123ef1000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007a07da198> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"ECCSender.1" #46 daemon prio=5 os_prio=31 tid=0x00007fcb14382800 nid=0x8303
in Object.wait() [0x0000000124279000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007a07d9278> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"RepeatingProbeTimer" #45 daemon prio=5 os_prio=31 tid=0x00007fcb146ef800
nid=0x8103 in Object.wait() [0x0000000124176000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000007a07d8ee8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"ECCQueueMonitor.40" #44 daemon prio=5 os_prio=31 tid=0x00007fcb16407000
nid=0x7f03 in Object.wait() [0x0000000123cdc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007a07dba20> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"RefreshMonitor.65" #43 daemon prio=5 os_prio=31 tid=0x00007fcb137f7800
nid=0x7d03 in Object.wait() [0x0000000123bd9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007a07db5e0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"pool-6-thread-1" #42 prio=5 os_prio=31 tid=0x00007fcb15386800 nid=0x6d17
waiting on condition [0x0000000121c2b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a07ecba8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"HttpClient-40" #40 daemon prio=5 os_prio=31 tid=0x00007fcb160c0800
nid=0x7903 waiting on condition [0x0000000122643000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a07dc3d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"HttpClient-39" #39 daemon prio=5 os_prio=31 tid=0x00007fcb12a64800
nid=0x7703 waiting on condition [0x0000000122540000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a07dc3d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"HttpClient-38" #38 daemon prio=5 os_prio=31 tid=0x00007fcb133c2000
nid=0x7503 waiting on condition [0x000000012243d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a07dc3d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"HttpClient-37" #37 daemon prio=5 os_prio=31 tid=0x00007fcb13033800
nid=0x7303 waiting on condition [0x000000012233a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a07dc3d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"HttpClient-36" #36 daemon prio=5 os_prio=31 tid=0x00007fcb140a8800
nid=0x7103 waiting on condition [0x0000000122237000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a07dc3d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"HttpClient-35" #35 daemon prio=5 os_prio=31 tid=0x00007fcb160bf800
nid=0x6f03 waiting on condition [0x0000000122134000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a07dc3d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"HttpClient-34 Selector0" #34 daemon prio=5 os_prio=31
tid=0x00007fcb1601d800 nid=0x6b0b runnable [0x0000000122031000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007a07ed338> (a sun.nio.ch.Util$2)
- locked <0x00000007a07ed328> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007a07dc780> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"HttpClient-33" #33 daemon prio=5 os_prio=31 tid=0x00007fcb13136000
nid=0x6a13 waiting on condition [0x0000000121f2e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.eclipse.jetty.client.HttpClient$1.run(HttpClient.java:445)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- None
"FileSyncer.1" #21 daemon prio=5 os_prio=31 tid=0x00007fcb12b06800
nid=0x390b in Object.wait() [0x000000012171d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007a0028a90> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"MultiThreadedHttpConnectionManager cleanup" #19 daemon prio=5 os_prio=31
tid=0x00007fcb12413800 nid=0x3c07 in Object.wait() [0x000000011d608000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007a00e6e18> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000007a00e6e18> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1119)
Locked ownable synchronizers:
- None
"com.google.inject.internal.util.$Finalizer" #18 daemon prio=5 os_prio=31
tid=0x00007fcb15365800 nid=0x6603 in Object.wait() [0x000000012161a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007a002a6d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000007a002a6d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)
Locked ownable synchronizers:
- None
"EventBulkSenderThread-1" #17 daemon prio=5 os_prio=31
tid=0x00007fcb15344000 nid=0x6403 waiting on condition [0x000000012076d000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
com.service_now.mid.probe.event.SNEventBulkSender.call(SNEventBulkSender.java:351)
at
com.service_now.mid.probe.event.SNEventBulkSender.call(SNEventBulkSender.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- <0x00000007a002aab0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"MetricEventLogging" #16 daemon prio=5 os_prio=31 tid=0x00007fcb12b3c000
nid=0x6203 in Object.wait() [0x000000012104b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007a002a7d0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Service Thread" #12 daemon prio=9 os_prio=31 tid=0x00007fcb15012000
nid=0x5803 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread3" #11 daemon prio=9 os_prio=31 tid=0x00007fcb15013000
nid=0x5603 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #10 daemon prio=9 os_prio=31 tid=0x00007fcb1301c000
nid=0x5403 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #9 daemon prio=9 os_prio=31 tid=0x00007fcb1301b000
nid=0x5203 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #8 daemon prio=9 os_prio=31 tid=0x00007fcb1301a800
nid=0x5003 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"JDWP Command Reader" #7 daemon prio=10 os_prio=31 tid=0x00007fcb1581a800
nid=0x4e03 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"JDWP Event Helper Thread" #6 daemon prio=10 os_prio=31
tid=0x00007fcb13019800 nid=0x4c03 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=31
tid=0x00007fcb16001000 nid=0x4a03 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fcb12062000
nid=0x3d17 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fcb1202c800 nid=0x3603 in
Object.wait() [0x000000011d4bc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000007a0038388> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:193)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fcb1202c000
nid=0x3403 in Object.wait() [0x000000011d3b9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:136)
- locked <0x00000007a00e6ed8> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"MIDServer" #1 prio=5 os_prio=31 tid=0x00007fcb12007800 nid=0x1403
sleeping[0x0000000107ba9000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.service_now.mid.MIDServer.run(MIDServer.java:199)
at com.service_now.mid.Main.runMID(Main.java:72)
at com.service_now.mid.Main.handleStart(Main.java:33)
at com.service_now.mid.Main.main(Main.java:27)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=31 tid=0x00007fcb1500f000 nid=0x3203 runnable
"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fcb13017800 nid=0x2203
runnable
"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007fcb12010000 nid=0x2403
runnable
"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007fcb13018800 nid=0x2603
runnable
"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007fcb12010800 nid=0x2803
runnable
"GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007fcb12011800 nid=0x2a03
runnable
"GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007fcb12012000 nid=0x2c03
runnable
"GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007fcb12012800 nid=0x2e03
runnable
"GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007fcb12013000 nid=0x3003
runnable
"VM Periodic Task Thread" os_prio=31 tid=0x00007fcb13019000 nid=0x5a03
waiting on condition
JNI global references: 12528
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Server-node-fails-to-start-with-exception-IllegalThreadStateException-tp14150p14172.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Server node fails to start with exception
IllegalThreadStateException
Posted by vkulichenko <va...@gmail.com>.
What threads are left behind? Do you have a thread dump that you can show?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Server-node-fails-to-start-with-exception-IllegalThreadStateException-tp14150p14171.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.