You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jeff Whiting <je...@qualtrics.com> on 2011/09/12 17:24:58 UTC
HRegionServer not shutting down
We are running cdh3u1 on centos 5.6 and we have frequent problems shutting down region servers.
They get stuck and the only thing I can do to stop them is a kill -9 which is a little harsh. I
tried doing a normal jstack but it doesn't respond. With the -F option I get what you see at the
bottom of the email. Also in the region server logs I see the following which would imply that it
should be shutdown.
2011-09-11 13:16:18,692 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
requests=0, regions=16, stores=27, storefiles=17, storefileIndexSize=1, memstoreSize=10,
compactionQueueSize=0, flushQueueSize=0, usedHeap=609, maxHeap=795, blockCacheSize=130088528,
blockCacheFree=36818560, blockCacheCount=1055, blockCacheHitCount=2038243,
blockCacheMissCount=92093, blockCacheEvictedCount=600, blockCacheHitRatio=95,
blockCacheHitCachingRatio=99
2011-09-11 13:16:18,692 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: connection
to cluster: dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef connection to cluster:
dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef received expired from ZooKeeper, aborting
2011-09-11 13:16:18,692 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2011-09-12 09:07:16,289 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
starting; hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-15,5,main]
2011-09-12 09:07:16,289 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
Mon Sep 12 09:14:26 MDT 2011 Killing regionserver
Any ideas as to why the region servers can't shutdown? My java version is:
[root@ds1 hbase]# java -version
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
Thanks,
~Jeff
[root@ds1 ~]# jstack 18789
18789: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
[root@ds1 ~]# jstack -F 18789
Attaching to process ID 18789, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 17.0-b16
Deadlock Detection:
No deadlocks found.
Thread 17578: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 17322: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 16419: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 16417: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 19017: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18901: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18871: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18870: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18815: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18868: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18843: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18840: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18823: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18822: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18821: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
Thread 18820: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException:
get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: HRegionServer not shutting down
Posted by Harsh J <ha...@cloudera.com>.
Hey Jeff,
Usually if jstack gives you such an error, you may try:
$ sudo -u hbase jstack <PID> # Or sudo to whatever the process owner is.
The -F opt is unhelpful at times, as in this case (don't see any useful info in the output - just thread numbers and exceptions everywhere).
P.s. There's also a new bin/graceful_stop.sh script to stop RegionServers elegantly in HBase 0.90 now. Not exactly related to your problem perhaps, but thought you might wanna know!
On 12-Sep-2011, at 8:54 PM, Jeff Whiting wrote:
> We are running cdh3u1 on centos 5.6 and we have frequent problems shutting down region servers. They get stuck and the only thing I can do to stop them is a kill -9 which is a little harsh. I tried doing a normal jstack but it doesn't respond. With the -F option I get what you see at the bottom of the email. Also in the region server logs I see the following which would imply that it should be shutdown.
>
> 2011-09-11 13:16:18,692 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requests=0, regions=16, stores=27, storefiles=17, storefileIndexSize=1, memstoreSize=10, compactionQueueSize=0, flushQueueSize=0, usedHeap=609, maxHeap=795, blockCacheSize=130088528, blockCacheFree=36818560, blockCacheCount=1055, blockCacheHitCount=2038243, blockCacheMissCount=92093, blockCacheEvictedCount=600, blockCacheHitRatio=95, blockCacheHitCachingRatio=99
> 2011-09-11 13:16:18,692 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: connection to cluster: dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef connection to cluster: dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef received expired from ZooKeeper, aborting
> 2011-09-11 13:16:18,692 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
> 2011-09-12 09:07:16,289 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-15,5,main]
> 2011-09-12 09:07:16,289 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
> Mon Sep 12 09:14:26 MDT 2011 Killing regionserver
>
> Any ideas as to why the region servers can't shutdown? My java version is:
>
> [root@ds1 hbase]# java -version
> java version "1.6.0_21"
> Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
>
> Thanks,
>
> ~Jeff
>
> [root@ds1 ~]# jstack 18789
> 18789: Unable to open socket file: target process not responding or HotSpot VM not loaded
> The -F option can be used when the target process is not responding
> [root@ds1 ~]# jstack -F 18789
> Attaching to process ID 18789, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 17.0-b16
> Deadlock Detection:
>
> No deadlocks found.
>
> Thread 17578: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 17322: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 16419: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 16417: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 19017: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18901: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18871: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18870: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18815: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18868: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18843: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18840: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18823: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18822: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18821: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18820: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com
>
Re: HRegionServer not shutting down
Posted by Jeff Whiting <je...@qualtrics.com>.
Yes we are replicating between this cluster and a backup cluster.
~Jeff
On 9/12/2011 9:47 PM, Stack wrote:
> Are you trying to do replication?
> St.Ack
>
> On Mon, Sep 12, 2011 at 3:00 PM, Jeff Whiting<je...@qualtrics.com> wrote:
>> I got some interesting stuff in the .out file. I've included it below. The
>> stack traces have hbase classes in them but I'm unsure if it is related to
>> hbase-4159.
>>
>> ~Jeff
>>
>> 2011-09-12 15:56:16
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode):
>>
>> "Shutdownhook:regionserver60020" prio=10 tid=0x0000000044c5b800 nid=0x39e2
>> in Object.wait() [0x000000004c9b4000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaab1204150> (a java.lang.Thread)
>> at java.lang.Thread.join(Thread.java:1143)
>> - locked<0x00002aaab1204150> (a java.lang.Thread)
>> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
>> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:79)
>> at
>> org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread.run(ShutdownHook.java:104)
>>
>> "SIGTERM handler" daemon prio=10 tid=0x00000000449f9800 nid=0x39e1 in
>> Object.wait() [0x0000000042d4e000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaab11dbb58> (a
>> org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
>> at java.lang.Thread.join(Thread.java:1143)
>> - locked<0x00002aaab11dbb58> (a
>> org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
>> at java.lang.Thread.join(Thread.java:1196)
>> at
>> java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
>> at
>> java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
>> at java.lang.Shutdown.runHooks(Shutdown.java:79)
>> at java.lang.Shutdown.sequence(Shutdown.java:123)
>> at java.lang.Shutdown.exit(Shutdown.java:168)
>> - locked<0x00002aaae05e04e8> (a java.lang.Class for java.lang.Shutdown)
>> at java.lang.Terminator$1.handle(Terminator.java:35)
>> at sun.misc.Signal$1.run(Signal.java:195)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aaaf036e800
>> nid=0x6a6b in Object.wait() [0x000000004c5b0000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaab273c0d0> (a
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>> at java.lang.Object.wait(Object.java:485)
>> at
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:519)
>> - locked<0x00002aaab273c0d0> (a
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>>
>> "regionserver60020.replicationSource,dev2" daemon prio=10
>> tid=0x00002aaaf0312800 nid=0x69f8 in Object.wait() [0x000000004533e000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaab12464c0> (a
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
>> at java.lang.Thread.join(Thread.java:1151)
>> - locked<0x00002aaab12464c0> (a
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
>> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:649)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$1.uncaughtException(ReplicationSource.java:628)
>> at java.lang.Thread.dispatchUncaughtException(Thread.java:1831)
>>
>> "Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aaaf4067000
>> nid=0x69e8 in Object.wait() [0x0000000043152000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaab26cb898> (a java.util.TaskQueue)
>> at java.util.TimerThread.mainLoop(Timer.java:509)
>> - locked<0x00002aaab26cb898> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:462)
>>
>> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00002aaaf4065800
>> nid=0x69e7 in Object.wait() [0x0000000043051000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaab217c3f0> (a java.util.TaskQueue)
>> at java.util.TimerThread.mainLoop(Timer.java:509)
>> - locked<0x00002aaab217c3f0> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:462)
>>
>> "LeaseChecker" daemon prio=10 tid=0x00002aaaf4060800 nid=0x69e5 waiting on
>> condition [0x0000000042e4f000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> "regionserver60020-EventThread" daemon prio=10 tid=0x00002aaaf4053800
>> nid=0x69e3 waiting on condition [0x0000000042c4d000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaab2719738> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>>
>> "regionserver60020-SendThread(ds1:2181)" daemon prio=10
>> tid=0x00002aaaf4031000 nid=0x69e2 runnable [0x0000000042b4c000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked<0x00002aaab11fffe0> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaab11fffc8> (a java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaab274f468> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>>
>> "DestroyJavaVM" prio=10 tid=0x00002aaaf0189800 nid=0x69c1 waiting on
>> condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "regionserver60020" prio=10 tid=0x00002aaaf01ca800 nid=0x69dc in
>> Object.wait() [0x0000000040bb2000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaab12464c0> (a
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
>> at java.lang.Thread.join(Thread.java:1151)
>> - locked<0x00002aaab12464c0> (a
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
>> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:649)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:636)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:225)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:97)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.join(HRegionServer.java:1419)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:683)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> "Timer thread for monitoring rpc" daemon prio=10 tid=0x00002aaaf0181800
>> nid=0x69db in Object.wait() [0x000000004294a000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaab0bb27e0> (a java.util.TaskQueue)
>> at java.util.TimerThread.mainLoop(Timer.java:509)
>> - locked<0x00002aaab0bb27e0> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:462)
>>
>> "Low Memory Detector" daemon prio=10 tid=0x0000000043934000 nid=0x69cc
>> runnable [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "CompilerThread1" daemon prio=10 tid=0x0000000043931800 nid=0x69cb waiting
>> on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "CompilerThread0" daemon prio=10 tid=0x000000004392d000 nid=0x69ca waiting
>> on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "Signal Dispatcher" daemon prio=10 tid=0x000000004392a800 nid=0x69c9 waiting
>> on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000043928800
>> nid=0x69c8 waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "Finalizer" daemon prio=10 tid=0x0000000043904000 nid=0x69c7 in
>> Object.wait() [0x0000000041d3e000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaab10dd220> (a java.lang.ref.ReferenceQueue$Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>> - locked<0x00002aaab10dd220> (a java.lang.ref.ReferenceQueue$Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>
>> "Reference Handler" daemon prio=10 tid=0x0000000043902000 nid=0x69c6 in
>> Object.wait() [0x0000000041c3d000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaab0bbab60> (a java.lang.ref.Reference$Lock)
>> at java.lang.Object.wait(Object.java:485)
>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>> - locked<0x00002aaab0bbab60> (a java.lang.ref.Reference$Lock)
>>
>> "VM Thread" prio=10 tid=0x00000000438fe000 nid=0x69c5 runnable
>>
>> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000043843800
>> nid=0x69c2 runnable
>>
>> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000043845000
>> nid=0x69c3 runnable
>>
>> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000438a3800 nid=0x69c4
>> runnable
>> "VM Periodic Task Thread" prio=10 tid=0x000000004393f000 nid=0x69cd waiting
>> on condition
>>
>> JNI global references: 1621
>>
>> Heap
>> par new generation total 19136K, used 9478K [0x00002aaaae200000,
>> 0x00002aaaaf6c0000, 0x00002aaab0b90000)
>> eden space 17024K, 54% used [0x00002aaaae200000, 0x00002aaaaeb115f0,
>> 0x00002aaaaf2a0000)
>> from space 2112K, 9% used [0x00002aaaaf2a0000, 0x00002aaaaf2d0248,
>> 0x00002aaaaf4b0000)
>> to space 2112K, 0% used [0x00002aaaaf4b0000, 0x00002aaaaf4b0000,
>> 0x00002aaaaf6c0000)
>> concurrent mark-sweep generation total 97196K, used 66304K
>> [0x00002aaab0b90000, 0x00002aaab6a7b000, 0x00002aaae0200000)
>> concurrent-mark-sweep perm gen total 32624K, used 19532K
>> [0x00002aaae0200000, 0x00002aaae21dc000, 0x00002aaae5600000)
>>
>>
>>
>>
>> On 9/12/2011 10:59 AM, Stack wrote:
>>> Yes. I'd think the RS is done. What happens if you do 'kill -QUIT
>>> <PID>'. Do you get output in the .out log? I'd be interested in
>>> checking it out Jeff. I wonder if it:
>>>
>>> HBASE-4159 HBaseServer - IPC Reader threads are not daemons (Douglas
>>> Campbell)
>>>
>>>
>>> St.Ack
>>>
>>> On Mon, Sep 12, 2011 at 8:24 AM, Jeff Whiting<je...@qualtrics.com> wrote:
>>>> We are running cdh3u1 on centos 5.6 and we have frequent problems
>>>> shutting
>>>> down region servers. They get stuck and the only thing I can do to stop
>>>> them is a kill -9 which is a little harsh. I tried doing a normal jstack
>>>> but it doesn't respond. With the -F option I get what you see at the
>>>> bottom
>>>> of the email. Also in the region server logs I see the following which
>>>> would imply that it should be shutdown.
>>>>
>>>> 2011-09-11 13:16:18,692 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>>>> requests=0, regions=16, stores=27, storefiles=17, storefileIndexSize=1,
>>>> memstoreSize=10, compactionQueueSize=0, flushQueueSize=0, usedHeap=609,
>>>> maxHeap=795, blockCacheSize=130088528, blockCacheFree=36818560,
>>>> blockCacheCount=1055, blockCacheHitCount=2038243,
>>>> blockCacheMissCount=92093,
>>>> blockCacheEvictedCount=600, blockCacheHitRatio=95,
>>>> blockCacheHitCachingRatio=99
>>>> 2011-09-11 13:16:18,692 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: connection
>>>> to
>>>> cluster: dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef
>>>> connection to cluster:
>>>> dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef received
>>>> expired
>>>> from ZooKeeper, aborting
>>>> 2011-09-11 13:16:18,692 INFO org.apache.zookeeper.ClientCnxn: EventThread
>>>> shut down
>>>> 2011-09-12 09:07:16,289 INFO
>>>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>>>> starting;
>>>> hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-15,5,main]
>>>> 2011-09-12 09:07:16,289 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>>>> hook
>>>> Mon Sep 12 09:14:26 MDT 2011 Killing regionserver
>>>>
>>>> Any ideas as to why the region servers can't shutdown? My java version
>>>> is:
>>>>
>>>> [root@ds1 hbase]# java -version
>>>> java version "1.6.0_21"
>>>> Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
>>>>
>>>> Thanks,
>>>>
>>>> ~Jeff
>>>>
>>>> [root@ds1 ~]# jstack 18789
>>>> 18789: Unable to open socket file: target process not responding or
>>>> HotSpot
>>>> VM not loaded
>>>> The -F option can be used when the target process is not responding
>>>> [root@ds1 ~]# jstack -F 18789
>>>> Attaching to process ID 18789, please wait...
>>>> Debugger attached successfully.
>>>> Server compiler detected.
>>>> JVM version is 17.0-b16
>>>> Deadlock Detection:
>>>>
>>>> No deadlocks found.
>>>>
>>>> Thread 17578: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 17322: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 16419: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 16417: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 19017: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18901: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18871: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18870: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18815: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18868: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18843: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18840: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18823: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18822: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18821: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> Thread 18820: (state = BLOCKED)
>>>> Error occurred during stack walking:
>>>> sun.jvm.hotspot.debugger.DebuggerException:
>>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>>> lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>>> at
>>>>
>>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>>> failed for a lwp
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>>> Method)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>>> at
>>>>
>>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>>
>>>>
>>>> --
>>>> Jeff Whiting
>>>> Qualtrics Senior Software Engineer
>>>> jeffw@qualtrics.com
>>>>
>>>>
>> --
>> Jeff Whiting
>> Qualtrics Senior Software Engineer
>> jeffw@qualtrics.com
>>
>>
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: HRegionServer not shutting down
Posted by Stack <st...@duboce.net>.
Are you trying to do replication?
St.Ack
On Mon, Sep 12, 2011 at 3:00 PM, Jeff Whiting <je...@qualtrics.com> wrote:
> I got some interesting stuff in the .out file. I've included it below. The
> stack traces have hbase classes in them but I'm unsure if it is related to
> hbase-4159.
>
> ~Jeff
>
> 2011-09-12 15:56:16
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode):
>
> "Shutdownhook:regionserver60020" prio=10 tid=0x0000000044c5b800 nid=0x39e2
> in Object.wait() [0x000000004c9b4000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab1204150> (a java.lang.Thread)
> at java.lang.Thread.join(Thread.java:1143)
> - locked <0x00002aaab1204150> (a java.lang.Thread)
> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:79)
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread.run(ShutdownHook.java:104)
>
> "SIGTERM handler" daemon prio=10 tid=0x00000000449f9800 nid=0x39e1 in
> Object.wait() [0x0000000042d4e000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab11dbb58> (a
> org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
> at java.lang.Thread.join(Thread.java:1143)
> - locked <0x00002aaab11dbb58> (a
> org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
> at java.lang.Thread.join(Thread.java:1196)
> at
> java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
> at
> java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
> at java.lang.Shutdown.runHooks(Shutdown.java:79)
> at java.lang.Shutdown.sequence(Shutdown.java:123)
> at java.lang.Shutdown.exit(Shutdown.java:168)
> - locked <0x00002aaae05e04e8> (a java.lang.Class for java.lang.Shutdown)
> at java.lang.Terminator$1.handle(Terminator.java:35)
> at sun.misc.Signal$1.run(Signal.java:195)
> at java.lang.Thread.run(Thread.java:619)
>
> "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aaaf036e800
> nid=0x6a6b in Object.wait() [0x000000004c5b0000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab273c0d0> (a
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:519)
> - locked <0x00002aaab273c0d0> (a
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>
> "regionserver60020.replicationSource,dev2" daemon prio=10
> tid=0x00002aaaf0312800 nid=0x69f8 in Object.wait() [0x000000004533e000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab12464c0> (a
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
> at java.lang.Thread.join(Thread.java:1151)
> - locked <0x00002aaab12464c0> (a
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:649)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$1.uncaughtException(ReplicationSource.java:628)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1831)
>
> "Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aaaf4067000
> nid=0x69e8 in Object.wait() [0x0000000043152000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab26cb898> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> - locked <0x00002aaab26cb898> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:462)
>
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00002aaaf4065800
> nid=0x69e7 in Object.wait() [0x0000000043051000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab217c3f0> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> - locked <0x00002aaab217c3f0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:462)
>
> "LeaseChecker" daemon prio=10 tid=0x00002aaaf4060800 nid=0x69e5 waiting on
> condition [0x0000000042e4f000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
> at java.lang.Thread.run(Thread.java:619)
>
> "regionserver60020-EventThread" daemon prio=10 tid=0x00002aaaf4053800
> nid=0x69e3 waiting on condition [0x0000000042c4d000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00002aaab2719738> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>
> "regionserver60020-SendThread(ds1:2181)" daemon prio=10
> tid=0x00002aaaf4031000 nid=0x69e2 runnable [0x0000000042b4c000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00002aaab11fffe0> (a sun.nio.ch.Util$1)
> - locked <0x00002aaab11fffc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00002aaab274f468> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>
> "DestroyJavaVM" prio=10 tid=0x00002aaaf0189800 nid=0x69c1 waiting on
> condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "regionserver60020" prio=10 tid=0x00002aaaf01ca800 nid=0x69dc in
> Object.wait() [0x0000000040bb2000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab12464c0> (a
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
> at java.lang.Thread.join(Thread.java:1151)
> - locked <0x00002aaab12464c0> (a
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:649)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:636)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:225)
> at
> org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:97)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.join(HRegionServer.java:1419)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:683)
> at java.lang.Thread.run(Thread.java:619)
>
> "Timer thread for monitoring rpc" daemon prio=10 tid=0x00002aaaf0181800
> nid=0x69db in Object.wait() [0x000000004294a000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab0bb27e0> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> - locked <0x00002aaab0bb27e0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:462)
>
> "Low Memory Detector" daemon prio=10 tid=0x0000000043934000 nid=0x69cc
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x0000000043931800 nid=0x69cb waiting
> on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x000000004392d000 nid=0x69ca waiting
> on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x000000004392a800 nid=0x69c9 waiting
> on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000043928800
> nid=0x69c8 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x0000000043904000 nid=0x69c7 in
> Object.wait() [0x0000000041d3e000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab10dd220> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x00002aaab10dd220> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x0000000043902000 nid=0x69c6 in
> Object.wait() [0x0000000041c3d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab0bbab60> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x00002aaab0bbab60> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00000000438fe000 nid=0x69c5 runnable
>
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000043843800
> nid=0x69c2 runnable
>
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000043845000
> nid=0x69c3 runnable
>
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000438a3800 nid=0x69c4
> runnable
> "VM Periodic Task Thread" prio=10 tid=0x000000004393f000 nid=0x69cd waiting
> on condition
>
> JNI global references: 1621
>
> Heap
> par new generation total 19136K, used 9478K [0x00002aaaae200000,
> 0x00002aaaaf6c0000, 0x00002aaab0b90000)
> eden space 17024K, 54% used [0x00002aaaae200000, 0x00002aaaaeb115f0,
> 0x00002aaaaf2a0000)
> from space 2112K, 9% used [0x00002aaaaf2a0000, 0x00002aaaaf2d0248,
> 0x00002aaaaf4b0000)
> to space 2112K, 0% used [0x00002aaaaf4b0000, 0x00002aaaaf4b0000,
> 0x00002aaaaf6c0000)
> concurrent mark-sweep generation total 97196K, used 66304K
> [0x00002aaab0b90000, 0x00002aaab6a7b000, 0x00002aaae0200000)
> concurrent-mark-sweep perm gen total 32624K, used 19532K
> [0x00002aaae0200000, 0x00002aaae21dc000, 0x00002aaae5600000)
>
>
>
>
> On 9/12/2011 10:59 AM, Stack wrote:
>>
>> Yes. I'd think the RS is done. What happens if you do 'kill -QUIT
>> <PID>'. Do you get output in the .out log? I'd be interested in
>> checking it out Jeff. I wonder if it:
>>
>> HBASE-4159 HBaseServer - IPC Reader threads are not daemons (Douglas
>> Campbell)
>>
>>
>> St.Ack
>>
>> On Mon, Sep 12, 2011 at 8:24 AM, Jeff Whiting<je...@qualtrics.com> wrote:
>>>
>>> We are running cdh3u1 on centos 5.6 and we have frequent problems
>>> shutting
>>> down region servers. They get stuck and the only thing I can do to stop
>>> them is a kill -9 which is a little harsh. I tried doing a normal jstack
>>> but it doesn't respond. With the -F option I get what you see at the
>>> bottom
>>> of the email. Also in the region server logs I see the following which
>>> would imply that it should be shutdown.
>>>
>>> 2011-09-11 13:16:18,692 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>>> requests=0, regions=16, stores=27, storefiles=17, storefileIndexSize=1,
>>> memstoreSize=10, compactionQueueSize=0, flushQueueSize=0, usedHeap=609,
>>> maxHeap=795, blockCacheSize=130088528, blockCacheFree=36818560,
>>> blockCacheCount=1055, blockCacheHitCount=2038243,
>>> blockCacheMissCount=92093,
>>> blockCacheEvictedCount=600, blockCacheHitRatio=95,
>>> blockCacheHitCachingRatio=99
>>> 2011-09-11 13:16:18,692 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: connection
>>> to
>>> cluster: dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef
>>> connection to cluster:
>>> dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef received
>>> expired
>>> from ZooKeeper, aborting
>>> 2011-09-11 13:16:18,692 INFO org.apache.zookeeper.ClientCnxn: EventThread
>>> shut down
>>> 2011-09-12 09:07:16,289 INFO
>>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>>> starting;
>>> hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-15,5,main]
>>> 2011-09-12 09:07:16,289 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>>> hook
>>> Mon Sep 12 09:14:26 MDT 2011 Killing regionserver
>>>
>>> Any ideas as to why the region servers can't shutdown? My java version
>>> is:
>>>
>>> [root@ds1 hbase]# java -version
>>> java version "1.6.0_21"
>>> Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
>>> Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
>>>
>>> Thanks,
>>>
>>> ~Jeff
>>>
>>> [root@ds1 ~]# jstack 18789
>>> 18789: Unable to open socket file: target process not responding or
>>> HotSpot
>>> VM not loaded
>>> The -F option can be used when the target process is not responding
>>> [root@ds1 ~]# jstack -F 18789
>>> Attaching to process ID 18789, please wait...
>>> Debugger attached successfully.
>>> Server compiler detected.
>>> JVM version is 17.0-b16
>>> Deadlock Detection:
>>>
>>> No deadlocks found.
>>>
>>> Thread 17578: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 17322: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 16419: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 16417: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 19017: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18901: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18871: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18870: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18815: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18868: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18843: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18840: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18823: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18822: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18821: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> Thread 18820: (state = BLOCKED)
>>> Error occurred during stack walking:
>>> sun.jvm.hotspot.debugger.DebuggerException:
>>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
>>> lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>>> at
>>>
>>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>>> at sun.tools.jstack.JStack.main(JStack.java:84)
>>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>>> failed for a lwp
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>>> Method)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>>> at
>>>
>>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>>
>>>
>>> --
>>> Jeff Whiting
>>> Qualtrics Senior Software Engineer
>>> jeffw@qualtrics.com
>>>
>>>
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com
>
>
Re: HRegionServer not shutting down
Posted by Jeff Whiting <je...@qualtrics.com>.
Thanks guys for figuring out the problem.
~Jeff
On 9/13/2011 2:59 PM, Jean-Daniel Cryans wrote:
> Just figured it out (with Stack's help):
> https://issues.apache.org/jira/browse/HBASE-4390
>
> J-D
>
> On Mon, Sep 12, 2011 at 3:00 PM, Jeff Whiting<je...@qualtrics.com> wrote:
>> I got some interesting stuff in the .out file. I've included it below. The
>> stack traces have hbase classes in them but I'm unsure if it is related to
>> hbase-4159.
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: HRegionServer not shutting down
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Just figured it out (with Stack's help):
https://issues.apache.org/jira/browse/HBASE-4390
J-D
On Mon, Sep 12, 2011 at 3:00 PM, Jeff Whiting <je...@qualtrics.com> wrote:
> I got some interesting stuff in the .out file. I've included it below. The
> stack traces have hbase classes in them but I'm unsure if it is related to
> hbase-4159.
RE: HRegionServer not shutting down
Posted by Ramkrishna S Vasudevan <ra...@huawei.com>.
Hi Jeff
I have a faced a similar issue and my thread dumps showed the following
2011-08-16 11:26:22
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode):
"Attach Listener" daemon prio=10 tid=0x0000000040fa3000 nid=0x4046 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"SIGTERM handler" daemon prio=10 tid=0x0000000040fa0800 nid=0x35b5 waiting
for monitor entry [0x00007f442f0f7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:168)
- waiting to lock <0x00007f44f8cf4900> (a java.lang.Class for
java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:35)
at sun.misc.Signal$1.run(Signal.java:195)
at java.lang.Thread.run(Thread.java:619)
"SIGTERM handler" daemon prio=10 tid=0x0000000040f9e800 nid=0x34e0 waiting
for monitor entry [0x00007f442f1f8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:168)
- waiting to lock <0x00007f44f8cf4900> (a java.lang.Class for
java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:35)
at sun.misc.Signal$1.run(Signal.java:195)
at java.lang.Thread.run(Thread.java:619)
"SIGTERM handler" daemon prio=10 tid=0x0000000040f9c800 nid=0x345b waiting
for monitor entry [0x00007f442f2f9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:168)
- waiting to lock <0x00007f44f8cf4900> (a java.lang.Class for
java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:35)
at sun.misc.Signal$1.run(Signal.java:195)
at java.lang.Thread.run(Thread.java:619)
I read somewhere like a problem with Shutdown hooks where this problem may
occur. That time dint mind much. May be we can dig in deeper.
Best Regards
Ram
-----Original Message-----
From: Jeff Whiting [mailto:jeffw@qualtrics.com]
Sent: Tuesday, September 13, 2011 3:31 AM
To: user@hbase.apache.org
Subject: Re: HRegionServer not shutting down
I got some interesting stuff in the .out file. I've included it below. The
stack traces have hbase
classes in them but I'm unsure if it is related to hbase-4159.
~Jeff
2011-09-12 15:56:16
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode):
"Shutdownhook:regionserver60020" prio=10 tid=0x0000000044c5b800 nid=0x39e2
in Object.wait()
[0x000000004c9b4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab1204150> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1143)
- locked <0x00002aaab1204150> (a java.lang.Thread)
at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:79)
at
org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread.run(Shu
tdownHook.java:104)
"SIGTERM handler" daemon prio=10 tid=0x00000000449f9800 nid=0x39e1 in
Object.wait() [0x0000000042d4e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab11dbb58> (a
org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
at java.lang.Thread.join(Thread.java:1143)
- locked <0x00002aaab11dbb58> (a
org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
at java.lang.Thread.join(Thread.java:1196)
at
java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79
)
at
java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
at java.lang.Shutdown.runHooks(Shutdown.java:79)
at java.lang.Shutdown.sequence(Shutdown.java:123)
at java.lang.Shutdown.exit(Shutdown.java:168)
- locked <0x00002aaae05e04e8> (a java.lang.Class for
java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:35)
at sun.misc.Signal$1.run(Signal.java:195)
at java.lang.Thread.run(Thread.java:619)
"LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aaaf036e800
nid=0x6a6b in Object.wait()
[0x000000004c5b0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab273c0d0> (a
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCa
che.java:519)
- locked <0x00002aaab273c0d0> (a
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
"regionserver60020.replicationSource,dev2" daemon prio=10
tid=0x00002aaaf0312800 nid=0x69f8 in
Object.wait() [0x000000004533e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab12464c0> (a
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
at java.lang.Thread.join(Thread.java:1151)
- locked <0x00002aaab12464c0> (a
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate
(ReplicationSource.java:649)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$1.uncaugh
tException(ReplicationSource.java:628)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1831)
"Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aaaf4067000
nid=0x69e8 in Object.wait()
[0x0000000043152000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab26cb898> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaab26cb898> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Timer thread for monitoring hbase" daemon prio=10 tid=0x00002aaaf4065800
nid=0x69e7 in
Object.wait() [0x0000000043051000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab217c3f0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaab217c3f0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"LeaseChecker" daemon prio=10 tid=0x00002aaaf4060800 nid=0x69e5 waiting on
condition
[0x0000000042e4f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
at java.lang.Thread.run(Thread.java:619)
"regionserver60020-EventThread" daemon prio=10 tid=0x00002aaaf4053800
nid=0x69e3 waiting on
condition [0x0000000042c4d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab2719738> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
"regionserver60020-SendThread(ds1:2181)" daemon prio=10
tid=0x00002aaaf4031000 nid=0x69e2 runnable
[0x0000000042b4c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaab11fffe0> (a sun.nio.ch.Util$1)
- locked <0x00002aaab11fffc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00002aaab274f468> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
"DestroyJavaVM" prio=10 tid=0x00002aaaf0189800 nid=0x69c1 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"regionserver60020" prio=10 tid=0x00002aaaf01ca800 nid=0x69dc in
Object.wait() [0x0000000040bb2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab12464c0> (a
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
at java.lang.Thread.join(Thread.java:1151)
- locked <0x00002aaab12464c0> (a
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate
(ReplicationSource.java:649)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate
(ReplicationSource.java:636)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.jo
in(ReplicationSourceManager.java:225)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replicatio
n.java:97)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.join(HRegionServer.java:1
419)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:68
3)
at java.lang.Thread.run(Thread.java:619)
"Timer thread for monitoring rpc" daemon prio=10 tid=0x00002aaaf0181800
nid=0x69db in Object.wait()
[0x000000004294a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab0bb27e0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaab0bb27e0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=10 tid=0x0000000043934000 nid=0x69cc
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x0000000043931800 nid=0x69cb waiting
on condition
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x000000004392d000 nid=0x69ca waiting
on condition
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x000000004392a800 nid=0x69c9 waiting
on condition
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000043928800
nid=0x69c8 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000043904000 nid=0x69c7 in
Object.wait() [0x0000000041d3e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab10dd220> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00002aaab10dd220> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x0000000043902000 nid=0x69c6 in
Object.wait()
[0x0000000041c3d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab0bbab60> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab0bbab60> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x00000000438fe000 nid=0x69c5 runnable
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000043843800
nid=0x69c2 runnable
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000043845000
nid=0x69c3 runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000438a3800 nid=0x69c4
runnable
"VM Periodic Task Thread" prio=10 tid=0x000000004393f000 nid=0x69cd waiting
on condition
JNI global references: 1621
Heap
par new generation total 19136K, used 9478K [0x00002aaaae200000,
0x00002aaaaf6c0000,
0x00002aaab0b90000)
eden space 17024K, 54% used [0x00002aaaae200000, 0x00002aaaaeb115f0,
0x00002aaaaf2a0000)
from space 2112K, 9% used [0x00002aaaaf2a0000, 0x00002aaaaf2d0248,
0x00002aaaaf4b0000)
to space 2112K, 0% used [0x00002aaaaf4b0000, 0x00002aaaaf4b0000,
0x00002aaaaf6c0000)
concurrent mark-sweep generation total 97196K, used 66304K
[0x00002aaab0b90000,
0x00002aaab6a7b000, 0x00002aaae0200000)
concurrent-mark-sweep perm gen total 32624K, used 19532K
[0x00002aaae0200000, 0x00002aaae21dc000,
0x00002aaae5600000)
On 9/12/2011 10:59 AM, Stack wrote:
> Yes. I'd think the RS is done. What happens if you do 'kill -QUIT
> <PID>'. Do you get output in the .out log? I'd be interested in
> checking it out Jeff. I wonder if it:
>
> HBASE-4159 HBaseServer - IPC Reader threads are not daemons (Douglas
> Campbell)
>
>
> St.Ack
>
> On Mon, Sep 12, 2011 at 8:24 AM, Jeff Whiting<je...@qualtrics.com> wrote:
>> We are running cdh3u1 on centos 5.6 and we have frequent problems
shutting
>> down region servers. They get stuck and the only thing I can do to stop
>> them is a kill -9 which is a little harsh. I tried doing a normal jstack
>> but it doesn't respond. With the -F option I get what you see at the
bottom
>> of the email. Also in the region server logs I see the following which
>> would imply that it should be shutdown.
>>
>> 2011-09-11 13:16:18,692 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>> requests=0, regions=16, stores=27, storefiles=17, storefileIndexSize=1,
>> memstoreSize=10, compactionQueueSize=0, flushQueueSize=0, usedHeap=609,
>> maxHeap=795, blockCacheSize=130088528, blockCacheFree=36818560,
>> blockCacheCount=1055, blockCacheHitCount=2038243,
blockCacheMissCount=92093,
>> blockCacheEvictedCount=600, blockCacheHitRatio=95,
>> blockCacheHitCachingRatio=99
>> 2011-09-11 13:16:18,692 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: connection
to
>> cluster: dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef
>> connection to cluster:
>> dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef received
expired
>> from ZooKeeper, aborting
>> 2011-09-11 13:16:18,692 INFO org.apache.zookeeper.ClientCnxn: EventThread
>> shut down
>> 2011-09-12 09:07:16,289 INFO
>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
starting;
>> hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-15,5,main]
>> 2011-09-12 09:07:16,289 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
hook
>> Mon Sep 12 09:14:26 MDT 2011 Killing regionserver
>>
>> Any ideas as to why the region servers can't shutdown? My java version
is:
>>
>> [root@ds1 hbase]# java -version
>> java version "1.6.0_21"
>> Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
>> Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
>>
>> Thanks,
>>
>> ~Jeff
>>
>> [root@ds1 ~]# jstack 18789
>> 18789: Unable to open socket file: target process not responding or
HotSpot
>> VM not loaded
>> The -F option can be used when the target process is not responding
>> [root@ds1 ~]# jstack -F 18789
>> Attaching to process ID 18789, please wait...
>> Debugger attached successfully.
>> Server compiler detected.
>> JVM version is 17.0-b16
>> Deadlock Detection:
>>
>> No deadlocks found.
>>
>> Thread 17578: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 17322: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 16419: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 16417: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 19017: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18901: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18871: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18870: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18815: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18868: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18843: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18840: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18823: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18822: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18821: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18820: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a
lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.execute(LinuxDebuggerLocal.java:152)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t(LinuxDebuggerLocal.java:466)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>>
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentF
rameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>>
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSe
t0(Native
>> Method)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLo
cal.java:51)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterS
etTask.doit(LinuxDebuggerLocal.java:460)
>> at
>>
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerTh
read.run(LinuxDebuggerLocal.java:127)
>>
>>
>> --
>> Jeff Whiting
>> Qualtrics Senior Software Engineer
>> jeffw@qualtrics.com
>>
>>
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: HRegionServer not shutting down
Posted by Jeff Whiting <je...@qualtrics.com>.
I got some interesting stuff in the .out file. I've included it below. The stack traces have hbase
classes in them but I'm unsure if it is related to hbase-4159.
~Jeff
2011-09-12 15:56:16
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode):
"Shutdownhook:regionserver60020" prio=10 tid=0x0000000044c5b800 nid=0x39e2 in Object.wait()
[0x000000004c9b4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab1204150> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1143)
- locked <0x00002aaab1204150> (a java.lang.Thread)
at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:79)
at org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread.run(ShutdownHook.java:104)
"SIGTERM handler" daemon prio=10 tid=0x00000000449f9800 nid=0x39e1 in Object.wait() [0x0000000042d4e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab11dbb58> (a
org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
at java.lang.Thread.join(Thread.java:1143)
- locked <0x00002aaab11dbb58> (a
org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
at java.lang.Thread.join(Thread.java:1196)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
at java.lang.Shutdown.runHooks(Shutdown.java:79)
at java.lang.Shutdown.sequence(Shutdown.java:123)
at java.lang.Shutdown.exit(Shutdown.java:168)
- locked <0x00002aaae05e04e8> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:35)
at sun.misc.Signal$1.run(Signal.java:195)
at java.lang.Thread.run(Thread.java:619)
"LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aaaf036e800 nid=0x6a6b in Object.wait()
[0x000000004c5b0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab273c0d0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:519)
- locked <0x00002aaab273c0d0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
"regionserver60020.replicationSource,dev2" daemon prio=10 tid=0x00002aaaf0312800 nid=0x69f8 in
Object.wait() [0x000000004533e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab12464c0> (a
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
at java.lang.Thread.join(Thread.java:1151)
- locked <0x00002aaab12464c0> (a
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:649)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$1.uncaughtException(ReplicationSource.java:628)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1831)
"Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aaaf4067000 nid=0x69e8 in Object.wait()
[0x0000000043152000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab26cb898> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaab26cb898> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Timer thread for monitoring hbase" daemon prio=10 tid=0x00002aaaf4065800 nid=0x69e7 in
Object.wait() [0x0000000043051000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab217c3f0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaab217c3f0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"LeaseChecker" daemon prio=10 tid=0x00002aaaf4060800 nid=0x69e5 waiting on condition
[0x0000000042e4f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
at java.lang.Thread.run(Thread.java:619)
"regionserver60020-EventThread" daemon prio=10 tid=0x00002aaaf4053800 nid=0x69e3 waiting on
condition [0x0000000042c4d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab2719738> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
"regionserver60020-SendThread(ds1:2181)" daemon prio=10 tid=0x00002aaaf4031000 nid=0x69e2 runnable
[0x0000000042b4c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaab11fffe0> (a sun.nio.ch.Util$1)
- locked <0x00002aaab11fffc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00002aaab274f468> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
"DestroyJavaVM" prio=10 tid=0x00002aaaf0189800 nid=0x69c1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"regionserver60020" prio=10 tid=0x00002aaaf01ca800 nid=0x69dc in Object.wait() [0x0000000040bb2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab12464c0> (a
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
at java.lang.Thread.join(Thread.java:1151)
- locked <0x00002aaab12464c0> (a
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource)
at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:91)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:649)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:636)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:225)
at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:97)
at org.apache.hadoop.hbase.regionserver.HRegionServer.join(HRegionServer.java:1419)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:683)
at java.lang.Thread.run(Thread.java:619)
"Timer thread for monitoring rpc" daemon prio=10 tid=0x00002aaaf0181800 nid=0x69db in Object.wait()
[0x000000004294a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab0bb27e0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaab0bb27e0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=10 tid=0x0000000043934000 nid=0x69cc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x0000000043931800 nid=0x69cb waiting on condition
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x000000004392d000 nid=0x69ca waiting on condition
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x000000004392a800 nid=0x69c9 waiting on condition
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000043928800 nid=0x69c8 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000043904000 nid=0x69c7 in Object.wait() [0x0000000041d3e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab10dd220> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00002aaab10dd220> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x0000000043902000 nid=0x69c6 in Object.wait()
[0x0000000041c3d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab0bbab60> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab0bbab60> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x00000000438fe000 nid=0x69c5 runnable
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000043843800 nid=0x69c2 runnable
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000043845000 nid=0x69c3 runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000438a3800 nid=0x69c4 runnable
"VM Periodic Task Thread" prio=10 tid=0x000000004393f000 nid=0x69cd waiting on condition
JNI global references: 1621
Heap
par new generation total 19136K, used 9478K [0x00002aaaae200000, 0x00002aaaaf6c0000,
0x00002aaab0b90000)
eden space 17024K, 54% used [0x00002aaaae200000, 0x00002aaaaeb115f0, 0x00002aaaaf2a0000)
from space 2112K, 9% used [0x00002aaaaf2a0000, 0x00002aaaaf2d0248, 0x00002aaaaf4b0000)
to space 2112K, 0% used [0x00002aaaaf4b0000, 0x00002aaaaf4b0000, 0x00002aaaaf6c0000)
concurrent mark-sweep generation total 97196K, used 66304K [0x00002aaab0b90000,
0x00002aaab6a7b000, 0x00002aaae0200000)
concurrent-mark-sweep perm gen total 32624K, used 19532K [0x00002aaae0200000, 0x00002aaae21dc000,
0x00002aaae5600000)
On 9/12/2011 10:59 AM, Stack wrote:
> Yes. I'd think the RS is done. What happens if you do 'kill -QUIT
> <PID>'. Do you get output in the .out log? I'd be interested in
> checking it out Jeff. I wonder if it:
>
> HBASE-4159 HBaseServer - IPC Reader threads are not daemons (Douglas
> Campbell)
>
>
> St.Ack
>
> On Mon, Sep 12, 2011 at 8:24 AM, Jeff Whiting<je...@qualtrics.com> wrote:
>> We are running cdh3u1 on centos 5.6 and we have frequent problems shutting
>> down region servers. They get stuck and the only thing I can do to stop
>> them is a kill -9 which is a little harsh. I tried doing a normal jstack
>> but it doesn't respond. With the -F option I get what you see at the bottom
>> of the email. Also in the region server logs I see the following which
>> would imply that it should be shutdown.
>>
>> 2011-09-11 13:16:18,692 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>> requests=0, regions=16, stores=27, storefiles=17, storefileIndexSize=1,
>> memstoreSize=10, compactionQueueSize=0, flushQueueSize=0, usedHeap=609,
>> maxHeap=795, blockCacheSize=130088528, blockCacheFree=36818560,
>> blockCacheCount=1055, blockCacheHitCount=2038243, blockCacheMissCount=92093,
>> blockCacheEvictedCount=600, blockCacheHitRatio=95,
>> blockCacheHitCachingRatio=99
>> 2011-09-11 13:16:18,692 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: connection to
>> cluster: dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef
>> connection to cluster:
>> dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef received expired
>> from ZooKeeper, aborting
>> 2011-09-11 13:16:18,692 INFO org.apache.zookeeper.ClientCnxn: EventThread
>> shut down
>> 2011-09-12 09:07:16,289 INFO
>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting;
>> hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-15,5,main]
>> 2011-09-12 09:07:16,289 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
>> Mon Sep 12 09:14:26 MDT 2011 Killing regionserver
>>
>> Any ideas as to why the region servers can't shutdown? My java version is:
>>
>> [root@ds1 hbase]# java -version
>> java version "1.6.0_21"
>> Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
>> Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
>>
>> Thanks,
>>
>> ~Jeff
>>
>> [root@ds1 ~]# jstack 18789
>> 18789: Unable to open socket file: target process not responding or HotSpot
>> VM not loaded
>> The -F option can be used when the target process is not responding
>> [root@ds1 ~]# jstack -F 18789
>> Attaching to process ID 18789, please wait...
>> Debugger attached successfully.
>> Server compiler detected.
>> JVM version is 17.0-b16
>> Deadlock Detection:
>>
>> No deadlocks found.
>>
>> Thread 17578: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 17322: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 16419: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 16417: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 19017: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18901: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18871: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18870: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18815: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18868: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18843: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18840: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18823: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18822: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18821: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> Thread 18820: (state = BLOCKED)
>> Error occurred during stack walking:
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
>> at
>> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
>> at
>> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> at sun.tools.jstack.JStack.main(JStack.java:84)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
>> failed for a lwp
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
>> Method)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
>> at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>>
>>
>> --
>> Jeff Whiting
>> Qualtrics Senior Software Engineer
>> jeffw@qualtrics.com
>>
>>
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: HRegionServer not shutting down
Posted by Stack <st...@duboce.net>.
Yes. I'd think the RS is done. What happens if you do 'kill -QUIT
<PID>'. Do you get output in the .out log? I'd be interested in
checking it out Jeff. I wonder if it:
HBASE-4159 HBaseServer - IPC Reader threads are not daemons (Douglas
Campbell)
St.Ack
On Mon, Sep 12, 2011 at 8:24 AM, Jeff Whiting <je...@qualtrics.com> wrote:
> We are running cdh3u1 on centos 5.6 and we have frequent problems shutting
> down region servers. They get stuck and the only thing I can do to stop
> them is a kill -9 which is a little harsh. I tried doing a normal jstack
> but it doesn't respond. With the -F option I get what you see at the bottom
> of the email. Also in the region server logs I see the following which
> would imply that it should be shutdown.
>
> 2011-09-11 13:16:18,692 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> requests=0, regions=16, stores=27, storefiles=17, storefileIndexSize=1,
> memstoreSize=10, compactionQueueSize=0, flushQueueSize=0, usedHeap=609,
> maxHeap=795, blockCacheSize=130088528, blockCacheFree=36818560,
> blockCacheCount=1055, blockCacheHitCount=2038243, blockCacheMissCount=92093,
> blockCacheEvictedCount=600, blockCacheHitRatio=95,
> blockCacheHitCachingRatio=99
> 2011-09-11 13:16:18,692 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: connection to
> cluster: dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef
> connection to cluster:
> dev2-0x331633b7ddb08ef-0x331633b7ddb08ef-0x331633b7ddb08ef received expired
> from ZooKeeper, aborting
> 2011-09-11 13:16:18,692 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2011-09-12 09:07:16,289 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting;
> hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-15,5,main]
> 2011-09-12 09:07:16,289 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
> Mon Sep 12 09:14:26 MDT 2011 Killing regionserver
>
> Any ideas as to why the region servers can't shutdown? My java version is:
>
> [root@ds1 hbase]# java -version
> java version "1.6.0_21"
> Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
>
> Thanks,
>
> ~Jeff
>
> [root@ds1 ~]# jstack 18789
> 18789: Unable to open socket file: target process not responding or HotSpot
> VM not loaded
> The -F option can be used when the target process is not responding
> [root@ds1 ~]# jstack -F 18789
> Attaching to process ID 18789, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 17.0-b16
> Deadlock Detection:
>
> No deadlocks found.
>
> Thread 17578: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 17322: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 16419: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 16417: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 19017: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18901: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18871: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18870: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18815: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18868: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18843: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18840: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18823: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18822: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18821: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> Thread 18820: (state = BLOCKED)
> Error occurred during stack walking:
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
> failed for a lwp
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
> Method)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
>
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com
>
>