You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Edson Ramiro <er...@gmail.com> on 2010/04/06 21:44:25 UTC
Jetty can't start the SelectChannelConnector
Hi all,
I configured the Hadoop in a cluster and the NameNode and JobTracker are
running ok, but the DataNode and TaskTracker Doesn't start, they stop and
keep waiting
when they are going to start the Jetty
I observed that Jetty can't start the _SelectChannelConnector_
Is there any Jetty configuration that should be changed ?
There is no log message in the NN and JT when I try to start the DN and TT.
The kernel I'm using is:
Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64 GNU/Linux
This is the message when I start the DN. It happens with TT too.
ramiro@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode
10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.1+169.56
STARTUP_MSG: build = -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3;
compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010
************************************************************/
10/04/06 16:24:14 INFO datanode.DataNode: Registered FSDatasetStatusMBean
10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at 50010
10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is 1048576
bytes/s
10/04/06 16:24:14 INFO mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
10/04/06 16:24:14 INFO http.HttpServer: Port returned by
webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the
listener on 50075
10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort() returned
50075 webServer.getConnectors()[0].getLocalPort() returned 50075
10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075
10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14
Thanks in Advance,
Edson Ramiro
Re: Jetty can't start the SelectChannelConnector
Posted by Edson Ramiro <er...@gmail.com>.
Hi Todd,
We use KVM to create the machines and (I don't know why) it was not
generating entropy.
We installed rngd, and now its all working ok.
Thanks a lot,
Edson Ramiro
On 6 April 2010 20:15, Todd Lipcon <to...@cloudera.com> wrote:
> The JVM uses "real entropy" to seed its random number generator. If
> the kernel's entropy pool is dry, it will block in SecureRandom
> waiting for entropy to show up.
>
> You can generate it by encouraging some network traffic or disk IO, or
> by faking it using the technique in that post.
>
> -Todd
>
> On Tue, Apr 6, 2010 at 4:12 PM, Edson Ramiro <er...@gmail.com> wrote:
> > Thanks Todd,
> >
> > but could you please explain why entropy is important ?
> >
> >
> > Edson Ramiro
> >
> >
> > On 6 April 2010 20:09, Todd Lipcon <to...@cloudera.com> wrote:
> >
> >> Not enough entropy on your system - you need to generate entropy or
> >> fake some using this technique:
> >>
> >>
> >>
> http://www.chrissearle.org/blog/technical/increase_entropy_26_kernel_linux_box
> >>
> >> -Todd
> >>
> >>
> >> On Tue, Apr 6, 2010 at 4:05 PM, Edson Ramiro <er...@gmail.com>
> wrote:
> >> > ok,
> >> >
> >> >
> >> > [erlf07@cohiba ~ ]$ cat java_native_frames_dn
> >> > Attaching to process ID 3836, please wait...
> >> > Debugger attached successfully.
> >> > Server compiler detected.
> >> > JVM version is 11.2-b01
> >> > Deadlock Detection:
> >> >
> >> > No deadlocks found.
> >> >
> >> > ----------------- 3864 -----------------
> >> > 0xf775a4f1 __libc_read + 0x41
> >> > 0xf702298c readBytes + 0xdc
> >> > 0xf701e717 Java_java_io_FileInputStream_readBytes + 0x47
> >> > 0xf400b4aa * java.io.FileInputStream.readBytes(byte[], int, int)
> bci:0
> >> > (Interpreted frame)
> >> > 0xf4003f69 * java.io.FileInputStream.read(byte[], int, int) bci:4
> >> > line:199 (Interpreted frame)
> >> > 0xf4003f69 * java.io.BufferedInputStream.read1(byte[], int, int)
> >> bci:39
> >> > line:256 (Interpreted frame)
> >> > 0xf4003f69 * java.io.BufferedInputStream.read(byte[], int, int)
> bci:49
> >> > line:317 (Interpreted frame)
> >> > 0xf4003f69 * java.io.BufferedInputStream.fill() bci:175 line:218
> >> > (Interpreted frame)
> >> > 0xf400408d * java.io.BufferedInputStream.read1(byte[], int, int)
> >> bci:44
> >> > line:258 (Interpreted frame)
> >> > 0xf4003f69 * java.io.BufferedInputStream.read(byte[], int, int)
> bci:49
> >> > line:317 (Interpreted frame)
> >> > 0xf4003f69 *
> >> > sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
> bci:12
> >> > line:453 (Interpreted frame)
> >> > 0xf4003e61 *
> sun.security.provider.SeedGenerator.getSeedBytes(byte[])
> >> > bci:11 line:123 (Interpreted frame)
> >> > 0xf400408d *
> sun.security.provider.SeedGenerator.generateSeed(byte[])
> >> > bci:4 line:118 (Interpreted frame)
> >> > 0xf400408d *
> >> sun.security.provider.SecureRandom.engineGenerateSeed(int)
> >> > bci:5 line:114 (Interpreted frame)
> >> > 0xf4003f27 *
> >> sun.security.provider.SecureRandom.engineNextBytes(byte[])
> >> > bci:40 line:171 (Interpreted frame)
> >> > 0xf400408d * java.security.SecureRandom.nextBytes(byte[]) bci:5
> >> line:433
> >> > (Interpreted frame)
> >> > 0xf400408d * java.security.SecureRandom.next(int) bci:17 line:455
> >> > (Interpreted frame)
> >> > 0xf4003f69 * java.util.Random.nextLong() bci:3 line:284
> (Interpreted
> >> > frame)
> >> > 0xf4003fab *
> org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
> >> > bci:73 line:139 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf4004569 *
> >> org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
> >> > bci:96 line:168 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.jetty.servlet.HashSessionManager.doStart()
> >> > bci:12 line:67 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf4004569 * org.mortbay.jetty.servlet.SessionHandler.doStart()
> bci:4
> >> > line:115 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf4004569 * org.mortbay.jetty.handler.HandlerWrapper.doStart()
> bci:11
> >> > line:130 (Interpreted frame)
> >> > 0xf400408d *
> org.mortbay.jetty.handler.ContextHandler.startContext()
> >> > bci:1 line:537 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.jetty.servlet.Context.startContext() bci:1
> >> > line:136 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.jetty.webapp.WebAppContext.startContext()
> >> > bci:123 line:1234 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.jetty.handler.ContextHandler.doStart()
> >> bci:140
> >> > line:517 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.jetty.webapp.WebAppContext.doStart()
> bci:170
> >> > line:460 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf4004569 * org.mortbay.jetty.handler.HandlerCollection.doStart()
> >> bci:32
> >> > line:152 (Interpreted frame)
> >> > 0xf400408d *
> >> org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
> >> > bci:5 line:156 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf4004569 * org.mortbay.jetty.handler.HandlerWrapper.doStart()
> bci:11
> >> > line:130 (Interpreted frame)
> >> > 0xf400408d * org.mortbay.jetty.Server.doStart() bci:201 line:222
> >> > (Interpreted frame)
> >> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf400408d * org.apache.hadoop.http.HttpServer.start() bci:383
> >> line:461
> >> > (Interpreted frame)
> >> > 0xf400408d *
> >> >
> >>
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
> >> > java.util.AbstractList) bci:916 line:375 (Interpreted frame)
> >> > 0xf400408d *
> >> >
> >>
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
> >> > java.util.AbstractList) bci:158 line:216 (Interpreted frame)
> >> > 0xf400408d *
> >> >
> >>
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
> >> > org.apache.hadoop.conf.Configuration) bci:96 line:1283 (Interpreted
> >> frame)
> >> > 0xf4003f27 *
> >> >
> >>
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
> >> > org.apache.hadoop.conf.Configuration) bci:92 line:1238 (Interpreted
> >> frame)
> >> > 0xf4003f27 *
> >> >
> >>
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
> >> > org.apache.hadoop.conf.Configuration) bci:2 line:1246 (Interpreted
> frame)
> >> > 0xf4003f27 *
> >> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
> >> > bci:12 line:1368 (Interpreted frame)
> >> > 0xf400134c <StubRoutines>
> >> > 0x06375c9d
> >> >
> >>
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> >> > + 0x15d
> >> > 0x065057f8
> >> >
> >>
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> >> > + 0x18
> >> > 0x06375b30
> >> >
> _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread
> >> +
> >> > 0x20
> >> > 0x0639f2e6
> >> >
> >>
> _Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
> >> > + 0x116
> >> > 0x06390ae8 jni_CallStaticVoidMethod + 0xf8
> >> > 0x08049b98 JavaMain + 0x2c8
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3865 -----------------
> >> > 0xf77578d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> >> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> >> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> >> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3866 -----------------
> >> > 0xf77578d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> >> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> >> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> >> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3869 -----------------
> >> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
> >> > 0x39d73c58 ????????
> >> > sun.jvm.hotspot.debugger.UnmappedAddressException
> >> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> >> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> >> > at
> >> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> >> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> >> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> >> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> > at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> >> > at sun.tools.jstack.JStack.main(JStack.java:84)
> >> > ----------------- 3870 -----------------
> >> > 0xf77578d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> >> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> >> > 0x063c834a JVM_MonitorWait + 0x1da
> >> > 0xf400b4aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> >> > 0xf400408d * java.lang.Object.wait() bci:2 line:485 (Interpreted
> >> frame)
> >> > 0xf400408d * java.lang.ref.Reference$ReferenceHandler.run() bci:46
> >> > line:116 (Interpreted frame)
> >> > 0xf400134c <StubRoutines>
> >> > 0x06375c9d
> >> >
> >>
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> >> > + 0x15d
> >> > 0x065057f8
> >> >
> >>
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> >> > + 0x18
> >> > 0x063755b0
> >> >
> >>
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> >> > + 0xa0
> >> > 0x0637563d
> >> >
> >>
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> >> > + 0x5d
> >> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3871 -----------------
> >> > 0xf77578d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> >> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> >> > 0x063c834a JVM_MonitorWait + 0x1da
> >> > 0xf400b4aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> >> > 0xf400408d * java.lang.ref.ReferenceQueue.remove(long) bci:44
> line:116
> >> > (Interpreted frame)
> >> > 0xf4003f27 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
> >> > (Interpreted frame)
> >> > 0xf4003f27 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3
> >> line:159
> >> > (Interpreted frame)
> >> > 0xf400134c <StubRoutines>
> >> > 0x06375c9d
> >> >
> >>
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> >> > + 0x15d
> >> > 0x065057f8
> >> >
> >>
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> >> > + 0x18
> >> > 0x063755b0
> >> >
> >>
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> >> > + 0xa0
> >> > 0x0637563d
> >> >
> >>
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> >> > + 0x5d
> >> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3872 -----------------
> >> > 0xf77598b0 __new_sem_wait + 0x40
> >> > 0x06503c0d _ZN2os11signal_waitEv + 0xd
> >> > 0x06500d28 _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3873 -----------------
> >> > 0xf77578d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> >> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> >> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> >> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3874 -----------------
> >> > 0xf77578d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> >> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> >> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> >> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3875 -----------------
> >> > 0xf77578d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> >> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> >> > 0x064bc935
> >> >
> >>
> _ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
> >> > + 0x85
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3878 -----------------
> >> > 0xf775a6fe __libc_accept + 0x3e
> >> > 0xb0c41a19 Java_java_net_PlainSocketImpl_socketAccept + 0x129
> >> > 0xf400b4aa *
> >> java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl)
> >> > bci:0 (Interpreted frame)
> >> > 0xf400408d * java.net.PlainSocketImpl.accept(java.net.SocketImpl)
> >> bci:7
> >> > line:384 (Interpreted frame)
> >> > 0xf400408d * java.net.ServerSocket.implAccept(java.net.Socket)
> bci:60
> >> > line:453 (Interpreted frame)
> >> > 0xf400408d * java.net.ServerSocket.accept() bci:48 line:421
> >> (Interpreted
> >> > frame)
> >> > 0xf4003f27 *
> >> > sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() bci:1
> >> > line:34 (Interpreted frame)
> >> > 0xf4003f27 *
> >> > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop()
> bci:55
> >> > line:369 (Interpreted frame)
> >> > 0xf400408d * sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run()
> bci:1
> >> > line:341 (Interpreted frame)
> >> > 0xf4004569 * java.lang.Thread.run() bci:11 line:619 (Interpreted
> >> frame)
> >> > 0xf400134c <StubRoutines>
> >> > 0x06375c9d
> >> >
> >>
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> >> > + 0x15d
> >> > 0x065057f8
> >> >
> >>
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> >> > + 0x18
> >> > 0x063755b0
> >> >
> >>
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> >> > + 0xa0
> >> > 0x0637563d
> >> >
> >>
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> >> > + 0x5d
> >> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3879 -----------------
> >> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
> >> > 0x1bef7958 ????????
> >> > sun.jvm.hotspot.debugger.UnmappedAddressException
> >> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> >> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> >> > at
> >> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> >> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> >> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> >> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> > at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> >> > at sun.tools.jstack.JStack.main(JStack.java:84)
> >> > ----------------- 4055 -----------------
> >> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
> >> > 0x14e31390 ????????
> >> > sun.jvm.hotspot.debugger.UnmappedAddressException
> >> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> >> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> >> > at
> >> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> >> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> >> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> >> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> > at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> >> > at sun.tools.jstack.JStack.main(JStack.java:84)
> >> > ----------------- 4060 -----------------
> >> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
> >> > 0x358d0f88 ????????
> >> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> >> > address: 0x4bbbaf53 with alignment: 4
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> >> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> >> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> >> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> > at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> >> > at sun.tools.jstack.JStack.main(JStack.java:84)
> >> > ----------------- 4062 -----------------
> >> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
> >> > 0x35c10c20 ????????
> >> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> >> > address: 0x4bbbaf53 with alignment: 4
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> >> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> >> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> >> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> > at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> >> > at sun.tools.jstack.JStack.main(JStack.java:84)
> >> > ----------------- 7801 -----------------
> >> > 0xf775a6fe __libc_accept + 0x3e
> >> > 0x061c1a78 _ZN14AttachListener7dequeueEv + 0x68
> >> > 0x061c0e55 _Z28attach_listener_thread_entryP10JavaThreadP6Thread +
> >> 0x35
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf7753f50 start_thread + 0xa0
> >> > ----------------- 3836 -----------------
> >> > 0xf7754452 pthread_join + 0x112
> >> > 0x0804dd78 ContinueInNewThread + 0x88
> >> > 0x080497f6 main + 0x276
> >> > 0xf7614775 ????????
> >> >
> >> >
> >> >
> >> > [erlf07@cohiba ~ ]$ cat java_native_frames_tt
> >> > Attaching to process ID 22510, please wait...
> >> > Debugger attached successfully.
> >> > Server compiler detected.
> >> > JVM version is 11.2-b01
> >> > Deadlock Detection:
> >> >
> >> > No deadlocks found.
> >> >
> >> > ----------------- 22544 -----------------
> >> > 0xf76e34f1 __libc_read + 0x41
> >> > 0xf6fab98c readBytes + 0xdc
> >> > 0xf6fa7717 Java_java_io_FileInputStream_readBytes + 0x47
> >> > 0xf3f944aa * java.io.FileInputStream.readBytes(byte[], int, int)
> bci:0
> >> > (Interpreted frame)
> >> > 0xf3f8cf69 * java.io.FileInputStream.read(byte[], int, int) bci:4
> >> > line:199 (Interpreted frame)
> >> > 0xf3f8cf69 * java.io.BufferedInputStream.read1(byte[], int, int)
> >> bci:39
> >> > line:256 (Interpreted frame)
> >> > 0xf3f8cf69 * java.io.BufferedInputStream.read(byte[], int, int)
> bci:49
> >> > line:317 (Interpreted frame)
> >> > 0xf3f8cf69 * java.io.BufferedInputStream.fill() bci:175 line:218
> >> > (Interpreted frame)
> >> > 0xf3f8d08d * java.io.BufferedInputStream.read1(byte[], int, int)
> >> bci:44
> >> > line:258 (Interpreted frame)
> >> > 0xf3f8cf69 * java.io.BufferedInputStream.read(byte[], int, int)
> bci:49
> >> > line:317 (Interpreted frame)
> >> > 0xf3f8cf69 *
> >> > sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
> bci:12
> >> > line:453 (Interpreted frame)
> >> > 0xf3f8ce61 *
> sun.security.provider.SeedGenerator.getSeedBytes(byte[])
> >> > bci:11 line:123 (Interpreted frame)
> >> > 0xf3f8d08d *
> sun.security.provider.SeedGenerator.generateSeed(byte[])
> >> > bci:4 line:118 (Interpreted frame)
> >> > 0xf3f8d08d *
> >> sun.security.provider.SecureRandom.engineGenerateSeed(int)
> >> > bci:5 line:114 (Interpreted frame)
> >> > 0xf3f8cf27 *
> >> sun.security.provider.SecureRandom.engineNextBytes(byte[])
> >> > bci:40 line:171 (Interpreted frame)
> >> > 0xf3f8d08d * java.security.SecureRandom.nextBytes(byte[]) bci:5
> >> line:433
> >> > (Interpreted frame)
> >> > 0xf3f8d08d * java.security.SecureRandom.next(int) bci:17 line:455
> >> > (Interpreted frame)
> >> > 0xf3f8cf69 * java.util.Random.nextLong() bci:3 line:284
> (Interpreted
> >> > frame)
> >> > 0xf3f8cfab *
> org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
> >> > bci:73 line:139 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf3f8d569 *
> >> org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
> >> > bci:96 line:168 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.jetty.servlet.HashSessionManager.doStart()
> >> > bci:12 line:67 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf3f8d569 * org.mortbay.jetty.servlet.SessionHandler.doStart()
> bci:4
> >> > line:115 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf3f8d569 * org.mortbay.jetty.handler.HandlerWrapper.doStart()
> bci:11
> >> > line:130 (Interpreted frame)
> >> > 0xf3f8d08d *
> org.mortbay.jetty.handler.ContextHandler.startContext()
> >> > bci:1 line:537 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.jetty.servlet.Context.startContext() bci:1
> >> > line:136 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.jetty.webapp.WebAppContext.startContext()
> >> > bci:123 line:1234 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.jetty.handler.ContextHandler.doStart()
> >> bci:140
> >> > line:517 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.jetty.webapp.WebAppContext.doStart()
> bci:170
> >> > line:460 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf3f8d569 * org.mortbay.jetty.handler.HandlerCollection.doStart()
> >> bci:32
> >> > line:152 (Interpreted frame)
> >> > 0xf3f8d08d *
> >> org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
> >> > bci:5 line:156 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf3f8d569 * org.mortbay.jetty.handler.HandlerWrapper.doStart()
> bci:11
> >> > line:130 (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.jetty.Server.doStart() bci:201 line:222
> >> > (Interpreted frame)
> >> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> >> > line:50 (Interpreted frame)
> >> > 0xf3f8d08d * org.apache.hadoop.http.HttpServer.start() bci:383
> >> line:461
> >> > (Interpreted frame)
> >> > 0xf3f8d08d *
> >> >
> >>
> org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
> >> > bci:519 line:931 (Interpreted frame)
> >> > 0xf3f8d08d *
> >> > org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) bci:52
> >> > line:2833 (Interpreted frame)
> >> > 0xf3f8a34c <StubRoutines>
> >> > 0x06375c9d
> >> >
> >>
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> >> > + 0x15d
> >> > 0x065057f8
> >> >
> >>
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> >> > + 0x18
> >> > 0x06375b30
> >> >
> _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread
> >> +
> >> > 0x20
> >> > 0x0639f2e6
> >> >
> >>
> _Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
> >> > + 0x116
> >> > 0x06390ae8 jni_CallStaticVoidMethod + 0xf8
> >> > 0x08049b98 JavaMain + 0x2c8
> >> > 0xf76dcf50 start_thread + 0xa0
> >> > ----------------- 22545 -----------------
> >> > 0xf76e08d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> >> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> >> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> >> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf76dcf50 start_thread + 0xa0
> >> > ----------------- 22546 -----------------
> >> > 0xf76e08d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> >> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> >> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> >> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf76dcf50 start_thread + 0xa0
> >> > ----------------- 22549 -----------------
> >> > 0xf76e0be8 __pthread_cond_timedwait + 0xe8
> >> > 0x340707e0 ????????
> >> > sun.jvm.hotspot.debugger.UnmappedAddressException
> >> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> >> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> >> > at
> >> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> >> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> >> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> >> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> > at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> >> > at sun.tools.jstack.JStack.main(JStack.java:84)
> >> > ----------------- 22550 -----------------
> >> > 0xf76e08d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> >> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> >> > 0x063c834a JVM_MonitorWait + 0x1da
> >> > 0xf3f944aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> >> > 0xf3f8d08d * java.lang.Object.wait() bci:2 line:485 (Interpreted
> >> frame)
> >> > 0xf3f8d08d * java.lang.ref.Reference$ReferenceHandler.run() bci:46
> >> > line:116 (Interpreted frame)
> >> > 0xf3f8a34c <StubRoutines>
> >> > 0x06375c9d
> >> >
> >>
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> >> > + 0x15d
> >> > 0x065057f8
> >> >
> >>
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> >> > + 0x18
> >> > 0x063755b0
> >> >
> >>
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> >> > + 0xa0
> >> > 0x0637563d
> >> >
> >>
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> >> > + 0x5d
> >> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf76dcf50 start_thread + 0xa0
> >> > ----------------- 22551 -----------------
> >> > 0xf76e08d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> >> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> >> > 0x063c834a JVM_MonitorWait + 0x1da
> >> > 0xf3f944aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> >> > 0xf3f8d08d * java.lang.ref.ReferenceQueue.remove(long) bci:44
> line:116
> >> > (Interpreted frame)
> >> > 0xf3f8cf27 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
> >> > (Interpreted frame)
> >> > 0xf3f8cf27 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3
> >> line:159
> >> > (Interpreted frame)
> >> > 0xf3f8a34c <StubRoutines>
> >> > 0x06375c9d
> >> >
> >>
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> >> > + 0x15d
> >> > 0x065057f8
> >> >
> >>
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> >> > + 0x18
> >> > 0x063755b0
> >> >
> >>
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> >> > + 0xa0
> >> > 0x0637563d
> >> >
> >>
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> >> > + 0x5d
> >> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf76dcf50 start_thread + 0xa0
> >> > ----------------- 22554 -----------------
> >> > 0xf76e28b0 __new_sem_wait + 0x40
> >> > 0x06503c0d _ZN2os11signal_waitEv + 0xd
> >> > 0x06500d28 _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf76dcf50 start_thread + 0xa0
> >> > ----------------- 22555 -----------------
> >> > 0xf76e08d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> >> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> >> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> >> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf76dcf50 start_thread + 0xa0
> >> > ----------------- 22556 -----------------
> >> > 0xf76e08d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> >> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> >> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> >> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf76dcf50 start_thread + 0xa0
> >> > ----------------- 22557 -----------------
> >> > 0xf76e08d0 __pthread_cond_wait + 0x90
> >> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> >> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> >> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> >> > 0x064bc935
> >> >
> >>
> _ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
> >> > + 0x85
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf76dcf50 start_thread + 0xa0
> >> > ----------------- 22558 -----------------
> >> > 0xf76e0be8 __pthread_cond_timedwait + 0xe8
> >> > 0x38ed9008 ????????
> >> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> >> > address: 0x4bbbb06f with alignment: 4
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> >> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> >> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> >> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> > at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> >> > at sun.tools.jstack.JStack.main(JStack.java:84)
> >> > ----------------- 22644 -----------------
> >> > 0xf76e0be8 __pthread_cond_timedwait + 0xe8
> >> > 0x07b9bcc8 ????????
> >> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> >> > address: 0x4bbbb087 with alignment: 4
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> >> > at
> >> >
> >>
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> >> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> >> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> >> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> >> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> > at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> >> > at sun.tools.jstack.JStack.main(JStack.java:84)
> >> > ----------------- 24136 -----------------
> >> > 0xf76e36fe __libc_accept + 0x3e
> >> > 0x061c1a78 _ZN14AttachListener7dequeueEv + 0x68
> >> > 0x061c0e55 _Z28attach_listener_thread_entryP10JavaThreadP6Thread +
> >> 0x35
> >> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> >> > 0x06506929 _Z10java_startP6Thread + 0x139
> >> > 0xf76dcf50 start_thread + 0xa0
> >> > ----------------- 22510 -----------------
> >> > 0xf76dd452 pthread_join + 0x112
> >> > 0x0804dd78 ContinueInNewThread + 0x88
> >> > 0x080497f6 main + 0x276
> >> > 0xf759d775 ????????
> >> >
> >> >
> >> > [erlf07@cohiba ~ ]$ cat thread_dump_dn
> >> > Attaching to process ID 3836, please wait...
> >> > Debugger attached successfully.
> >> > Server compiler detected.
> >> > JVM version is 11.2-b01
> >> > Deadlock Detection:
> >> >
> >> > No deadlocks found.
> >> >
> >> > Thread 7801: (state = BLOCKED)
> >> >
> >> >
> >> > Thread 4062: (state = BLOCKED)
> >> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> >> > - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275,
> >> line=565
> >> > (Interpreted frame)
> >> >
> >> >
> >> > Thread 4060: (state = BLOCKED)
> >> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> >> > - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275,
> >> line=565
> >> > (Interpreted frame)
> >> >
> >> >
> >> > Thread 4055: (state = BLOCKED)
> >> > - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
> >> > - org.apache.hadoop.fs.DU$DURefreshThread.run() @bci=17, line=80
> >> > (Interpreted frame)
> >> > - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
> >> >
> >> >
> >> > Thread 3878: (state = IN_NATIVE)
> >> > - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0
> >> > (Interpreted frame)
> >> > - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7,
> line=384
> >> > (Interpreted frame)
> >> > - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=453
> >> > (Interpreted frame)
> >> > - java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted
> frame)
> >> > - sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept()
> >> @bci=1,
> >> > line=34 (Interpreted frame)
> >> > - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop()
> >> > @bci=55, line=369 (Interpreted frame)
> >> > - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1,
> line=341
> >> > (Interpreted frame)
> >> > - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
> >> >
> >> >
> >> > Thread 3872: (state = BLOCKED)
> >> >
> >> >
> >> > Thread 3871: (state = BLOCKED)
> >> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> >> > - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116
> >> (Interpreted
> >> > frame)
> >> > - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
> >> > frame)
> >> > - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
> >> > (Interpreted frame)
> >> >
> >> >
> >> > Thread 3870: (state = BLOCKED)
> >> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> >> > - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
> >> > - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
> >> > (Interpreted frame)
> >> >
> >> >
> >> > Thread 3864: (state = IN_NATIVE)
> >> > - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0
> >> (Interpreted
> >> > frame)
> >> > - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
> >> > (Interpreted frame)
> >> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39,
> line=256
> >> > (Interpreted frame)
> >> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49,
> line=317
> >> > (Interpreted frame)
> >> > - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted
> >> frame)
> >> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44,
> line=258
> >> > (Interpreted frame)
> >> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49,
> line=317
> >> > (Interpreted frame)
> >> > - sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
> >> > @bci=12, line=453 (Interpreted frame)
> >> > - sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
> >> > line=123 (Interpreted frame)
> >> > - sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4,
> >> line=118
> >> > (Interpreted frame)
> >> > - sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
> >> > line=114 (Interpreted frame)
> >> > - sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
> >> > line=171 (Interpreted frame)
> >> > - java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
> >> > (Interpreted frame)
> >> > - java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
> >> > frame)
> >> > - java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
> >> > - org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
> >> > line=139 (Interpreted frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
> >> > line=168 (Interpreted frame)
> >> > - org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12,
> >> line=67
> >> > (Interpreted frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
> >> > (Interpreted frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11,
> line=130
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1,
> >> line=537
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123,
> >> line=1234
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140,
> line=517
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
> >> > (Interpreted frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32,
> >> line=152
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
> @bci=5,
> >> > line=156 (Interpreted frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11,
> line=130
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted
> >> frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.apache.hadoop.http.HttpServer.start() @bci=383, line=461
> >> (Interpreted
> >> > frame)
> >> > -
> >> >
> >>
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
> >> > java.util.AbstractList) @bci=916, line=375 (Interpreted frame)
> >> > -
> >> >
> >>
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
> >> > java.util.AbstractList) @bci=158, line=216 (Interpreted frame)
> >> > -
> >> >
> >>
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
> >> > org.apache.hadoop.conf.Configuration) @bci=96, line=1283 (Interpreted
> >> frame)
> >> > -
> >> >
> >>
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
> >> > org.apache.hadoop.conf.Configuration) @bci=92, line=1238 (Interpreted
> >> frame)
> >> > -
> >> >
> >>
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
> >> > org.apache.hadoop.conf.Configuration) @bci=2, line=1246 (Interpreted
> >> frame)
> >> > -
> >> org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
> >> > @bci=12, line=1368 (Interpreted frame)
> >> >
> >> >
> >> >
> >> >
> >> > [erlf07@cohiba ~ ]$ cat thread_dump_tt
> >> > Attaching to process ID 22510, please wait...
> >> > Debugger attached successfully.
> >> > Server compiler detected.
> >> > JVM version is 11.2-b01
> >> > Deadlock Detection:
> >> >
> >> > No deadlocks found.
> >> >
> >> > Thread 24136: (state = BLOCKED)
> >> >
> >> >
> >> > Thread 22644: (state = BLOCKED)
> >> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> >> > - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275,
> >> line=565
> >> > (Interpreted frame)
> >> >
> >> >
> >> > Thread 22554: (state = BLOCKED)
> >> >
> >> >
> >> > Thread 22551: (state = BLOCKED)
> >> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> >> > - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116
> >> (Interpreted
> >> > frame)
> >> > - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
> >> > frame)
> >> > - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
> >> > (Interpreted frame)
> >> >
> >> >
> >> > Thread 22550: (state = BLOCKED)
> >> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> >> > - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
> >> > - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
> >> > (Interpreted frame)
> >> >
> >> >
> >> > Thread 22544: (state = IN_NATIVE)
> >> > - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0
> >> (Interpreted
> >> > frame)
> >> > - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
> >> > (Interpreted frame)
> >> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39,
> line=256
> >> > (Interpreted frame)
> >> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49,
> line=317
> >> > (Interpreted frame)
> >> > - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted
> >> frame)
> >> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44,
> line=258
> >> > (Interpreted frame)
> >> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49,
> line=317
> >> > (Interpreted frame)
> >> > - sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
> >> > @bci=12, line=453 (Interpreted frame)
> >> > - sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
> >> > line=123 (Interpreted frame)
> >> > - sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4,
> >> line=118
> >> > (Interpreted frame)
> >> > - sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
> >> > line=114 (Interpreted frame)
> >> > - sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
> >> > line=171 (Interpreted frame)
> >> > - java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
> >> > (Interpreted frame)
> >> > - java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
> >> > frame)
> >> > - java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
> >> > - org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
> >> > line=139 (Interpreted frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
> >> > line=168 (Interpreted frame)
> >> > - org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12,
> >> line=67
> >> > (Interpreted frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
> >> > (Interpreted frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11,
> line=130
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1,
> >> line=537
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123,
> >> line=1234
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140,
> line=517
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
> >> > (Interpreted frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32,
> >> line=152
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
> @bci=5,
> >> > line=156 (Interpreted frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11,
> line=130
> >> > (Interpreted frame)
> >> > - org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted
> >> frame)
> >> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> >> > (Interpreted frame)
> >> > - org.apache.hadoop.http.HttpServer.start() @bci=383, line=461
> >> (Interpreted
> >> > frame)
> >> > -
> >> >
> >>
> org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
> >> > @bci=519, line=931 (Interpreted frame)
> >> > - org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[])
> @bci=52,
> >> > line=2833 (Interpreted frame)
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Edson Ramiro
> >> >
> >> >
> >> > On 6 April 2010 19:47, Todd Lipcon <to...@cloudera.com> wrote:
> >> >
> >> >> Hi Edson,
> >> >>
> >> >> Your attachments did not come through - can you put them on pastebin?
> >> >>
> >> >> -Todd
> >> >>
> >> >> On Tue, Apr 6, 2010 at 3:37 PM, Edson Ramiro <er...@gmail.com>
> >> wrote:
> >> >> > Hi Todd,
> >> >> >
> >> >> > I'm getting this behavior in another cluster too, there the same
> thing
> >> >> > happens.
> >> >> > and as I don't have a jstack installed in the first cluster and
> >> >> > I'm not the admin, I'm sending the results of the second cluster.
> >> >> >
> >> >> > These are the results:
> >> >> >
> >> >> > [erlf07@cohiba ~ ]$ jstack -l 22510
> >> >> > 22510: well-known file is not secure
> >> >> > [erlf07@cohiba ~ ]$ jstack -l 3836
> >> >> > 3836: well-known file is not secure
> >> >> >
> >> >> > The jstack -F result is in thread_dump files and the jstack -m
> result
> >> is
> >> >> in
> >> >> > java_native_frames.
> >> >> >
> >> >> > The files ending with nn are the namenode results and the files
> ending
> >> >> with
> >> >> > dn are the datanode results.
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> > Edson Ramiro
> >> >> >
> >> >> >
> >> >> > On 6 April 2010 18:19, Todd Lipcon <to...@cloudera.com> wrote:
> >> >> >>
> >> >> >> Hi Edson,
> >> >> >>
> >> >> >> Can you please run "jstack" on the daemons in question and paste
> the
> >> >> >> output here?
> >> >> >>
> >> >> >> -Todd
> >> >> >>
> >> >> >> On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro <erlfilho@gmail.com
> >
> >> >> wrote:
> >> >> >> > Hi all,
> >> >> >> >
> >> >> >> > I configured the Hadoop in a cluster and the NameNode and
> >> JobTracker
> >> >> are
> >> >> >> > running ok, but the DataNode and TaskTracker Doesn't start, they
> >> stop
> >> >> >> > and
> >> >> >> > keep waiting
> >> >> >> > when they are going to start the Jetty
> >> >> >> >
> >> >> >> > I observed that Jetty can't start the _SelectChannelConnector_
> >> >> >> >
> >> >> >> > Is there any Jetty configuration that should be changed ?
> >> >> >> >
> >> >> >> > There is no log message in the NN and JT when I try to start the
> DN
> >> >> and
> >> >> >> > TT.
> >> >> >> >
> >> >> >> > The kernel I'm using is:
> >> >> >> > Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64
> >> >> GNU/Linux
> >> >> >> >
> >> >> >> > This is the message when I start the DN. It happens with TT too.
> >> >> >> >
> >> >> >> > ramiro@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode
> >> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG:
> >> >> >> > /************************************************************
> >> >> >> > STARTUP_MSG: Starting DataNode
> >> >> >> > STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115
> >> >> >> > STARTUP_MSG: args = []
> >> >> >> > STARTUP_MSG: version = 0.20.1+169.56
> >> >> >> > STARTUP_MSG: build = -r
> >> 8e662cb065be1c4bc61c55e6bff161e09c1d36f3;
> >> >> >> > compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010
> >> >> >> > ************************************************************/
> >> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Registered
> >> >> >> > FSDatasetStatusMBean
> >> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at
> >> 50010
> >> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is
> >> >> 1048576
> >> >> >> > bytes/s
> >> >> >> > 10/04/06 16:24:14 INFO mortbay.log: Logging to
> >> >> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> >> >> >> > org.mortbay.log.Slf4jLog
> >> >> >> > 10/04/06 16:24:14 INFO http.HttpServer: Port returned by
> >> >> >> > webServer.getConnectors()[0].getLocalPort() before open() is -1.
> >> >> Opening
> >> >> >> > the
> >> >> >> > listener on 50075
> >> >> >> > 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort()
> >> >> returned
> >> >> >> > 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
> >> >> >> > 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port
> 50075
> >> >> >> > 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14
> >> >> >> >
> >> >> >> > Thanks in Advance,
> >> >> >> >
> >> >> >> > Edson Ramiro
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Todd Lipcon
> >> >> >> Software Engineer, Cloudera
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Todd Lipcon
> >> >> Software Engineer, Cloudera
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Jetty can't start the SelectChannelConnector
Posted by Todd Lipcon <to...@cloudera.com>.
The JVM uses "real entropy" to seed its random number generator. If
the kernel's entropy pool is dry, it will block in SecureRandom
waiting for entropy to show up.
You can generate it by encouraging some network traffic or disk IO, or
by faking it using the technique in that post.
-Todd
On Tue, Apr 6, 2010 at 4:12 PM, Edson Ramiro <er...@gmail.com> wrote:
> Thanks Todd,
>
> but could you please explain why entropy is important ?
>
>
> Edson Ramiro
>
>
> On 6 April 2010 20:09, Todd Lipcon <to...@cloudera.com> wrote:
>
>> Not enough entropy on your system - you need to generate entropy or
>> fake some using this technique:
>>
>>
>> http://www.chrissearle.org/blog/technical/increase_entropy_26_kernel_linux_box
>>
>> -Todd
>>
>>
>> On Tue, Apr 6, 2010 at 4:05 PM, Edson Ramiro <er...@gmail.com> wrote:
>> > ok,
>> >
>> >
>> > [erlf07@cohiba ~ ]$ cat java_native_frames_dn
>> > Attaching to process ID 3836, please wait...
>> > Debugger attached successfully.
>> > Server compiler detected.
>> > JVM version is 11.2-b01
>> > Deadlock Detection:
>> >
>> > No deadlocks found.
>> >
>> > ----------------- 3864 -----------------
>> > 0xf775a4f1 __libc_read + 0x41
>> > 0xf702298c readBytes + 0xdc
>> > 0xf701e717 Java_java_io_FileInputStream_readBytes + 0x47
>> > 0xf400b4aa * java.io.FileInputStream.readBytes(byte[], int, int) bci:0
>> > (Interpreted frame)
>> > 0xf4003f69 * java.io.FileInputStream.read(byte[], int, int) bci:4
>> > line:199 (Interpreted frame)
>> > 0xf4003f69 * java.io.BufferedInputStream.read1(byte[], int, int)
>> bci:39
>> > line:256 (Interpreted frame)
>> > 0xf4003f69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
>> > line:317 (Interpreted frame)
>> > 0xf4003f69 * java.io.BufferedInputStream.fill() bci:175 line:218
>> > (Interpreted frame)
>> > 0xf400408d * java.io.BufferedInputStream.read1(byte[], int, int)
>> bci:44
>> > line:258 (Interpreted frame)
>> > 0xf4003f69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
>> > line:317 (Interpreted frame)
>> > 0xf4003f69 *
>> > sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12
>> > line:453 (Interpreted frame)
>> > 0xf4003e61 * sun.security.provider.SeedGenerator.getSeedBytes(byte[])
>> > bci:11 line:123 (Interpreted frame)
>> > 0xf400408d * sun.security.provider.SeedGenerator.generateSeed(byte[])
>> > bci:4 line:118 (Interpreted frame)
>> > 0xf400408d *
>> sun.security.provider.SecureRandom.engineGenerateSeed(int)
>> > bci:5 line:114 (Interpreted frame)
>> > 0xf4003f27 *
>> sun.security.provider.SecureRandom.engineNextBytes(byte[])
>> > bci:40 line:171 (Interpreted frame)
>> > 0xf400408d * java.security.SecureRandom.nextBytes(byte[]) bci:5
>> line:433
>> > (Interpreted frame)
>> > 0xf400408d * java.security.SecureRandom.next(int) bci:17 line:455
>> > (Interpreted frame)
>> > 0xf4003f69 * java.util.Random.nextLong() bci:3 line:284 (Interpreted
>> > frame)
>> > 0xf4003fab * org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
>> > bci:73 line:139 (Interpreted frame)
>> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf4004569 *
>> org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
>> > bci:96 line:168 (Interpreted frame)
>> > 0xf400408d * org.mortbay.jetty.servlet.HashSessionManager.doStart()
>> > bci:12 line:67 (Interpreted frame)
>> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf4004569 * org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4
>> > line:115 (Interpreted frame)
>> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf4004569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
>> > line:130 (Interpreted frame)
>> > 0xf400408d * org.mortbay.jetty.handler.ContextHandler.startContext()
>> > bci:1 line:537 (Interpreted frame)
>> > 0xf400408d * org.mortbay.jetty.servlet.Context.startContext() bci:1
>> > line:136 (Interpreted frame)
>> > 0xf400408d * org.mortbay.jetty.webapp.WebAppContext.startContext()
>> > bci:123 line:1234 (Interpreted frame)
>> > 0xf400408d * org.mortbay.jetty.handler.ContextHandler.doStart()
>> bci:140
>> > line:517 (Interpreted frame)
>> > 0xf400408d * org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170
>> > line:460 (Interpreted frame)
>> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf4004569 * org.mortbay.jetty.handler.HandlerCollection.doStart()
>> bci:32
>> > line:152 (Interpreted frame)
>> > 0xf400408d *
>> org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
>> > bci:5 line:156 (Interpreted frame)
>> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf4004569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
>> > line:130 (Interpreted frame)
>> > 0xf400408d * org.mortbay.jetty.Server.doStart() bci:201 line:222
>> > (Interpreted frame)
>> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf400408d * org.apache.hadoop.http.HttpServer.start() bci:383
>> line:461
>> > (Interpreted frame)
>> > 0xf400408d *
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
>> > java.util.AbstractList) bci:916 line:375 (Interpreted frame)
>> > 0xf400408d *
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
>> > java.util.AbstractList) bci:158 line:216 (Interpreted frame)
>> > 0xf400408d *
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
>> > org.apache.hadoop.conf.Configuration) bci:96 line:1283 (Interpreted
>> frame)
>> > 0xf4003f27 *
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
>> > org.apache.hadoop.conf.Configuration) bci:92 line:1238 (Interpreted
>> frame)
>> > 0xf4003f27 *
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
>> > org.apache.hadoop.conf.Configuration) bci:2 line:1246 (Interpreted frame)
>> > 0xf4003f27 *
>> > org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
>> > bci:12 line:1368 (Interpreted frame)
>> > 0xf400134c <StubRoutines>
>> > 0x06375c9d
>> >
>> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
>> > + 0x15d
>> > 0x065057f8
>> >
>> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
>> > + 0x18
>> > 0x06375b30
>> > _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread
>> +
>> > 0x20
>> > 0x0639f2e6
>> >
>> _Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
>> > + 0x116
>> > 0x06390ae8 jni_CallStaticVoidMethod + 0xf8
>> > 0x08049b98 JavaMain + 0x2c8
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3865 -----------------
>> > 0xf77578d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
>> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
>> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
>> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3866 -----------------
>> > 0xf77578d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
>> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
>> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
>> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3869 -----------------
>> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
>> > 0x39d73c58 ????????
>> > sun.jvm.hotspot.debugger.UnmappedAddressException
>> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
>> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
>> > at
>> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
>> > at
>> >
>> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
>> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> > at sun.tools.jstack.JStack.main(JStack.java:84)
>> > ----------------- 3870 -----------------
>> > 0xf77578d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
>> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
>> > 0x063c834a JVM_MonitorWait + 0x1da
>> > 0xf400b4aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
>> > 0xf400408d * java.lang.Object.wait() bci:2 line:485 (Interpreted
>> frame)
>> > 0xf400408d * java.lang.ref.Reference$ReferenceHandler.run() bci:46
>> > line:116 (Interpreted frame)
>> > 0xf400134c <StubRoutines>
>> > 0x06375c9d
>> >
>> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
>> > + 0x15d
>> > 0x065057f8
>> >
>> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
>> > + 0x18
>> > 0x063755b0
>> >
>> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
>> > + 0xa0
>> > 0x0637563d
>> >
>> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
>> > + 0x5d
>> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3871 -----------------
>> > 0xf77578d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
>> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
>> > 0x063c834a JVM_MonitorWait + 0x1da
>> > 0xf400b4aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
>> > 0xf400408d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116
>> > (Interpreted frame)
>> > 0xf4003f27 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
>> > (Interpreted frame)
>> > 0xf4003f27 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3
>> line:159
>> > (Interpreted frame)
>> > 0xf400134c <StubRoutines>
>> > 0x06375c9d
>> >
>> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
>> > + 0x15d
>> > 0x065057f8
>> >
>> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
>> > + 0x18
>> > 0x063755b0
>> >
>> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
>> > + 0xa0
>> > 0x0637563d
>> >
>> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
>> > + 0x5d
>> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3872 -----------------
>> > 0xf77598b0 __new_sem_wait + 0x40
>> > 0x06503c0d _ZN2os11signal_waitEv + 0xd
>> > 0x06500d28 _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3873 -----------------
>> > 0xf77578d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
>> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
>> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
>> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3874 -----------------
>> > 0xf77578d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
>> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
>> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
>> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3875 -----------------
>> > 0xf77578d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
>> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
>> > 0x064bc935
>> >
>> _ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
>> > + 0x85
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3878 -----------------
>> > 0xf775a6fe __libc_accept + 0x3e
>> > 0xb0c41a19 Java_java_net_PlainSocketImpl_socketAccept + 0x129
>> > 0xf400b4aa *
>> java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl)
>> > bci:0 (Interpreted frame)
>> > 0xf400408d * java.net.PlainSocketImpl.accept(java.net.SocketImpl)
>> bci:7
>> > line:384 (Interpreted frame)
>> > 0xf400408d * java.net.ServerSocket.implAccept(java.net.Socket) bci:60
>> > line:453 (Interpreted frame)
>> > 0xf400408d * java.net.ServerSocket.accept() bci:48 line:421
>> (Interpreted
>> > frame)
>> > 0xf4003f27 *
>> > sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() bci:1
>> > line:34 (Interpreted frame)
>> > 0xf4003f27 *
>> > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() bci:55
>> > line:369 (Interpreted frame)
>> > 0xf400408d * sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() bci:1
>> > line:341 (Interpreted frame)
>> > 0xf4004569 * java.lang.Thread.run() bci:11 line:619 (Interpreted
>> frame)
>> > 0xf400134c <StubRoutines>
>> > 0x06375c9d
>> >
>> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
>> > + 0x15d
>> > 0x065057f8
>> >
>> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
>> > + 0x18
>> > 0x063755b0
>> >
>> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
>> > + 0xa0
>> > 0x0637563d
>> >
>> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
>> > + 0x5d
>> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3879 -----------------
>> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
>> > 0x1bef7958 ????????
>> > sun.jvm.hotspot.debugger.UnmappedAddressException
>> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
>> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
>> > at
>> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
>> > at
>> >
>> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
>> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> > at sun.tools.jstack.JStack.main(JStack.java:84)
>> > ----------------- 4055 -----------------
>> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
>> > 0x14e31390 ????????
>> > sun.jvm.hotspot.debugger.UnmappedAddressException
>> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
>> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
>> > at
>> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
>> > at
>> >
>> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
>> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> > at sun.tools.jstack.JStack.main(JStack.java:84)
>> > ----------------- 4060 -----------------
>> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
>> > 0x358d0f88 ????????
>> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
>> > address: 0x4bbbaf53 with alignment: 4
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
>> > at
>> >
>> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
>> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> > at sun.tools.jstack.JStack.main(JStack.java:84)
>> > ----------------- 4062 -----------------
>> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
>> > 0x35c10c20 ????????
>> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
>> > address: 0x4bbbaf53 with alignment: 4
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
>> > at
>> >
>> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
>> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> > at sun.tools.jstack.JStack.main(JStack.java:84)
>> > ----------------- 7801 -----------------
>> > 0xf775a6fe __libc_accept + 0x3e
>> > 0x061c1a78 _ZN14AttachListener7dequeueEv + 0x68
>> > 0x061c0e55 _Z28attach_listener_thread_entryP10JavaThreadP6Thread +
>> 0x35
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf7753f50 start_thread + 0xa0
>> > ----------------- 3836 -----------------
>> > 0xf7754452 pthread_join + 0x112
>> > 0x0804dd78 ContinueInNewThread + 0x88
>> > 0x080497f6 main + 0x276
>> > 0xf7614775 ????????
>> >
>> >
>> >
>> > [erlf07@cohiba ~ ]$ cat java_native_frames_tt
>> > Attaching to process ID 22510, please wait...
>> > Debugger attached successfully.
>> > Server compiler detected.
>> > JVM version is 11.2-b01
>> > Deadlock Detection:
>> >
>> > No deadlocks found.
>> >
>> > ----------------- 22544 -----------------
>> > 0xf76e34f1 __libc_read + 0x41
>> > 0xf6fab98c readBytes + 0xdc
>> > 0xf6fa7717 Java_java_io_FileInputStream_readBytes + 0x47
>> > 0xf3f944aa * java.io.FileInputStream.readBytes(byte[], int, int) bci:0
>> > (Interpreted frame)
>> > 0xf3f8cf69 * java.io.FileInputStream.read(byte[], int, int) bci:4
>> > line:199 (Interpreted frame)
>> > 0xf3f8cf69 * java.io.BufferedInputStream.read1(byte[], int, int)
>> bci:39
>> > line:256 (Interpreted frame)
>> > 0xf3f8cf69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
>> > line:317 (Interpreted frame)
>> > 0xf3f8cf69 * java.io.BufferedInputStream.fill() bci:175 line:218
>> > (Interpreted frame)
>> > 0xf3f8d08d * java.io.BufferedInputStream.read1(byte[], int, int)
>> bci:44
>> > line:258 (Interpreted frame)
>> > 0xf3f8cf69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
>> > line:317 (Interpreted frame)
>> > 0xf3f8cf69 *
>> > sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12
>> > line:453 (Interpreted frame)
>> > 0xf3f8ce61 * sun.security.provider.SeedGenerator.getSeedBytes(byte[])
>> > bci:11 line:123 (Interpreted frame)
>> > 0xf3f8d08d * sun.security.provider.SeedGenerator.generateSeed(byte[])
>> > bci:4 line:118 (Interpreted frame)
>> > 0xf3f8d08d *
>> sun.security.provider.SecureRandom.engineGenerateSeed(int)
>> > bci:5 line:114 (Interpreted frame)
>> > 0xf3f8cf27 *
>> sun.security.provider.SecureRandom.engineNextBytes(byte[])
>> > bci:40 line:171 (Interpreted frame)
>> > 0xf3f8d08d * java.security.SecureRandom.nextBytes(byte[]) bci:5
>> line:433
>> > (Interpreted frame)
>> > 0xf3f8d08d * java.security.SecureRandom.next(int) bci:17 line:455
>> > (Interpreted frame)
>> > 0xf3f8cf69 * java.util.Random.nextLong() bci:3 line:284 (Interpreted
>> > frame)
>> > 0xf3f8cfab * org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
>> > bci:73 line:139 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf3f8d569 *
>> org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
>> > bci:96 line:168 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.jetty.servlet.HashSessionManager.doStart()
>> > bci:12 line:67 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf3f8d569 * org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4
>> > line:115 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf3f8d569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
>> > line:130 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.jetty.handler.ContextHandler.startContext()
>> > bci:1 line:537 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.jetty.servlet.Context.startContext() bci:1
>> > line:136 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.jetty.webapp.WebAppContext.startContext()
>> > bci:123 line:1234 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.jetty.handler.ContextHandler.doStart()
>> bci:140
>> > line:517 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170
>> > line:460 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf3f8d569 * org.mortbay.jetty.handler.HandlerCollection.doStart()
>> bci:32
>> > line:152 (Interpreted frame)
>> > 0xf3f8d08d *
>> org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
>> > bci:5 line:156 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf3f8d569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
>> > line:130 (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.jetty.Server.doStart() bci:201 line:222
>> > (Interpreted frame)
>> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
>> > line:50 (Interpreted frame)
>> > 0xf3f8d08d * org.apache.hadoop.http.HttpServer.start() bci:383
>> line:461
>> > (Interpreted frame)
>> > 0xf3f8d08d *
>> >
>> org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
>> > bci:519 line:931 (Interpreted frame)
>> > 0xf3f8d08d *
>> > org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) bci:52
>> > line:2833 (Interpreted frame)
>> > 0xf3f8a34c <StubRoutines>
>> > 0x06375c9d
>> >
>> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
>> > + 0x15d
>> > 0x065057f8
>> >
>> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
>> > + 0x18
>> > 0x06375b30
>> > _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread
>> +
>> > 0x20
>> > 0x0639f2e6
>> >
>> _Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
>> > + 0x116
>> > 0x06390ae8 jni_CallStaticVoidMethod + 0xf8
>> > 0x08049b98 JavaMain + 0x2c8
>> > 0xf76dcf50 start_thread + 0xa0
>> > ----------------- 22545 -----------------
>> > 0xf76e08d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
>> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
>> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
>> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf76dcf50 start_thread + 0xa0
>> > ----------------- 22546 -----------------
>> > 0xf76e08d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
>> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
>> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
>> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf76dcf50 start_thread + 0xa0
>> > ----------------- 22549 -----------------
>> > 0xf76e0be8 __pthread_cond_timedwait + 0xe8
>> > 0x340707e0 ????????
>> > sun.jvm.hotspot.debugger.UnmappedAddressException
>> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
>> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
>> > at
>> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
>> > at
>> >
>> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
>> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> > at sun.tools.jstack.JStack.main(JStack.java:84)
>> > ----------------- 22550 -----------------
>> > 0xf76e08d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
>> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
>> > 0x063c834a JVM_MonitorWait + 0x1da
>> > 0xf3f944aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
>> > 0xf3f8d08d * java.lang.Object.wait() bci:2 line:485 (Interpreted
>> frame)
>> > 0xf3f8d08d * java.lang.ref.Reference$ReferenceHandler.run() bci:46
>> > line:116 (Interpreted frame)
>> > 0xf3f8a34c <StubRoutines>
>> > 0x06375c9d
>> >
>> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
>> > + 0x15d
>> > 0x065057f8
>> >
>> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
>> > + 0x18
>> > 0x063755b0
>> >
>> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
>> > + 0xa0
>> > 0x0637563d
>> >
>> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
>> > + 0x5d
>> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf76dcf50 start_thread + 0xa0
>> > ----------------- 22551 -----------------
>> > 0xf76e08d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
>> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
>> > 0x063c834a JVM_MonitorWait + 0x1da
>> > 0xf3f944aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
>> > 0xf3f8d08d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116
>> > (Interpreted frame)
>> > 0xf3f8cf27 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
>> > (Interpreted frame)
>> > 0xf3f8cf27 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3
>> line:159
>> > (Interpreted frame)
>> > 0xf3f8a34c <StubRoutines>
>> > 0x06375c9d
>> >
>> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
>> > + 0x15d
>> > 0x065057f8
>> >
>> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
>> > + 0x18
>> > 0x063755b0
>> >
>> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
>> > + 0xa0
>> > 0x0637563d
>> >
>> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
>> > + 0x5d
>> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf76dcf50 start_thread + 0xa0
>> > ----------------- 22554 -----------------
>> > 0xf76e28b0 __new_sem_wait + 0x40
>> > 0x06503c0d _ZN2os11signal_waitEv + 0xd
>> > 0x06500d28 _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf76dcf50 start_thread + 0xa0
>> > ----------------- 22555 -----------------
>> > 0xf76e08d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
>> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
>> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
>> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf76dcf50 start_thread + 0xa0
>> > ----------------- 22556 -----------------
>> > 0xf76e08d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
>> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
>> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
>> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf76dcf50 start_thread + 0xa0
>> > ----------------- 22557 -----------------
>> > 0xf76e08d0 __pthread_cond_wait + 0x90
>> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
>> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
>> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
>> > 0x064bc935
>> >
>> _ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
>> > + 0x85
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf76dcf50 start_thread + 0xa0
>> > ----------------- 22558 -----------------
>> > 0xf76e0be8 __pthread_cond_timedwait + 0xe8
>> > 0x38ed9008 ????????
>> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
>> > address: 0x4bbbb06f with alignment: 4
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
>> > at
>> >
>> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
>> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> > at sun.tools.jstack.JStack.main(JStack.java:84)
>> > ----------------- 22644 -----------------
>> > 0xf76e0be8 __pthread_cond_timedwait + 0xe8
>> > 0x07b9bcc8 ????????
>> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
>> > address: 0x4bbbb087 with alignment: 4
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
>> > at
>> >
>> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
>> > at
>> >
>> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
>> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
>> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
>> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
>> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
>> > at sun.tools.jstack.JStack.main(JStack.java:84)
>> > ----------------- 24136 -----------------
>> > 0xf76e36fe __libc_accept + 0x3e
>> > 0x061c1a78 _ZN14AttachListener7dequeueEv + 0x68
>> > 0x061c0e55 _Z28attach_listener_thread_entryP10JavaThreadP6Thread +
>> 0x35
>> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
>> > 0x06506929 _Z10java_startP6Thread + 0x139
>> > 0xf76dcf50 start_thread + 0xa0
>> > ----------------- 22510 -----------------
>> > 0xf76dd452 pthread_join + 0x112
>> > 0x0804dd78 ContinueInNewThread + 0x88
>> > 0x080497f6 main + 0x276
>> > 0xf759d775 ????????
>> >
>> >
>> > [erlf07@cohiba ~ ]$ cat thread_dump_dn
>> > Attaching to process ID 3836, please wait...
>> > Debugger attached successfully.
>> > Server compiler detected.
>> > JVM version is 11.2-b01
>> > Deadlock Detection:
>> >
>> > No deadlocks found.
>> >
>> > Thread 7801: (state = BLOCKED)
>> >
>> >
>> > Thread 4062: (state = BLOCKED)
>> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>> > - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275,
>> line=565
>> > (Interpreted frame)
>> >
>> >
>> > Thread 4060: (state = BLOCKED)
>> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>> > - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275,
>> line=565
>> > (Interpreted frame)
>> >
>> >
>> > Thread 4055: (state = BLOCKED)
>> > - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
>> > - org.apache.hadoop.fs.DU$DURefreshThread.run() @bci=17, line=80
>> > (Interpreted frame)
>> > - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
>> >
>> >
>> > Thread 3878: (state = IN_NATIVE)
>> > - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0
>> > (Interpreted frame)
>> > - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=384
>> > (Interpreted frame)
>> > - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=453
>> > (Interpreted frame)
>> > - java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame)
>> > - sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept()
>> @bci=1,
>> > line=34 (Interpreted frame)
>> > - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop()
>> > @bci=55, line=369 (Interpreted frame)
>> > - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=341
>> > (Interpreted frame)
>> > - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
>> >
>> >
>> > Thread 3872: (state = BLOCKED)
>> >
>> >
>> > Thread 3871: (state = BLOCKED)
>> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>> > - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116
>> (Interpreted
>> > frame)
>> > - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
>> > frame)
>> > - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
>> > (Interpreted frame)
>> >
>> >
>> > Thread 3870: (state = BLOCKED)
>> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>> > - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
>> > - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
>> > (Interpreted frame)
>> >
>> >
>> > Thread 3864: (state = IN_NATIVE)
>> > - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0
>> (Interpreted
>> > frame)
>> > - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
>> > (Interpreted frame)
>> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=256
>> > (Interpreted frame)
>> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
>> > (Interpreted frame)
>> > - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted
>> frame)
>> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258
>> > (Interpreted frame)
>> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
>> > (Interpreted frame)
>> > - sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
>> > @bci=12, line=453 (Interpreted frame)
>> > - sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
>> > line=123 (Interpreted frame)
>> > - sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4,
>> line=118
>> > (Interpreted frame)
>> > - sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
>> > line=114 (Interpreted frame)
>> > - sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
>> > line=171 (Interpreted frame)
>> > - java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
>> > (Interpreted frame)
>> > - java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
>> > frame)
>> > - java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
>> > - org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
>> > line=139 (Interpreted frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
>> > line=168 (Interpreted frame)
>> > - org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12,
>> line=67
>> > (Interpreted frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
>> > (Interpreted frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1,
>> line=537
>> > (Interpreted frame)
>> > - org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
>> > (Interpreted frame)
>> > - org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123,
>> line=1234
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140, line=517
>> > (Interpreted frame)
>> > - org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
>> > (Interpreted frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32,
>> line=152
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.ContextHandlerCollection.doStart() @bci=5,
>> > line=156 (Interpreted frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
>> > (Interpreted frame)
>> > - org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted
>> frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.apache.hadoop.http.HttpServer.start() @bci=383, line=461
>> (Interpreted
>> > frame)
>> > -
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
>> > java.util.AbstractList) @bci=916, line=375 (Interpreted frame)
>> > -
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
>> > java.util.AbstractList) @bci=158, line=216 (Interpreted frame)
>> > -
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
>> > org.apache.hadoop.conf.Configuration) @bci=96, line=1283 (Interpreted
>> frame)
>> > -
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
>> > org.apache.hadoop.conf.Configuration) @bci=92, line=1238 (Interpreted
>> frame)
>> > -
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
>> > org.apache.hadoop.conf.Configuration) @bci=2, line=1246 (Interpreted
>> frame)
>> > -
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
>> > @bci=12, line=1368 (Interpreted frame)
>> >
>> >
>> >
>> >
>> > [erlf07@cohiba ~ ]$ cat thread_dump_tt
>> > Attaching to process ID 22510, please wait...
>> > Debugger attached successfully.
>> > Server compiler detected.
>> > JVM version is 11.2-b01
>> > Deadlock Detection:
>> >
>> > No deadlocks found.
>> >
>> > Thread 24136: (state = BLOCKED)
>> >
>> >
>> > Thread 22644: (state = BLOCKED)
>> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>> > - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275,
>> line=565
>> > (Interpreted frame)
>> >
>> >
>> > Thread 22554: (state = BLOCKED)
>> >
>> >
>> > Thread 22551: (state = BLOCKED)
>> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>> > - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116
>> (Interpreted
>> > frame)
>> > - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
>> > frame)
>> > - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
>> > (Interpreted frame)
>> >
>> >
>> > Thread 22550: (state = BLOCKED)
>> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>> > - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
>> > - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
>> > (Interpreted frame)
>> >
>> >
>> > Thread 22544: (state = IN_NATIVE)
>> > - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0
>> (Interpreted
>> > frame)
>> > - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
>> > (Interpreted frame)
>> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=256
>> > (Interpreted frame)
>> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
>> > (Interpreted frame)
>> > - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted
>> frame)
>> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258
>> > (Interpreted frame)
>> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
>> > (Interpreted frame)
>> > - sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
>> > @bci=12, line=453 (Interpreted frame)
>> > - sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
>> > line=123 (Interpreted frame)
>> > - sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4,
>> line=118
>> > (Interpreted frame)
>> > - sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
>> > line=114 (Interpreted frame)
>> > - sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
>> > line=171 (Interpreted frame)
>> > - java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
>> > (Interpreted frame)
>> > - java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
>> > frame)
>> > - java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
>> > - org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
>> > line=139 (Interpreted frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
>> > line=168 (Interpreted frame)
>> > - org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12,
>> line=67
>> > (Interpreted frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
>> > (Interpreted frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1,
>> line=537
>> > (Interpreted frame)
>> > - org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
>> > (Interpreted frame)
>> > - org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123,
>> line=1234
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140, line=517
>> > (Interpreted frame)
>> > - org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
>> > (Interpreted frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32,
>> line=152
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.ContextHandlerCollection.doStart() @bci=5,
>> > line=156 (Interpreted frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
>> > (Interpreted frame)
>> > - org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted
>> frame)
>> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
>> > (Interpreted frame)
>> > - org.apache.hadoop.http.HttpServer.start() @bci=383, line=461
>> (Interpreted
>> > frame)
>> > -
>> >
>> org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
>> > @bci=519, line=931 (Interpreted frame)
>> > - org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) @bci=52,
>> > line=2833 (Interpreted frame)
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Edson Ramiro
>> >
>> >
>> > On 6 April 2010 19:47, Todd Lipcon <to...@cloudera.com> wrote:
>> >
>> >> Hi Edson,
>> >>
>> >> Your attachments did not come through - can you put them on pastebin?
>> >>
>> >> -Todd
>> >>
>> >> On Tue, Apr 6, 2010 at 3:37 PM, Edson Ramiro <er...@gmail.com>
>> wrote:
>> >> > Hi Todd,
>> >> >
>> >> > I'm getting this behavior in another cluster too, there the same thing
>> >> > happens.
>> >> > and as I don't have a jstack installed in the first cluster and
>> >> > I'm not the admin, I'm sending the results of the second cluster.
>> >> >
>> >> > These are the results:
>> >> >
>> >> > [erlf07@cohiba ~ ]$ jstack -l 22510
>> >> > 22510: well-known file is not secure
>> >> > [erlf07@cohiba ~ ]$ jstack -l 3836
>> >> > 3836: well-known file is not secure
>> >> >
>> >> > The jstack -F result is in thread_dump files and the jstack -m result
>> is
>> >> in
>> >> > java_native_frames.
>> >> >
>> >> > The files ending with nn are the namenode results and the files ending
>> >> with
>> >> > dn are the datanode results.
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Edson Ramiro
>> >> >
>> >> >
>> >> > On 6 April 2010 18:19, Todd Lipcon <to...@cloudera.com> wrote:
>> >> >>
>> >> >> Hi Edson,
>> >> >>
>> >> >> Can you please run "jstack" on the daemons in question and paste the
>> >> >> output here?
>> >> >>
>> >> >> -Todd
>> >> >>
>> >> >> On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro <er...@gmail.com>
>> >> wrote:
>> >> >> > Hi all,
>> >> >> >
>> >> >> > I configured the Hadoop in a cluster and the NameNode and
>> JobTracker
>> >> are
>> >> >> > running ok, but the DataNode and TaskTracker Doesn't start, they
>> stop
>> >> >> > and
>> >> >> > keep waiting
>> >> >> > when they are going to start the Jetty
>> >> >> >
>> >> >> > I observed that Jetty can't start the _SelectChannelConnector_
>> >> >> >
>> >> >> > Is there any Jetty configuration that should be changed ?
>> >> >> >
>> >> >> > There is no log message in the NN and JT when I try to start the DN
>> >> and
>> >> >> > TT.
>> >> >> >
>> >> >> > The kernel I'm using is:
>> >> >> > Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64
>> >> GNU/Linux
>> >> >> >
>> >> >> > This is the message when I start the DN. It happens with TT too.
>> >> >> >
>> >> >> > ramiro@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode
>> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG:
>> >> >> > /************************************************************
>> >> >> > STARTUP_MSG: Starting DataNode
>> >> >> > STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115
>> >> >> > STARTUP_MSG: args = []
>> >> >> > STARTUP_MSG: version = 0.20.1+169.56
>> >> >> > STARTUP_MSG: build = -r
>> 8e662cb065be1c4bc61c55e6bff161e09c1d36f3;
>> >> >> > compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010
>> >> >> > ************************************************************/
>> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Registered
>> >> >> > FSDatasetStatusMBean
>> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at
>> 50010
>> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is
>> >> 1048576
>> >> >> > bytes/s
>> >> >> > 10/04/06 16:24:14 INFO mortbay.log: Logging to
>> >> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> >> >> > org.mortbay.log.Slf4jLog
>> >> >> > 10/04/06 16:24:14 INFO http.HttpServer: Port returned by
>> >> >> > webServer.getConnectors()[0].getLocalPort() before open() is -1.
>> >> Opening
>> >> >> > the
>> >> >> > listener on 50075
>> >> >> > 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort()
>> >> returned
>> >> >> > 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
>> >> >> > 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075
>> >> >> > 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14
>> >> >> >
>> >> >> > Thanks in Advance,
>> >> >> >
>> >> >> > Edson Ramiro
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Todd Lipcon
>> >> >> Software Engineer, Cloudera
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Todd Lipcon
>> >> Software Engineer, Cloudera
>> >>
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Jetty can't start the SelectChannelConnector
Posted by Edson Ramiro <er...@gmail.com>.
Thanks Todd,
but could you please explain why entropy is important ?
Edson Ramiro
On 6 April 2010 20:09, Todd Lipcon <to...@cloudera.com> wrote:
> Not enough entropy on your system - you need to generate entropy or
> fake some using this technique:
>
>
> http://www.chrissearle.org/blog/technical/increase_entropy_26_kernel_linux_box
>
> -Todd
>
>
> On Tue, Apr 6, 2010 at 4:05 PM, Edson Ramiro <er...@gmail.com> wrote:
> > ok,
> >
> >
> > [erlf07@cohiba ~ ]$ cat java_native_frames_dn
> > Attaching to process ID 3836, please wait...
> > Debugger attached successfully.
> > Server compiler detected.
> > JVM version is 11.2-b01
> > Deadlock Detection:
> >
> > No deadlocks found.
> >
> > ----------------- 3864 -----------------
> > 0xf775a4f1 __libc_read + 0x41
> > 0xf702298c readBytes + 0xdc
> > 0xf701e717 Java_java_io_FileInputStream_readBytes + 0x47
> > 0xf400b4aa * java.io.FileInputStream.readBytes(byte[], int, int) bci:0
> > (Interpreted frame)
> > 0xf4003f69 * java.io.FileInputStream.read(byte[], int, int) bci:4
> > line:199 (Interpreted frame)
> > 0xf4003f69 * java.io.BufferedInputStream.read1(byte[], int, int)
> bci:39
> > line:256 (Interpreted frame)
> > 0xf4003f69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
> > line:317 (Interpreted frame)
> > 0xf4003f69 * java.io.BufferedInputStream.fill() bci:175 line:218
> > (Interpreted frame)
> > 0xf400408d * java.io.BufferedInputStream.read1(byte[], int, int)
> bci:44
> > line:258 (Interpreted frame)
> > 0xf4003f69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
> > line:317 (Interpreted frame)
> > 0xf4003f69 *
> > sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12
> > line:453 (Interpreted frame)
> > 0xf4003e61 * sun.security.provider.SeedGenerator.getSeedBytes(byte[])
> > bci:11 line:123 (Interpreted frame)
> > 0xf400408d * sun.security.provider.SeedGenerator.generateSeed(byte[])
> > bci:4 line:118 (Interpreted frame)
> > 0xf400408d *
> sun.security.provider.SecureRandom.engineGenerateSeed(int)
> > bci:5 line:114 (Interpreted frame)
> > 0xf4003f27 *
> sun.security.provider.SecureRandom.engineNextBytes(byte[])
> > bci:40 line:171 (Interpreted frame)
> > 0xf400408d * java.security.SecureRandom.nextBytes(byte[]) bci:5
> line:433
> > (Interpreted frame)
> > 0xf400408d * java.security.SecureRandom.next(int) bci:17 line:455
> > (Interpreted frame)
> > 0xf4003f69 * java.util.Random.nextLong() bci:3 line:284 (Interpreted
> > frame)
> > 0xf4003fab * org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
> > bci:73 line:139 (Interpreted frame)
> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf4004569 *
> org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
> > bci:96 line:168 (Interpreted frame)
> > 0xf400408d * org.mortbay.jetty.servlet.HashSessionManager.doStart()
> > bci:12 line:67 (Interpreted frame)
> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf4004569 * org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4
> > line:115 (Interpreted frame)
> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf4004569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
> > line:130 (Interpreted frame)
> > 0xf400408d * org.mortbay.jetty.handler.ContextHandler.startContext()
> > bci:1 line:537 (Interpreted frame)
> > 0xf400408d * org.mortbay.jetty.servlet.Context.startContext() bci:1
> > line:136 (Interpreted frame)
> > 0xf400408d * org.mortbay.jetty.webapp.WebAppContext.startContext()
> > bci:123 line:1234 (Interpreted frame)
> > 0xf400408d * org.mortbay.jetty.handler.ContextHandler.doStart()
> bci:140
> > line:517 (Interpreted frame)
> > 0xf400408d * org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170
> > line:460 (Interpreted frame)
> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf4004569 * org.mortbay.jetty.handler.HandlerCollection.doStart()
> bci:32
> > line:152 (Interpreted frame)
> > 0xf400408d *
> org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
> > bci:5 line:156 (Interpreted frame)
> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf4004569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
> > line:130 (Interpreted frame)
> > 0xf400408d * org.mortbay.jetty.Server.doStart() bci:201 line:222
> > (Interpreted frame)
> > 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf400408d * org.apache.hadoop.http.HttpServer.start() bci:383
> line:461
> > (Interpreted frame)
> > 0xf400408d *
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
> > java.util.AbstractList) bci:916 line:375 (Interpreted frame)
> > 0xf400408d *
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
> > java.util.AbstractList) bci:158 line:216 (Interpreted frame)
> > 0xf400408d *
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
> > org.apache.hadoop.conf.Configuration) bci:96 line:1283 (Interpreted
> frame)
> > 0xf4003f27 *
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
> > org.apache.hadoop.conf.Configuration) bci:92 line:1238 (Interpreted
> frame)
> > 0xf4003f27 *
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
> > org.apache.hadoop.conf.Configuration) bci:2 line:1246 (Interpreted frame)
> > 0xf4003f27 *
> > org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
> > bci:12 line:1368 (Interpreted frame)
> > 0xf400134c <StubRoutines>
> > 0x06375c9d
> >
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> > + 0x15d
> > 0x065057f8
> >
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> > + 0x18
> > 0x06375b30
> > _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread
> +
> > 0x20
> > 0x0639f2e6
> >
> _Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
> > + 0x116
> > 0x06390ae8 jni_CallStaticVoidMethod + 0xf8
> > 0x08049b98 JavaMain + 0x2c8
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3865 -----------------
> > 0xf77578d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3866 -----------------
> > 0xf77578d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3869 -----------------
> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
> > 0x39d73c58 ????????
> > sun.jvm.hotspot.debugger.UnmappedAddressException
> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> > at
> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> > at
> >
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> > at
> >
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> > at sun.tools.jstack.JStack.main(JStack.java:84)
> > ----------------- 3870 -----------------
> > 0xf77578d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> > 0x063c834a JVM_MonitorWait + 0x1da
> > 0xf400b4aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> > 0xf400408d * java.lang.Object.wait() bci:2 line:485 (Interpreted
> frame)
> > 0xf400408d * java.lang.ref.Reference$ReferenceHandler.run() bci:46
> > line:116 (Interpreted frame)
> > 0xf400134c <StubRoutines>
> > 0x06375c9d
> >
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> > + 0x15d
> > 0x065057f8
> >
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> > + 0x18
> > 0x063755b0
> >
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> > + 0xa0
> > 0x0637563d
> >
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> > + 0x5d
> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3871 -----------------
> > 0xf77578d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> > 0x063c834a JVM_MonitorWait + 0x1da
> > 0xf400b4aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> > 0xf400408d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116
> > (Interpreted frame)
> > 0xf4003f27 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
> > (Interpreted frame)
> > 0xf4003f27 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3
> line:159
> > (Interpreted frame)
> > 0xf400134c <StubRoutines>
> > 0x06375c9d
> >
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> > + 0x15d
> > 0x065057f8
> >
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> > + 0x18
> > 0x063755b0
> >
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> > + 0xa0
> > 0x0637563d
> >
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> > + 0x5d
> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3872 -----------------
> > 0xf77598b0 __new_sem_wait + 0x40
> > 0x06503c0d _ZN2os11signal_waitEv + 0xd
> > 0x06500d28 _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3873 -----------------
> > 0xf77578d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3874 -----------------
> > 0xf77578d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3875 -----------------
> > 0xf77578d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> > 0x064bc935
> >
> _ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
> > + 0x85
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3878 -----------------
> > 0xf775a6fe __libc_accept + 0x3e
> > 0xb0c41a19 Java_java_net_PlainSocketImpl_socketAccept + 0x129
> > 0xf400b4aa *
> java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl)
> > bci:0 (Interpreted frame)
> > 0xf400408d * java.net.PlainSocketImpl.accept(java.net.SocketImpl)
> bci:7
> > line:384 (Interpreted frame)
> > 0xf400408d * java.net.ServerSocket.implAccept(java.net.Socket) bci:60
> > line:453 (Interpreted frame)
> > 0xf400408d * java.net.ServerSocket.accept() bci:48 line:421
> (Interpreted
> > frame)
> > 0xf4003f27 *
> > sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() bci:1
> > line:34 (Interpreted frame)
> > 0xf4003f27 *
> > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() bci:55
> > line:369 (Interpreted frame)
> > 0xf400408d * sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() bci:1
> > line:341 (Interpreted frame)
> > 0xf4004569 * java.lang.Thread.run() bci:11 line:619 (Interpreted
> frame)
> > 0xf400134c <StubRoutines>
> > 0x06375c9d
> >
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> > + 0x15d
> > 0x065057f8
> >
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> > + 0x18
> > 0x063755b0
> >
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> > + 0xa0
> > 0x0637563d
> >
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> > + 0x5d
> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3879 -----------------
> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
> > 0x1bef7958 ????????
> > sun.jvm.hotspot.debugger.UnmappedAddressException
> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> > at
> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> > at
> >
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> > at
> >
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> > at sun.tools.jstack.JStack.main(JStack.java:84)
> > ----------------- 4055 -----------------
> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
> > 0x14e31390 ????????
> > sun.jvm.hotspot.debugger.UnmappedAddressException
> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> > at
> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> > at
> >
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> > at
> >
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> > at sun.tools.jstack.JStack.main(JStack.java:84)
> > ----------------- 4060 -----------------
> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
> > 0x358d0f88 ????????
> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> > address: 0x4bbbaf53 with alignment: 4
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> > at
> >
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> > at
> >
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> > at sun.tools.jstack.JStack.main(JStack.java:84)
> > ----------------- 4062 -----------------
> > 0xf7757be8 __pthread_cond_timedwait + 0xe8
> > 0x35c10c20 ????????
> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> > address: 0x4bbbaf53 with alignment: 4
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> > at
> >
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> > at
> >
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> > at sun.tools.jstack.JStack.main(JStack.java:84)
> > ----------------- 7801 -----------------
> > 0xf775a6fe __libc_accept + 0x3e
> > 0x061c1a78 _ZN14AttachListener7dequeueEv + 0x68
> > 0x061c0e55 _Z28attach_listener_thread_entryP10JavaThreadP6Thread +
> 0x35
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf7753f50 start_thread + 0xa0
> > ----------------- 3836 -----------------
> > 0xf7754452 pthread_join + 0x112
> > 0x0804dd78 ContinueInNewThread + 0x88
> > 0x080497f6 main + 0x276
> > 0xf7614775 ????????
> >
> >
> >
> > [erlf07@cohiba ~ ]$ cat java_native_frames_tt
> > Attaching to process ID 22510, please wait...
> > Debugger attached successfully.
> > Server compiler detected.
> > JVM version is 11.2-b01
> > Deadlock Detection:
> >
> > No deadlocks found.
> >
> > ----------------- 22544 -----------------
> > 0xf76e34f1 __libc_read + 0x41
> > 0xf6fab98c readBytes + 0xdc
> > 0xf6fa7717 Java_java_io_FileInputStream_readBytes + 0x47
> > 0xf3f944aa * java.io.FileInputStream.readBytes(byte[], int, int) bci:0
> > (Interpreted frame)
> > 0xf3f8cf69 * java.io.FileInputStream.read(byte[], int, int) bci:4
> > line:199 (Interpreted frame)
> > 0xf3f8cf69 * java.io.BufferedInputStream.read1(byte[], int, int)
> bci:39
> > line:256 (Interpreted frame)
> > 0xf3f8cf69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
> > line:317 (Interpreted frame)
> > 0xf3f8cf69 * java.io.BufferedInputStream.fill() bci:175 line:218
> > (Interpreted frame)
> > 0xf3f8d08d * java.io.BufferedInputStream.read1(byte[], int, int)
> bci:44
> > line:258 (Interpreted frame)
> > 0xf3f8cf69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
> > line:317 (Interpreted frame)
> > 0xf3f8cf69 *
> > sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12
> > line:453 (Interpreted frame)
> > 0xf3f8ce61 * sun.security.provider.SeedGenerator.getSeedBytes(byte[])
> > bci:11 line:123 (Interpreted frame)
> > 0xf3f8d08d * sun.security.provider.SeedGenerator.generateSeed(byte[])
> > bci:4 line:118 (Interpreted frame)
> > 0xf3f8d08d *
> sun.security.provider.SecureRandom.engineGenerateSeed(int)
> > bci:5 line:114 (Interpreted frame)
> > 0xf3f8cf27 *
> sun.security.provider.SecureRandom.engineNextBytes(byte[])
> > bci:40 line:171 (Interpreted frame)
> > 0xf3f8d08d * java.security.SecureRandom.nextBytes(byte[]) bci:5
> line:433
> > (Interpreted frame)
> > 0xf3f8d08d * java.security.SecureRandom.next(int) bci:17 line:455
> > (Interpreted frame)
> > 0xf3f8cf69 * java.util.Random.nextLong() bci:3 line:284 (Interpreted
> > frame)
> > 0xf3f8cfab * org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
> > bci:73 line:139 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf3f8d569 *
> org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
> > bci:96 line:168 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.jetty.servlet.HashSessionManager.doStart()
> > bci:12 line:67 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf3f8d569 * org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4
> > line:115 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf3f8d569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
> > line:130 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.jetty.handler.ContextHandler.startContext()
> > bci:1 line:537 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.jetty.servlet.Context.startContext() bci:1
> > line:136 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.jetty.webapp.WebAppContext.startContext()
> > bci:123 line:1234 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.jetty.handler.ContextHandler.doStart()
> bci:140
> > line:517 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170
> > line:460 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf3f8d569 * org.mortbay.jetty.handler.HandlerCollection.doStart()
> bci:32
> > line:152 (Interpreted frame)
> > 0xf3f8d08d *
> org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
> > bci:5 line:156 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf3f8d569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
> > line:130 (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.jetty.Server.doStart() bci:201 line:222
> > (Interpreted frame)
> > 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> > line:50 (Interpreted frame)
> > 0xf3f8d08d * org.apache.hadoop.http.HttpServer.start() bci:383
> line:461
> > (Interpreted frame)
> > 0xf3f8d08d *
> >
> org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
> > bci:519 line:931 (Interpreted frame)
> > 0xf3f8d08d *
> > org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) bci:52
> > line:2833 (Interpreted frame)
> > 0xf3f8a34c <StubRoutines>
> > 0x06375c9d
> >
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> > + 0x15d
> > 0x065057f8
> >
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> > + 0x18
> > 0x06375b30
> > _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread
> +
> > 0x20
> > 0x0639f2e6
> >
> _Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
> > + 0x116
> > 0x06390ae8 jni_CallStaticVoidMethod + 0xf8
> > 0x08049b98 JavaMain + 0x2c8
> > 0xf76dcf50 start_thread + 0xa0
> > ----------------- 22545 -----------------
> > 0xf76e08d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf76dcf50 start_thread + 0xa0
> > ----------------- 22546 -----------------
> > 0xf76e08d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> > 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> > 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf76dcf50 start_thread + 0xa0
> > ----------------- 22549 -----------------
> > 0xf76e0be8 __pthread_cond_timedwait + 0xe8
> > 0x340707e0 ????????
> > sun.jvm.hotspot.debugger.UnmappedAddressException
> > at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> > at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> > at
> > sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> > at
> >
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> > at
> >
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> > at sun.tools.jstack.JStack.main(JStack.java:84)
> > ----------------- 22550 -----------------
> > 0xf76e08d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> > 0x063c834a JVM_MonitorWait + 0x1da
> > 0xf3f944aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> > 0xf3f8d08d * java.lang.Object.wait() bci:2 line:485 (Interpreted
> frame)
> > 0xf3f8d08d * java.lang.ref.Reference$ReferenceHandler.run() bci:46
> > line:116 (Interpreted frame)
> > 0xf3f8a34c <StubRoutines>
> > 0x06375c9d
> >
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> > + 0x15d
> > 0x065057f8
> >
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> > + 0x18
> > 0x063755b0
> >
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> > + 0xa0
> > 0x0637563d
> >
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> > + 0x5d
> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf76dcf50 start_thread + 0xa0
> > ----------------- 22551 -----------------
> > 0xf76e08d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> > 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> > 0x063c834a JVM_MonitorWait + 0x1da
> > 0xf3f944aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> > 0xf3f8d08d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116
> > (Interpreted frame)
> > 0xf3f8cf27 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
> > (Interpreted frame)
> > 0xf3f8cf27 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3
> line:159
> > (Interpreted frame)
> > 0xf3f8a34c <StubRoutines>
> > 0x06375c9d
> >
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> > + 0x15d
> > 0x065057f8
> >
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> > + 0x18
> > 0x063755b0
> >
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> > + 0xa0
> > 0x0637563d
> >
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> > + 0x5d
> > 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf76dcf50 start_thread + 0xa0
> > ----------------- 22554 -----------------
> > 0xf76e28b0 __new_sem_wait + 0x40
> > 0x06503c0d _ZN2os11signal_waitEv + 0xd
> > 0x06500d28 _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf76dcf50 start_thread + 0xa0
> > ----------------- 22555 -----------------
> > 0xf76e08d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf76dcf50 start_thread + 0xa0
> > ----------------- 22556 -----------------
> > 0xf76e08d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> > 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> > 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> > 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf76dcf50 start_thread + 0xa0
> > ----------------- 22557 -----------------
> > 0xf76e08d0 __pthread_cond_wait + 0x90
> > 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> > 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> > 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> > 0x064bc935
> >
> _ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
> > + 0x85
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf76dcf50 start_thread + 0xa0
> > ----------------- 22558 -----------------
> > 0xf76e0be8 __pthread_cond_timedwait + 0xe8
> > 0x38ed9008 ????????
> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> > address: 0x4bbbb06f with alignment: 4
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> > at
> >
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> > at
> >
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> > at sun.tools.jstack.JStack.main(JStack.java:84)
> > ----------------- 22644 -----------------
> > 0xf76e0be8 __pthread_cond_timedwait + 0xe8
> > 0x07b9bcc8 ????????
> > sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> > address: 0x4bbbb087 with alignment: 4
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> > at
> >
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> > at
> >
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> > at
> >
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> > at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> > at sun.tools.jstack.JStack.main(JStack.java:84)
> > ----------------- 24136 -----------------
> > 0xf76e36fe __libc_accept + 0x3e
> > 0x061c1a78 _ZN14AttachListener7dequeueEv + 0x68
> > 0x061c0e55 _Z28attach_listener_thread_entryP10JavaThreadP6Thread +
> 0x35
> > 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> > 0x06506929 _Z10java_startP6Thread + 0x139
> > 0xf76dcf50 start_thread + 0xa0
> > ----------------- 22510 -----------------
> > 0xf76dd452 pthread_join + 0x112
> > 0x0804dd78 ContinueInNewThread + 0x88
> > 0x080497f6 main + 0x276
> > 0xf759d775 ????????
> >
> >
> > [erlf07@cohiba ~ ]$ cat thread_dump_dn
> > Attaching to process ID 3836, please wait...
> > Debugger attached successfully.
> > Server compiler detected.
> > JVM version is 11.2-b01
> > Deadlock Detection:
> >
> > No deadlocks found.
> >
> > Thread 7801: (state = BLOCKED)
> >
> >
> > Thread 4062: (state = BLOCKED)
> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> > - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275,
> line=565
> > (Interpreted frame)
> >
> >
> > Thread 4060: (state = BLOCKED)
> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> > - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275,
> line=565
> > (Interpreted frame)
> >
> >
> > Thread 4055: (state = BLOCKED)
> > - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
> > - org.apache.hadoop.fs.DU$DURefreshThread.run() @bci=17, line=80
> > (Interpreted frame)
> > - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
> >
> >
> > Thread 3878: (state = IN_NATIVE)
> > - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0
> > (Interpreted frame)
> > - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=384
> > (Interpreted frame)
> > - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=453
> > (Interpreted frame)
> > - java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame)
> > - sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept()
> @bci=1,
> > line=34 (Interpreted frame)
> > - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop()
> > @bci=55, line=369 (Interpreted frame)
> > - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=341
> > (Interpreted frame)
> > - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
> >
> >
> > Thread 3872: (state = BLOCKED)
> >
> >
> > Thread 3871: (state = BLOCKED)
> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> > - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116
> (Interpreted
> > frame)
> > - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
> > frame)
> > - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
> > (Interpreted frame)
> >
> >
> > Thread 3870: (state = BLOCKED)
> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> > - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
> > - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
> > (Interpreted frame)
> >
> >
> > Thread 3864: (state = IN_NATIVE)
> > - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0
> (Interpreted
> > frame)
> > - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
> > (Interpreted frame)
> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=256
> > (Interpreted frame)
> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
> > (Interpreted frame)
> > - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted
> frame)
> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258
> > (Interpreted frame)
> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
> > (Interpreted frame)
> > - sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
> > @bci=12, line=453 (Interpreted frame)
> > - sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
> > line=123 (Interpreted frame)
> > - sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4,
> line=118
> > (Interpreted frame)
> > - sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
> > line=114 (Interpreted frame)
> > - sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
> > line=171 (Interpreted frame)
> > - java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
> > (Interpreted frame)
> > - java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
> > frame)
> > - java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
> > - org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
> > line=139 (Interpreted frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
> > line=168 (Interpreted frame)
> > - org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12,
> line=67
> > (Interpreted frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
> > (Interpreted frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1,
> line=537
> > (Interpreted frame)
> > - org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
> > (Interpreted frame)
> > - org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123,
> line=1234
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140, line=517
> > (Interpreted frame)
> > - org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
> > (Interpreted frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32,
> line=152
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.ContextHandlerCollection.doStart() @bci=5,
> > line=156 (Interpreted frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
> > (Interpreted frame)
> > - org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted
> frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.apache.hadoop.http.HttpServer.start() @bci=383, line=461
> (Interpreted
> > frame)
> > -
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
> > java.util.AbstractList) @bci=916, line=375 (Interpreted frame)
> > -
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
> > java.util.AbstractList) @bci=158, line=216 (Interpreted frame)
> > -
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
> > org.apache.hadoop.conf.Configuration) @bci=96, line=1283 (Interpreted
> frame)
> > -
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
> > org.apache.hadoop.conf.Configuration) @bci=92, line=1238 (Interpreted
> frame)
> > -
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
> > org.apache.hadoop.conf.Configuration) @bci=2, line=1246 (Interpreted
> frame)
> > -
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
> > @bci=12, line=1368 (Interpreted frame)
> >
> >
> >
> >
> > [erlf07@cohiba ~ ]$ cat thread_dump_tt
> > Attaching to process ID 22510, please wait...
> > Debugger attached successfully.
> > Server compiler detected.
> > JVM version is 11.2-b01
> > Deadlock Detection:
> >
> > No deadlocks found.
> >
> > Thread 24136: (state = BLOCKED)
> >
> >
> > Thread 22644: (state = BLOCKED)
> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> > - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275,
> line=565
> > (Interpreted frame)
> >
> >
> > Thread 22554: (state = BLOCKED)
> >
> >
> > Thread 22551: (state = BLOCKED)
> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> > - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116
> (Interpreted
> > frame)
> > - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
> > frame)
> > - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
> > (Interpreted frame)
> >
> >
> > Thread 22550: (state = BLOCKED)
> > - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> > - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
> > - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
> > (Interpreted frame)
> >
> >
> > Thread 22544: (state = IN_NATIVE)
> > - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0
> (Interpreted
> > frame)
> > - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
> > (Interpreted frame)
> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=256
> > (Interpreted frame)
> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
> > (Interpreted frame)
> > - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted
> frame)
> > - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258
> > (Interpreted frame)
> > - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
> > (Interpreted frame)
> > - sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
> > @bci=12, line=453 (Interpreted frame)
> > - sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
> > line=123 (Interpreted frame)
> > - sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4,
> line=118
> > (Interpreted frame)
> > - sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
> > line=114 (Interpreted frame)
> > - sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
> > line=171 (Interpreted frame)
> > - java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
> > (Interpreted frame)
> > - java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
> > frame)
> > - java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
> > - org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
> > line=139 (Interpreted frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
> > line=168 (Interpreted frame)
> > - org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12,
> line=67
> > (Interpreted frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
> > (Interpreted frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1,
> line=537
> > (Interpreted frame)
> > - org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
> > (Interpreted frame)
> > - org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123,
> line=1234
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140, line=517
> > (Interpreted frame)
> > - org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
> > (Interpreted frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32,
> line=152
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.ContextHandlerCollection.doStart() @bci=5,
> > line=156 (Interpreted frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
> > (Interpreted frame)
> > - org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted
> frame)
> > - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> > (Interpreted frame)
> > - org.apache.hadoop.http.HttpServer.start() @bci=383, line=461
> (Interpreted
> > frame)
> > -
> >
> org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
> > @bci=519, line=931 (Interpreted frame)
> > - org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) @bci=52,
> > line=2833 (Interpreted frame)
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Edson Ramiro
> >
> >
> > On 6 April 2010 19:47, Todd Lipcon <to...@cloudera.com> wrote:
> >
> >> Hi Edson,
> >>
> >> Your attachments did not come through - can you put them on pastebin?
> >>
> >> -Todd
> >>
> >> On Tue, Apr 6, 2010 at 3:37 PM, Edson Ramiro <er...@gmail.com>
> wrote:
> >> > Hi Todd,
> >> >
> >> > I'm getting this behavior in another cluster too, there the same thing
> >> > happens.
> >> > and as I don't have a jstack installed in the first cluster and
> >> > I'm not the admin, I'm sending the results of the second cluster.
> >> >
> >> > These are the results:
> >> >
> >> > [erlf07@cohiba ~ ]$ jstack -l 22510
> >> > 22510: well-known file is not secure
> >> > [erlf07@cohiba ~ ]$ jstack -l 3836
> >> > 3836: well-known file is not secure
> >> >
> >> > The jstack -F result is in thread_dump files and the jstack -m result
> is
> >> in
> >> > java_native_frames.
> >> >
> >> > The files ending with nn are the namenode results and the files ending
> >> with
> >> > dn are the datanode results.
> >> >
> >> > Thanks,
> >> >
> >> > Edson Ramiro
> >> >
> >> >
> >> > On 6 April 2010 18:19, Todd Lipcon <to...@cloudera.com> wrote:
> >> >>
> >> >> Hi Edson,
> >> >>
> >> >> Can you please run "jstack" on the daemons in question and paste the
> >> >> output here?
> >> >>
> >> >> -Todd
> >> >>
> >> >> On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro <er...@gmail.com>
> >> wrote:
> >> >> > Hi all,
> >> >> >
> >> >> > I configured the Hadoop in a cluster and the NameNode and
> JobTracker
> >> are
> >> >> > running ok, but the DataNode and TaskTracker Doesn't start, they
> stop
> >> >> > and
> >> >> > keep waiting
> >> >> > when they are going to start the Jetty
> >> >> >
> >> >> > I observed that Jetty can't start the _SelectChannelConnector_
> >> >> >
> >> >> > Is there any Jetty configuration that should be changed ?
> >> >> >
> >> >> > There is no log message in the NN and JT when I try to start the DN
> >> and
> >> >> > TT.
> >> >> >
> >> >> > The kernel I'm using is:
> >> >> > Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64
> >> GNU/Linux
> >> >> >
> >> >> > This is the message when I start the DN. It happens with TT too.
> >> >> >
> >> >> > ramiro@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode
> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG:
> >> >> > /************************************************************
> >> >> > STARTUP_MSG: Starting DataNode
> >> >> > STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115
> >> >> > STARTUP_MSG: args = []
> >> >> > STARTUP_MSG: version = 0.20.1+169.56
> >> >> > STARTUP_MSG: build = -r
> 8e662cb065be1c4bc61c55e6bff161e09c1d36f3;
> >> >> > compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010
> >> >> > ************************************************************/
> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Registered
> >> >> > FSDatasetStatusMBean
> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at
> 50010
> >> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is
> >> 1048576
> >> >> > bytes/s
> >> >> > 10/04/06 16:24:14 INFO mortbay.log: Logging to
> >> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> >> >> > org.mortbay.log.Slf4jLog
> >> >> > 10/04/06 16:24:14 INFO http.HttpServer: Port returned by
> >> >> > webServer.getConnectors()[0].getLocalPort() before open() is -1.
> >> Opening
> >> >> > the
> >> >> > listener on 50075
> >> >> > 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort()
> >> returned
> >> >> > 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
> >> >> > 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075
> >> >> > 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14
> >> >> >
> >> >> > Thanks in Advance,
> >> >> >
> >> >> > Edson Ramiro
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Todd Lipcon
> >> >> Software Engineer, Cloudera
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Jetty can't start the SelectChannelConnector
Posted by Todd Lipcon <to...@cloudera.com>.
Not enough entropy on your system - you need to generate entropy or
fake some using this technique:
http://www.chrissearle.org/blog/technical/increase_entropy_26_kernel_linux_box
-Todd
On Tue, Apr 6, 2010 at 4:05 PM, Edson Ramiro <er...@gmail.com> wrote:
> ok,
>
>
> [erlf07@cohiba ~ ]$ cat java_native_frames_dn
> Attaching to process ID 3836, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 11.2-b01
> Deadlock Detection:
>
> No deadlocks found.
>
> ----------------- 3864 -----------------
> 0xf775a4f1 __libc_read + 0x41
> 0xf702298c readBytes + 0xdc
> 0xf701e717 Java_java_io_FileInputStream_readBytes + 0x47
> 0xf400b4aa * java.io.FileInputStream.readBytes(byte[], int, int) bci:0
> (Interpreted frame)
> 0xf4003f69 * java.io.FileInputStream.read(byte[], int, int) bci:4
> line:199 (Interpreted frame)
> 0xf4003f69 * java.io.BufferedInputStream.read1(byte[], int, int) bci:39
> line:256 (Interpreted frame)
> 0xf4003f69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
> line:317 (Interpreted frame)
> 0xf4003f69 * java.io.BufferedInputStream.fill() bci:175 line:218
> (Interpreted frame)
> 0xf400408d * java.io.BufferedInputStream.read1(byte[], int, int) bci:44
> line:258 (Interpreted frame)
> 0xf4003f69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
> line:317 (Interpreted frame)
> 0xf4003f69 *
> sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12
> line:453 (Interpreted frame)
> 0xf4003e61 * sun.security.provider.SeedGenerator.getSeedBytes(byte[])
> bci:11 line:123 (Interpreted frame)
> 0xf400408d * sun.security.provider.SeedGenerator.generateSeed(byte[])
> bci:4 line:118 (Interpreted frame)
> 0xf400408d * sun.security.provider.SecureRandom.engineGenerateSeed(int)
> bci:5 line:114 (Interpreted frame)
> 0xf4003f27 * sun.security.provider.SecureRandom.engineNextBytes(byte[])
> bci:40 line:171 (Interpreted frame)
> 0xf400408d * java.security.SecureRandom.nextBytes(byte[]) bci:5 line:433
> (Interpreted frame)
> 0xf400408d * java.security.SecureRandom.next(int) bci:17 line:455
> (Interpreted frame)
> 0xf4003f69 * java.util.Random.nextLong() bci:3 line:284 (Interpreted
> frame)
> 0xf4003fab * org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
> bci:73 line:139 (Interpreted frame)
> 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf4004569 * org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
> bci:96 line:168 (Interpreted frame)
> 0xf400408d * org.mortbay.jetty.servlet.HashSessionManager.doStart()
> bci:12 line:67 (Interpreted frame)
> 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf4004569 * org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4
> line:115 (Interpreted frame)
> 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf4004569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
> line:130 (Interpreted frame)
> 0xf400408d * org.mortbay.jetty.handler.ContextHandler.startContext()
> bci:1 line:537 (Interpreted frame)
> 0xf400408d * org.mortbay.jetty.servlet.Context.startContext() bci:1
> line:136 (Interpreted frame)
> 0xf400408d * org.mortbay.jetty.webapp.WebAppContext.startContext()
> bci:123 line:1234 (Interpreted frame)
> 0xf400408d * org.mortbay.jetty.handler.ContextHandler.doStart() bci:140
> line:517 (Interpreted frame)
> 0xf400408d * org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170
> line:460 (Interpreted frame)
> 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf4004569 * org.mortbay.jetty.handler.HandlerCollection.doStart() bci:32
> line:152 (Interpreted frame)
> 0xf400408d * org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
> bci:5 line:156 (Interpreted frame)
> 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf4004569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
> line:130 (Interpreted frame)
> 0xf400408d * org.mortbay.jetty.Server.doStart() bci:201 line:222
> (Interpreted frame)
> 0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf400408d * org.apache.hadoop.http.HttpServer.start() bci:383 line:461
> (Interpreted frame)
> 0xf400408d *
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
> java.util.AbstractList) bci:916 line:375 (Interpreted frame)
> 0xf400408d *
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
> java.util.AbstractList) bci:158 line:216 (Interpreted frame)
> 0xf400408d *
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
> org.apache.hadoop.conf.Configuration) bci:96 line:1283 (Interpreted frame)
> 0xf4003f27 *
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
> org.apache.hadoop.conf.Configuration) bci:92 line:1238 (Interpreted frame)
> 0xf4003f27 *
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
> org.apache.hadoop.conf.Configuration) bci:2 line:1246 (Interpreted frame)
> 0xf4003f27 *
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
> bci:12 line:1368 (Interpreted frame)
> 0xf400134c <StubRoutines>
> 0x06375c9d
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> + 0x15d
> 0x065057f8
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> + 0x18
> 0x06375b30
> _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread +
> 0x20
> 0x0639f2e6
> _Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
> + 0x116
> 0x06390ae8 jni_CallStaticVoidMethod + 0xf8
> 0x08049b98 JavaMain + 0x2c8
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3865 -----------------
> 0xf77578d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3866 -----------------
> 0xf77578d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3869 -----------------
> 0xf7757be8 __pthread_cond_timedwait + 0xe8
> 0x39d73c58 ????????
> sun.jvm.hotspot.debugger.UnmappedAddressException
> at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> at
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> at
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> ----------------- 3870 -----------------
> 0xf77578d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> 0x063c834a JVM_MonitorWait + 0x1da
> 0xf400b4aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> 0xf400408d * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
> 0xf400408d * java.lang.ref.Reference$ReferenceHandler.run() bci:46
> line:116 (Interpreted frame)
> 0xf400134c <StubRoutines>
> 0x06375c9d
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> + 0x15d
> 0x065057f8
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> + 0x18
> 0x063755b0
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> + 0xa0
> 0x0637563d
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> + 0x5d
> 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3871 -----------------
> 0xf77578d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> 0x063c834a JVM_MonitorWait + 0x1da
> 0xf400b4aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> 0xf400408d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116
> (Interpreted frame)
> 0xf4003f27 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
> (Interpreted frame)
> 0xf4003f27 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159
> (Interpreted frame)
> 0xf400134c <StubRoutines>
> 0x06375c9d
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> + 0x15d
> 0x065057f8
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> + 0x18
> 0x063755b0
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> + 0xa0
> 0x0637563d
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> + 0x5d
> 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3872 -----------------
> 0xf77598b0 __new_sem_wait + 0x40
> 0x06503c0d _ZN2os11signal_waitEv + 0xd
> 0x06500d28 _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3873 -----------------
> 0xf77578d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3874 -----------------
> 0xf77578d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3875 -----------------
> 0xf77578d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> 0x064bc935
> _ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
> + 0x85
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3878 -----------------
> 0xf775a6fe __libc_accept + 0x3e
> 0xb0c41a19 Java_java_net_PlainSocketImpl_socketAccept + 0x129
> 0xf400b4aa * java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl)
> bci:0 (Interpreted frame)
> 0xf400408d * java.net.PlainSocketImpl.accept(java.net.SocketImpl) bci:7
> line:384 (Interpreted frame)
> 0xf400408d * java.net.ServerSocket.implAccept(java.net.Socket) bci:60
> line:453 (Interpreted frame)
> 0xf400408d * java.net.ServerSocket.accept() bci:48 line:421 (Interpreted
> frame)
> 0xf4003f27 *
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() bci:1
> line:34 (Interpreted frame)
> 0xf4003f27 *
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() bci:55
> line:369 (Interpreted frame)
> 0xf400408d * sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() bci:1
> line:341 (Interpreted frame)
> 0xf4004569 * java.lang.Thread.run() bci:11 line:619 (Interpreted frame)
> 0xf400134c <StubRoutines>
> 0x06375c9d
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> + 0x15d
> 0x065057f8
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> + 0x18
> 0x063755b0
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> + 0xa0
> 0x0637563d
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> + 0x5d
> 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3879 -----------------
> 0xf7757be8 __pthread_cond_timedwait + 0xe8
> 0x1bef7958 ????????
> sun.jvm.hotspot.debugger.UnmappedAddressException
> at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> at
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> at
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> ----------------- 4055 -----------------
> 0xf7757be8 __pthread_cond_timedwait + 0xe8
> 0x14e31390 ????????
> sun.jvm.hotspot.debugger.UnmappedAddressException
> at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> at
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> at
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> ----------------- 4060 -----------------
> 0xf7757be8 __pthread_cond_timedwait + 0xe8
> 0x358d0f88 ????????
> sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> address: 0x4bbbaf53 with alignment: 4
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> at
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> at
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> ----------------- 4062 -----------------
> 0xf7757be8 __pthread_cond_timedwait + 0xe8
> 0x35c10c20 ????????
> sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> address: 0x4bbbaf53 with alignment: 4
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> at
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> at
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> ----------------- 7801 -----------------
> 0xf775a6fe __libc_accept + 0x3e
> 0x061c1a78 _ZN14AttachListener7dequeueEv + 0x68
> 0x061c0e55 _Z28attach_listener_thread_entryP10JavaThreadP6Thread + 0x35
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf7753f50 start_thread + 0xa0
> ----------------- 3836 -----------------
> 0xf7754452 pthread_join + 0x112
> 0x0804dd78 ContinueInNewThread + 0x88
> 0x080497f6 main + 0x276
> 0xf7614775 ????????
>
>
>
> [erlf07@cohiba ~ ]$ cat java_native_frames_tt
> Attaching to process ID 22510, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 11.2-b01
> Deadlock Detection:
>
> No deadlocks found.
>
> ----------------- 22544 -----------------
> 0xf76e34f1 __libc_read + 0x41
> 0xf6fab98c readBytes + 0xdc
> 0xf6fa7717 Java_java_io_FileInputStream_readBytes + 0x47
> 0xf3f944aa * java.io.FileInputStream.readBytes(byte[], int, int) bci:0
> (Interpreted frame)
> 0xf3f8cf69 * java.io.FileInputStream.read(byte[], int, int) bci:4
> line:199 (Interpreted frame)
> 0xf3f8cf69 * java.io.BufferedInputStream.read1(byte[], int, int) bci:39
> line:256 (Interpreted frame)
> 0xf3f8cf69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
> line:317 (Interpreted frame)
> 0xf3f8cf69 * java.io.BufferedInputStream.fill() bci:175 line:218
> (Interpreted frame)
> 0xf3f8d08d * java.io.BufferedInputStream.read1(byte[], int, int) bci:44
> line:258 (Interpreted frame)
> 0xf3f8cf69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
> line:317 (Interpreted frame)
> 0xf3f8cf69 *
> sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12
> line:453 (Interpreted frame)
> 0xf3f8ce61 * sun.security.provider.SeedGenerator.getSeedBytes(byte[])
> bci:11 line:123 (Interpreted frame)
> 0xf3f8d08d * sun.security.provider.SeedGenerator.generateSeed(byte[])
> bci:4 line:118 (Interpreted frame)
> 0xf3f8d08d * sun.security.provider.SecureRandom.engineGenerateSeed(int)
> bci:5 line:114 (Interpreted frame)
> 0xf3f8cf27 * sun.security.provider.SecureRandom.engineNextBytes(byte[])
> bci:40 line:171 (Interpreted frame)
> 0xf3f8d08d * java.security.SecureRandom.nextBytes(byte[]) bci:5 line:433
> (Interpreted frame)
> 0xf3f8d08d * java.security.SecureRandom.next(int) bci:17 line:455
> (Interpreted frame)
> 0xf3f8cf69 * java.util.Random.nextLong() bci:3 line:284 (Interpreted
> frame)
> 0xf3f8cfab * org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
> bci:73 line:139 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf3f8d569 * org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
> bci:96 line:168 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.jetty.servlet.HashSessionManager.doStart()
> bci:12 line:67 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf3f8d569 * org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4
> line:115 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf3f8d569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
> line:130 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.jetty.handler.ContextHandler.startContext()
> bci:1 line:537 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.jetty.servlet.Context.startContext() bci:1
> line:136 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.jetty.webapp.WebAppContext.startContext()
> bci:123 line:1234 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.jetty.handler.ContextHandler.doStart() bci:140
> line:517 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170
> line:460 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf3f8d569 * org.mortbay.jetty.handler.HandlerCollection.doStart() bci:32
> line:152 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
> bci:5 line:156 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf3f8d569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
> line:130 (Interpreted frame)
> 0xf3f8d08d * org.mortbay.jetty.Server.doStart() bci:201 line:222
> (Interpreted frame)
> 0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
> line:50 (Interpreted frame)
> 0xf3f8d08d * org.apache.hadoop.http.HttpServer.start() bci:383 line:461
> (Interpreted frame)
> 0xf3f8d08d *
> org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
> bci:519 line:931 (Interpreted frame)
> 0xf3f8d08d *
> org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) bci:52
> line:2833 (Interpreted frame)
> 0xf3f8a34c <StubRoutines>
> 0x06375c9d
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> + 0x15d
> 0x065057f8
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> + 0x18
> 0x06375b30
> _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread +
> 0x20
> 0x0639f2e6
> _Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
> + 0x116
> 0x06390ae8 jni_CallStaticVoidMethod + 0xf8
> 0x08049b98 JavaMain + 0x2c8
> 0xf76dcf50 start_thread + 0xa0
> ----------------- 22545 -----------------
> 0xf76e08d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf76dcf50 start_thread + 0xa0
> ----------------- 22546 -----------------
> 0xf76e08d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> 0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
> 0x0632843b _ZN12GCTaskThread3runEv + 0xdb
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf76dcf50 start_thread + 0xa0
> ----------------- 22549 -----------------
> 0xf76e0be8 __pthread_cond_timedwait + 0xe8
> 0x340707e0 ????????
> sun.jvm.hotspot.debugger.UnmappedAddressException
> at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> at
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> at
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> ----------------- 22550 -----------------
> 0xf76e08d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> 0x063c834a JVM_MonitorWait + 0x1da
> 0xf3f944aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> 0xf3f8d08d * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
> 0xf3f8d08d * java.lang.ref.Reference$ReferenceHandler.run() bci:46
> line:116 (Interpreted frame)
> 0xf3f8a34c <StubRoutines>
> 0x06375c9d
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> + 0x15d
> 0x065057f8
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> + 0x18
> 0x063755b0
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> + 0xa0
> 0x0637563d
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> + 0x5d
> 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf76dcf50 start_thread + 0xa0
> ----------------- 22551 -----------------
> 0xf76e08d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
> 0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
> 0x063c834a JVM_MonitorWait + 0x1da
> 0xf3f944aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
> 0xf3f8d08d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116
> (Interpreted frame)
> 0xf3f8cf27 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
> (Interpreted frame)
> 0xf3f8cf27 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159
> (Interpreted frame)
> 0xf3f8a34c <StubRoutines>
> 0x06375c9d
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
> + 0x15d
> 0x065057f8
> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
> + 0x18
> 0x063755b0
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
> + 0xa0
> 0x0637563d
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
> + 0x5d
> 0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf76dcf50 start_thread + 0xa0
> ----------------- 22554 -----------------
> 0xf76e28b0 __new_sem_wait + 0x40
> 0x06503c0d _ZN2os11signal_waitEv + 0xd
> 0x06500d28 _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf76dcf50 start_thread + 0xa0
> ----------------- 22555 -----------------
> 0xf76e08d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf76dcf50 start_thread + 0xa0
> ----------------- 22556 -----------------
> 0xf76e08d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> 0x064edf21 _ZN7Monitor4waitEblb + 0xd1
> 0x062ada2f _ZN12CompileQueue3getEv + 0x5f
> 0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf76dcf50 start_thread + 0xa0
> ----------------- 22557 -----------------
> 0xf76e08d0 __pthread_cond_wait + 0x90
> 0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
> 0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
> 0x064ee071 _ZN7Monitor4waitEblb + 0x221
> 0x064bc935
> _ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
> + 0x85
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf76dcf50 start_thread + 0xa0
> ----------------- 22558 -----------------
> 0xf76e0be8 __pthread_cond_timedwait + 0xe8
> 0x38ed9008 ????????
> sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> address: 0x4bbbb06f with alignment: 4
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> at
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> at
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> ----------------- 22644 -----------------
> 0xf76e0be8 __pthread_cond_timedwait + 0xe8
> 0x07b9bcc8 ????????
> sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
> address: 0x4bbbb087 with alignment: 4
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
> at
> sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> at
> sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> at
> sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> ----------------- 24136 -----------------
> 0xf76e36fe __libc_accept + 0x3e
> 0x061c1a78 _ZN14AttachListener7dequeueEv + 0x68
> 0x061c0e55 _Z28attach_listener_thread_entryP10JavaThreadP6Thread + 0x35
> 0x065cde5d _ZN10JavaThread3runEv + 0xfd
> 0x06506929 _Z10java_startP6Thread + 0x139
> 0xf76dcf50 start_thread + 0xa0
> ----------------- 22510 -----------------
> 0xf76dd452 pthread_join + 0x112
> 0x0804dd78 ContinueInNewThread + 0x88
> 0x080497f6 main + 0x276
> 0xf759d775 ????????
>
>
> [erlf07@cohiba ~ ]$ cat thread_dump_dn
> Attaching to process ID 3836, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 11.2-b01
> Deadlock Detection:
>
> No deadlocks found.
>
> Thread 7801: (state = BLOCKED)
>
>
> Thread 4062: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275, line=565
> (Interpreted frame)
>
>
> Thread 4060: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275, line=565
> (Interpreted frame)
>
>
> Thread 4055: (state = BLOCKED)
> - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
> - org.apache.hadoop.fs.DU$DURefreshThread.run() @bci=17, line=80
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
>
>
> Thread 3878: (state = IN_NATIVE)
> - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0
> (Interpreted frame)
> - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=384
> (Interpreted frame)
> - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=453
> (Interpreted frame)
> - java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame)
> - sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() @bci=1,
> line=34 (Interpreted frame)
> - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop()
> @bci=55, line=369 (Interpreted frame)
> - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=341
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
>
>
> Thread 3872: (state = BLOCKED)
>
>
> Thread 3871: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted
> frame)
> - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
> frame)
> - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
> (Interpreted frame)
>
>
> Thread 3870: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
> - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
> (Interpreted frame)
>
>
> Thread 3864: (state = IN_NATIVE)
> - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Interpreted
> frame)
> - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
> (Interpreted frame)
> - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=256
> (Interpreted frame)
> - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
> (Interpreted frame)
> - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted frame)
> - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258
> (Interpreted frame)
> - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
> (Interpreted frame)
> - sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
> @bci=12, line=453 (Interpreted frame)
> - sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
> line=123 (Interpreted frame)
> - sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4, line=118
> (Interpreted frame)
> - sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
> line=114 (Interpreted frame)
> - sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
> line=171 (Interpreted frame)
> - java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
> (Interpreted frame)
> - java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
> frame)
> - java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
> - org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
> line=139 (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
> line=168 (Interpreted frame)
> - org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12, line=67
> (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
> (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
> (Interpreted frame)
> - org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1, line=537
> (Interpreted frame)
> - org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
> (Interpreted frame)
> - org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123, line=1234
> (Interpreted frame)
> - org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140, line=517
> (Interpreted frame)
> - org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
> (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32, line=152
> (Interpreted frame)
> - org.mortbay.jetty.handler.ContextHandlerCollection.doStart() @bci=5,
> line=156 (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
> (Interpreted frame)
> - org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.apache.hadoop.http.HttpServer.start() @bci=383, line=461 (Interpreted
> frame)
> -
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
> java.util.AbstractList) @bci=916, line=375 (Interpreted frame)
> -
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
> java.util.AbstractList) @bci=158, line=216 (Interpreted frame)
> -
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
> org.apache.hadoop.conf.Configuration) @bci=96, line=1283 (Interpreted frame)
> -
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
> org.apache.hadoop.conf.Configuration) @bci=92, line=1238 (Interpreted frame)
> -
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
> org.apache.hadoop.conf.Configuration) @bci=2, line=1246 (Interpreted frame)
> - org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
> @bci=12, line=1368 (Interpreted frame)
>
>
>
>
> [erlf07@cohiba ~ ]$ cat thread_dump_tt
> Attaching to process ID 22510, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 11.2-b01
> Deadlock Detection:
>
> No deadlocks found.
>
> Thread 24136: (state = BLOCKED)
>
>
> Thread 22644: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275, line=565
> (Interpreted frame)
>
>
> Thread 22554: (state = BLOCKED)
>
>
> Thread 22551: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted
> frame)
> - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
> frame)
> - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
> (Interpreted frame)
>
>
> Thread 22550: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
> - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
> (Interpreted frame)
>
>
> Thread 22544: (state = IN_NATIVE)
> - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Interpreted
> frame)
> - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
> (Interpreted frame)
> - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=256
> (Interpreted frame)
> - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
> (Interpreted frame)
> - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted frame)
> - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258
> (Interpreted frame)
> - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
> (Interpreted frame)
> - sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
> @bci=12, line=453 (Interpreted frame)
> - sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
> line=123 (Interpreted frame)
> - sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4, line=118
> (Interpreted frame)
> - sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
> line=114 (Interpreted frame)
> - sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
> line=171 (Interpreted frame)
> - java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
> (Interpreted frame)
> - java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
> frame)
> - java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
> - org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
> line=139 (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
> line=168 (Interpreted frame)
> - org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12, line=67
> (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
> (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
> (Interpreted frame)
> - org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1, line=537
> (Interpreted frame)
> - org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
> (Interpreted frame)
> - org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123, line=1234
> (Interpreted frame)
> - org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140, line=517
> (Interpreted frame)
> - org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
> (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32, line=152
> (Interpreted frame)
> - org.mortbay.jetty.handler.ContextHandlerCollection.doStart() @bci=5,
> line=156 (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
> (Interpreted frame)
> - org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted frame)
> - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
> (Interpreted frame)
> - org.apache.hadoop.http.HttpServer.start() @bci=383, line=461 (Interpreted
> frame)
> -
> org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
> @bci=519, line=931 (Interpreted frame)
> - org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) @bci=52,
> line=2833 (Interpreted frame)
>
>
>
>
>
>
>
>
>
> Edson Ramiro
>
>
> On 6 April 2010 19:47, Todd Lipcon <to...@cloudera.com> wrote:
>
>> Hi Edson,
>>
>> Your attachments did not come through - can you put them on pastebin?
>>
>> -Todd
>>
>> On Tue, Apr 6, 2010 at 3:37 PM, Edson Ramiro <er...@gmail.com> wrote:
>> > Hi Todd,
>> >
>> > I'm getting this behavior in another cluster too, there the same thing
>> > happens.
>> > and as I don't have a jstack installed in the first cluster and
>> > I'm not the admin, I'm sending the results of the second cluster.
>> >
>> > These are the results:
>> >
>> > [erlf07@cohiba ~ ]$ jstack -l 22510
>> > 22510: well-known file is not secure
>> > [erlf07@cohiba ~ ]$ jstack -l 3836
>> > 3836: well-known file is not secure
>> >
>> > The jstack -F result is in thread_dump files and the jstack -m result is
>> in
>> > java_native_frames.
>> >
>> > The files ending with nn are the namenode results and the files ending
>> with
>> > dn are the datanode results.
>> >
>> > Thanks,
>> >
>> > Edson Ramiro
>> >
>> >
>> > On 6 April 2010 18:19, Todd Lipcon <to...@cloudera.com> wrote:
>> >>
>> >> Hi Edson,
>> >>
>> >> Can you please run "jstack" on the daemons in question and paste the
>> >> output here?
>> >>
>> >> -Todd
>> >>
>> >> On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro <er...@gmail.com>
>> wrote:
>> >> > Hi all,
>> >> >
>> >> > I configured the Hadoop in a cluster and the NameNode and JobTracker
>> are
>> >> > running ok, but the DataNode and TaskTracker Doesn't start, they stop
>> >> > and
>> >> > keep waiting
>> >> > when they are going to start the Jetty
>> >> >
>> >> > I observed that Jetty can't start the _SelectChannelConnector_
>> >> >
>> >> > Is there any Jetty configuration that should be changed ?
>> >> >
>> >> > There is no log message in the NN and JT when I try to start the DN
>> and
>> >> > TT.
>> >> >
>> >> > The kernel I'm using is:
>> >> > Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64
>> GNU/Linux
>> >> >
>> >> > This is the message when I start the DN. It happens with TT too.
>> >> >
>> >> > ramiro@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode
>> >> > 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG:
>> >> > /************************************************************
>> >> > STARTUP_MSG: Starting DataNode
>> >> > STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115
>> >> > STARTUP_MSG: args = []
>> >> > STARTUP_MSG: version = 0.20.1+169.56
>> >> > STARTUP_MSG: build = -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3;
>> >> > compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010
>> >> > ************************************************************/
>> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Registered
>> >> > FSDatasetStatusMBean
>> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at 50010
>> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is
>> 1048576
>> >> > bytes/s
>> >> > 10/04/06 16:24:14 INFO mortbay.log: Logging to
>> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> >> > org.mortbay.log.Slf4jLog
>> >> > 10/04/06 16:24:14 INFO http.HttpServer: Port returned by
>> >> > webServer.getConnectors()[0].getLocalPort() before open() is -1.
>> Opening
>> >> > the
>> >> > listener on 50075
>> >> > 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort()
>> returned
>> >> > 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
>> >> > 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075
>> >> > 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14
>> >> >
>> >> > Thanks in Advance,
>> >> >
>> >> > Edson Ramiro
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Todd Lipcon
>> >> Software Engineer, Cloudera
>> >
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Jetty can't start the SelectChannelConnector
Posted by Edson Ramiro <er...@gmail.com>.
ok,
[erlf07@cohiba ~ ]$ cat java_native_frames_dn
Attaching to process ID 3836, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.2-b01
Deadlock Detection:
No deadlocks found.
----------------- 3864 -----------------
0xf775a4f1 __libc_read + 0x41
0xf702298c readBytes + 0xdc
0xf701e717 Java_java_io_FileInputStream_readBytes + 0x47
0xf400b4aa * java.io.FileInputStream.readBytes(byte[], int, int) bci:0
(Interpreted frame)
0xf4003f69 * java.io.FileInputStream.read(byte[], int, int) bci:4
line:199 (Interpreted frame)
0xf4003f69 * java.io.BufferedInputStream.read1(byte[], int, int) bci:39
line:256 (Interpreted frame)
0xf4003f69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
line:317 (Interpreted frame)
0xf4003f69 * java.io.BufferedInputStream.fill() bci:175 line:218
(Interpreted frame)
0xf400408d * java.io.BufferedInputStream.read1(byte[], int, int) bci:44
line:258 (Interpreted frame)
0xf4003f69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
line:317 (Interpreted frame)
0xf4003f69 *
sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12
line:453 (Interpreted frame)
0xf4003e61 * sun.security.provider.SeedGenerator.getSeedBytes(byte[])
bci:11 line:123 (Interpreted frame)
0xf400408d * sun.security.provider.SeedGenerator.generateSeed(byte[])
bci:4 line:118 (Interpreted frame)
0xf400408d * sun.security.provider.SecureRandom.engineGenerateSeed(int)
bci:5 line:114 (Interpreted frame)
0xf4003f27 * sun.security.provider.SecureRandom.engineNextBytes(byte[])
bci:40 line:171 (Interpreted frame)
0xf400408d * java.security.SecureRandom.nextBytes(byte[]) bci:5 line:433
(Interpreted frame)
0xf400408d * java.security.SecureRandom.next(int) bci:17 line:455
(Interpreted frame)
0xf4003f69 * java.util.Random.nextLong() bci:3 line:284 (Interpreted
frame)
0xf4003fab * org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
bci:73 line:139 (Interpreted frame)
0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf4004569 * org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
bci:96 line:168 (Interpreted frame)
0xf400408d * org.mortbay.jetty.servlet.HashSessionManager.doStart()
bci:12 line:67 (Interpreted frame)
0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf4004569 * org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4
line:115 (Interpreted frame)
0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf4004569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
line:130 (Interpreted frame)
0xf400408d * org.mortbay.jetty.handler.ContextHandler.startContext()
bci:1 line:537 (Interpreted frame)
0xf400408d * org.mortbay.jetty.servlet.Context.startContext() bci:1
line:136 (Interpreted frame)
0xf400408d * org.mortbay.jetty.webapp.WebAppContext.startContext()
bci:123 line:1234 (Interpreted frame)
0xf400408d * org.mortbay.jetty.handler.ContextHandler.doStart() bci:140
line:517 (Interpreted frame)
0xf400408d * org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170
line:460 (Interpreted frame)
0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf4004569 * org.mortbay.jetty.handler.HandlerCollection.doStart() bci:32
line:152 (Interpreted frame)
0xf400408d * org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
bci:5 line:156 (Interpreted frame)
0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf4004569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
line:130 (Interpreted frame)
0xf400408d * org.mortbay.jetty.Server.doStart() bci:201 line:222
(Interpreted frame)
0xf400408d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf400408d * org.apache.hadoop.http.HttpServer.start() bci:383 line:461
(Interpreted frame)
0xf400408d *
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
java.util.AbstractList) bci:916 line:375 (Interpreted frame)
0xf400408d *
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
java.util.AbstractList) bci:158 line:216 (Interpreted frame)
0xf400408d *
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
org.apache.hadoop.conf.Configuration) bci:96 line:1283 (Interpreted frame)
0xf4003f27 *
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
org.apache.hadoop.conf.Configuration) bci:92 line:1238 (Interpreted frame)
0xf4003f27 *
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
org.apache.hadoop.conf.Configuration) bci:2 line:1246 (Interpreted frame)
0xf4003f27 *
org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
bci:12 line:1368 (Interpreted frame)
0xf400134c <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x06375b30
_ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread +
0x20
0x0639f2e6
_Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
+ 0x116
0x06390ae8 jni_CallStaticVoidMethod + 0xf8
0x08049b98 JavaMain + 0x2c8
0xf7753f50 start_thread + 0xa0
----------------- 3865 -----------------
0xf77578d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071 _ZN7Monitor4waitEblb + 0x221
0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
0x0632843b _ZN12GCTaskThread3runEv + 0xdb
0x06506929 _Z10java_startP6Thread + 0x139
0xf7753f50 start_thread + 0xa0
----------------- 3866 -----------------
0xf77578d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071 _ZN7Monitor4waitEblb + 0x221
0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
0x0632843b _ZN12GCTaskThread3runEv + 0xdb
0x06506929 _Z10java_startP6Thread + 0x139
0xf7753f50 start_thread + 0xa0
----------------- 3869 -----------------
0xf7757be8 __pthread_cond_timedwait + 0xe8
0x39d73c58 ????????
sun.jvm.hotspot.debugger.UnmappedAddressException
at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
at
sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 3870 -----------------
0xf77578d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
0x063c834a JVM_MonitorWait + 0x1da
0xf400b4aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xf400408d * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
0xf400408d * java.lang.ref.Reference$ReferenceHandler.run() bci:46
line:116 (Interpreted frame)
0xf400134c <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x063755b0
_ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
+ 0xa0
0x0637563d
_ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
+ 0x5d
0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf7753f50 start_thread + 0xa0
----------------- 3871 -----------------
0xf77578d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
0x063c834a JVM_MonitorWait + 0x1da
0xf400b4aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xf400408d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116
(Interpreted frame)
0xf4003f27 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
(Interpreted frame)
0xf4003f27 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159
(Interpreted frame)
0xf400134c <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x063755b0
_ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
+ 0xa0
0x0637563d
_ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
+ 0x5d
0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf7753f50 start_thread + 0xa0
----------------- 3872 -----------------
0xf77598b0 __new_sem_wait + 0x40
0x06503c0d _ZN2os11signal_waitEv + 0xd
0x06500d28 _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf7753f50 start_thread + 0xa0
----------------- 3873 -----------------
0xf77578d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064edf21 _ZN7Monitor4waitEblb + 0xd1
0x062ada2f _ZN12CompileQueue3getEv + 0x5f
0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf7753f50 start_thread + 0xa0
----------------- 3874 -----------------
0xf77578d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064edf21 _ZN7Monitor4waitEblb + 0xd1
0x062ada2f _ZN12CompileQueue3getEv + 0x5f
0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf7753f50 start_thread + 0xa0
----------------- 3875 -----------------
0xf77578d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071 _ZN7Monitor4waitEblb + 0x221
0x064bc935
_ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
+ 0x85
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf7753f50 start_thread + 0xa0
----------------- 3878 -----------------
0xf775a6fe __libc_accept + 0x3e
0xb0c41a19 Java_java_net_PlainSocketImpl_socketAccept + 0x129
0xf400b4aa * java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl)
bci:0 (Interpreted frame)
0xf400408d * java.net.PlainSocketImpl.accept(java.net.SocketImpl) bci:7
line:384 (Interpreted frame)
0xf400408d * java.net.ServerSocket.implAccept(java.net.Socket) bci:60
line:453 (Interpreted frame)
0xf400408d * java.net.ServerSocket.accept() bci:48 line:421 (Interpreted
frame)
0xf4003f27 *
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() bci:1
line:34 (Interpreted frame)
0xf4003f27 *
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() bci:55
line:369 (Interpreted frame)
0xf400408d * sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() bci:1
line:341 (Interpreted frame)
0xf4004569 * java.lang.Thread.run() bci:11 line:619 (Interpreted frame)
0xf400134c <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x063755b0
_ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
+ 0xa0
0x0637563d
_ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
+ 0x5d
0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf7753f50 start_thread + 0xa0
----------------- 3879 -----------------
0xf7757be8 __pthread_cond_timedwait + 0xe8
0x1bef7958 ????????
sun.jvm.hotspot.debugger.UnmappedAddressException
at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
at
sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 4055 -----------------
0xf7757be8 __pthread_cond_timedwait + 0xe8
0x14e31390 ????????
sun.jvm.hotspot.debugger.UnmappedAddressException
at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
at
sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 4060 -----------------
0xf7757be8 __pthread_cond_timedwait + 0xe8
0x358d0f88 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
address: 0x4bbbaf53 with alignment: 4
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 4062 -----------------
0xf7757be8 __pthread_cond_timedwait + 0xe8
0x35c10c20 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
address: 0x4bbbaf53 with alignment: 4
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 7801 -----------------
0xf775a6fe __libc_accept + 0x3e
0x061c1a78 _ZN14AttachListener7dequeueEv + 0x68
0x061c0e55 _Z28attach_listener_thread_entryP10JavaThreadP6Thread + 0x35
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf7753f50 start_thread + 0xa0
----------------- 3836 -----------------
0xf7754452 pthread_join + 0x112
0x0804dd78 ContinueInNewThread + 0x88
0x080497f6 main + 0x276
0xf7614775 ????????
[erlf07@cohiba ~ ]$ cat java_native_frames_tt
Attaching to process ID 22510, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.2-b01
Deadlock Detection:
No deadlocks found.
----------------- 22544 -----------------
0xf76e34f1 __libc_read + 0x41
0xf6fab98c readBytes + 0xdc
0xf6fa7717 Java_java_io_FileInputStream_readBytes + 0x47
0xf3f944aa * java.io.FileInputStream.readBytes(byte[], int, int) bci:0
(Interpreted frame)
0xf3f8cf69 * java.io.FileInputStream.read(byte[], int, int) bci:4
line:199 (Interpreted frame)
0xf3f8cf69 * java.io.BufferedInputStream.read1(byte[], int, int) bci:39
line:256 (Interpreted frame)
0xf3f8cf69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
line:317 (Interpreted frame)
0xf3f8cf69 * java.io.BufferedInputStream.fill() bci:175 line:218
(Interpreted frame)
0xf3f8d08d * java.io.BufferedInputStream.read1(byte[], int, int) bci:44
line:258 (Interpreted frame)
0xf3f8cf69 * java.io.BufferedInputStream.read(byte[], int, int) bci:49
line:317 (Interpreted frame)
0xf3f8cf69 *
sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12
line:453 (Interpreted frame)
0xf3f8ce61 * sun.security.provider.SeedGenerator.getSeedBytes(byte[])
bci:11 line:123 (Interpreted frame)
0xf3f8d08d * sun.security.provider.SeedGenerator.generateSeed(byte[])
bci:4 line:118 (Interpreted frame)
0xf3f8d08d * sun.security.provider.SecureRandom.engineGenerateSeed(int)
bci:5 line:114 (Interpreted frame)
0xf3f8cf27 * sun.security.provider.SecureRandom.engineNextBytes(byte[])
bci:40 line:171 (Interpreted frame)
0xf3f8d08d * java.security.SecureRandom.nextBytes(byte[]) bci:5 line:433
(Interpreted frame)
0xf3f8d08d * java.security.SecureRandom.next(int) bci:17 line:455
(Interpreted frame)
0xf3f8cf69 * java.util.Random.nextLong() bci:3 line:284 (Interpreted
frame)
0xf3f8cfab * org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
bci:73 line:139 (Interpreted frame)
0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d569 * org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
bci:96 line:168 (Interpreted frame)
0xf3f8d08d * org.mortbay.jetty.servlet.HashSessionManager.doStart()
bci:12 line:67 (Interpreted frame)
0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d569 * org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4
line:115 (Interpreted frame)
0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
line:130 (Interpreted frame)
0xf3f8d08d * org.mortbay.jetty.handler.ContextHandler.startContext()
bci:1 line:537 (Interpreted frame)
0xf3f8d08d * org.mortbay.jetty.servlet.Context.startContext() bci:1
line:136 (Interpreted frame)
0xf3f8d08d * org.mortbay.jetty.webapp.WebAppContext.startContext()
bci:123 line:1234 (Interpreted frame)
0xf3f8d08d * org.mortbay.jetty.handler.ContextHandler.doStart() bci:140
line:517 (Interpreted frame)
0xf3f8d08d * org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170
line:460 (Interpreted frame)
0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d569 * org.mortbay.jetty.handler.HandlerCollection.doStart() bci:32
line:152 (Interpreted frame)
0xf3f8d08d * org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
bci:5 line:156 (Interpreted frame)
0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d569 * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
line:130 (Interpreted frame)
0xf3f8d08d * org.mortbay.jetty.Server.doStart() bci:201 line:222
(Interpreted frame)
0xf3f8d08d * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d08d * org.apache.hadoop.http.HttpServer.start() bci:383 line:461
(Interpreted frame)
0xf3f8d08d *
org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
bci:519 line:931 (Interpreted frame)
0xf3f8d08d *
org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) bci:52
line:2833 (Interpreted frame)
0xf3f8a34c <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x06375b30
_ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread +
0x20
0x0639f2e6
_Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
+ 0x116
0x06390ae8 jni_CallStaticVoidMethod + 0xf8
0x08049b98 JavaMain + 0x2c8
0xf76dcf50 start_thread + 0xa0
----------------- 22545 -----------------
0xf76e08d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071 _ZN7Monitor4waitEblb + 0x221
0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
0x0632843b _ZN12GCTaskThread3runEv + 0xdb
0x06506929 _Z10java_startP6Thread + 0x139
0xf76dcf50 start_thread + 0xa0
----------------- 22546 -----------------
0xf76e08d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071 _ZN7Monitor4waitEblb + 0x221
0x063272f6 _ZN13GCTaskManager8get_taskEj + 0x66
0x0632843b _ZN12GCTaskThread3runEv + 0xdb
0x06506929 _Z10java_startP6Thread + 0x139
0xf76dcf50 start_thread + 0xa0
----------------- 22549 -----------------
0xf76e0be8 __pthread_cond_timedwait + 0xe8
0x340707e0 ????????
sun.jvm.hotspot.debugger.UnmappedAddressException
at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
at
sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 22550 -----------------
0xf76e08d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
0x063c834a JVM_MonitorWait + 0x1da
0xf3f944aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xf3f8d08d * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
0xf3f8d08d * java.lang.ref.Reference$ReferenceHandler.run() bci:46
line:116 (Interpreted frame)
0xf3f8a34c <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x063755b0
_ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
+ 0xa0
0x0637563d
_ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
+ 0x5d
0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf76dcf50 start_thread + 0xa0
----------------- 22551 -----------------
0xf76e08d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x065a1a31 _ZN13ObjectMonitor4waitExbP6Thread + 0x571
0x0659f3a2 _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
0x063c834a JVM_MonitorWait + 0x1da
0xf3f944aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xf3f8d08d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116
(Interpreted frame)
0xf3f8cf27 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
(Interpreted frame)
0xf3f8cf27 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159
(Interpreted frame)
0xf3f8a34c <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x063755b0
_ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
+ 0xa0
0x0637563d
_ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
+ 0x5d
0x063e5ca5 _Z12thread_entryP10JavaThreadP6Thread + 0x75
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf76dcf50 start_thread + 0xa0
----------------- 22554 -----------------
0xf76e28b0 __new_sem_wait + 0x40
0x06503c0d _ZN2os11signal_waitEv + 0xd
0x06500d28 _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf76dcf50 start_thread + 0xa0
----------------- 22555 -----------------
0xf76e08d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064edf21 _ZN7Monitor4waitEblb + 0xd1
0x062ada2f _ZN12CompileQueue3getEv + 0x5f
0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf76dcf50 start_thread + 0xa0
----------------- 22556 -----------------
0xf76e08d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064edf21 _ZN7Monitor4waitEblb + 0xd1
0x062ada2f _ZN12CompileQueue3getEv + 0x5f
0x062af2b4 _ZN13CompileBroker20compiler_thread_loopEv + 0x184
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf76dcf50 start_thread + 0xa0
----------------- 22557 -----------------
0xf76e08d0 __pthread_cond_wait + 0x90
0x065060c9 _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071 _ZN7Monitor4waitEblb + 0x221
0x064bc935
_ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
+ 0x85
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf76dcf50 start_thread + 0xa0
----------------- 22558 -----------------
0xf76e0be8 __pthread_cond_timedwait + 0xe8
0x38ed9008 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
address: 0x4bbbb06f with alignment: 4
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 22644 -----------------
0xf76e0be8 __pthread_cond_timedwait + 0xe8
0x07b9bcc8 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
address: 0x4bbbb087 with alignment: 4
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 24136 -----------------
0xf76e36fe __libc_accept + 0x3e
0x061c1a78 _ZN14AttachListener7dequeueEv + 0x68
0x061c0e55 _Z28attach_listener_thread_entryP10JavaThreadP6Thread + 0x35
0x065cde5d _ZN10JavaThread3runEv + 0xfd
0x06506929 _Z10java_startP6Thread + 0x139
0xf76dcf50 start_thread + 0xa0
----------------- 22510 -----------------
0xf76dd452 pthread_join + 0x112
0x0804dd78 ContinueInNewThread + 0x88
0x080497f6 main + 0x276
0xf759d775 ????????
[erlf07@cohiba ~ ]$ cat thread_dump_dn
Attaching to process ID 3836, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.2-b01
Deadlock Detection:
No deadlocks found.
Thread 7801: (state = BLOCKED)
Thread 4062: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275, line=565
(Interpreted frame)
Thread 4060: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275, line=565
(Interpreted frame)
Thread 4055: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- org.apache.hadoop.fs.DU$DURefreshThread.run() @bci=17, line=80
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 3878: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0
(Interpreted frame)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=384
(Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=453
(Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame)
- sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() @bci=1,
line=34 (Interpreted frame)
- sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop()
@bci=55, line=369 (Interpreted frame)
- sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=341
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 3872: (state = BLOCKED)
Thread 3871: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted
frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
(Interpreted frame)
Thread 3870: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
(Interpreted frame)
Thread 3864: (state = IN_NATIVE)
- java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Interpreted
frame)
- java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
(Interpreted frame)
- java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=256
(Interpreted frame)
- java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
(Interpreted frame)
- java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted frame)
- java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258
(Interpreted frame)
- java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
(Interpreted frame)
- sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
@bci=12, line=453 (Interpreted frame)
- sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
line=123 (Interpreted frame)
- sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4, line=118
(Interpreted frame)
- sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
line=114 (Interpreted frame)
- sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
line=171 (Interpreted frame)
- java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
(Interpreted frame)
- java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
frame)
- java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
- org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
line=139 (Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
line=168 (Interpreted frame)
- org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12, line=67
(Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
(Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
(Interpreted frame)
- org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1, line=537
(Interpreted frame)
- org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
(Interpreted frame)
- org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123, line=1234
(Interpreted frame)
- org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140, line=517
(Interpreted frame)
- org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
(Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32, line=152
(Interpreted frame)
- org.mortbay.jetty.handler.ContextHandlerCollection.doStart() @bci=5,
line=156 (Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
(Interpreted frame)
- org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.apache.hadoop.http.HttpServer.start() @bci=383, line=461 (Interpreted
frame)
-
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
java.util.AbstractList) @bci=916, line=375 (Interpreted frame)
-
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
java.util.AbstractList) @bci=158, line=216 (Interpreted frame)
-
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
org.apache.hadoop.conf.Configuration) @bci=96, line=1283 (Interpreted frame)
-
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
org.apache.hadoop.conf.Configuration) @bci=92, line=1238 (Interpreted frame)
-
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
org.apache.hadoop.conf.Configuration) @bci=2, line=1246 (Interpreted frame)
- org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
@bci=12, line=1368 (Interpreted frame)
[erlf07@cohiba ~ ]$ cat thread_dump_tt
Attaching to process ID 22510, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.2-b01
Deadlock Detection:
No deadlocks found.
Thread 24136: (state = BLOCKED)
Thread 22644: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275, line=565
(Interpreted frame)
Thread 22554: (state = BLOCKED)
Thread 22551: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted
frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
(Interpreted frame)
Thread 22550: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
(Interpreted frame)
Thread 22544: (state = IN_NATIVE)
- java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Interpreted
frame)
- java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
(Interpreted frame)
- java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=256
(Interpreted frame)
- java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
(Interpreted frame)
- java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted frame)
- java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258
(Interpreted frame)
- java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
(Interpreted frame)
- sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
@bci=12, line=453 (Interpreted frame)
- sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
line=123 (Interpreted frame)
- sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4, line=118
(Interpreted frame)
- sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
line=114 (Interpreted frame)
- sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
line=171 (Interpreted frame)
- java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
(Interpreted frame)
- java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
frame)
- java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
- org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
line=139 (Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
line=168 (Interpreted frame)
- org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12, line=67
(Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
(Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
(Interpreted frame)
- org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1, line=537
(Interpreted frame)
- org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
(Interpreted frame)
- org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123, line=1234
(Interpreted frame)
- org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140, line=517
(Interpreted frame)
- org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
(Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32, line=152
(Interpreted frame)
- org.mortbay.jetty.handler.ContextHandlerCollection.doStart() @bci=5,
line=156 (Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
(Interpreted frame)
- org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted frame)
- org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
- org.apache.hadoop.http.HttpServer.start() @bci=383, line=461 (Interpreted
frame)
-
org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
@bci=519, line=931 (Interpreted frame)
- org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) @bci=52,
line=2833 (Interpreted frame)
Edson Ramiro
On 6 April 2010 19:47, Todd Lipcon <to...@cloudera.com> wrote:
> Hi Edson,
>
> Your attachments did not come through - can you put them on pastebin?
>
> -Todd
>
> On Tue, Apr 6, 2010 at 3:37 PM, Edson Ramiro <er...@gmail.com> wrote:
> > Hi Todd,
> >
> > I'm getting this behavior in another cluster too, there the same thing
> > happens.
> > and as I don't have a jstack installed in the first cluster and
> > I'm not the admin, I'm sending the results of the second cluster.
> >
> > These are the results:
> >
> > [erlf07@cohiba ~ ]$ jstack -l 22510
> > 22510: well-known file is not secure
> > [erlf07@cohiba ~ ]$ jstack -l 3836
> > 3836: well-known file is not secure
> >
> > The jstack -F result is in thread_dump files and the jstack -m result is
> in
> > java_native_frames.
> >
> > The files ending with nn are the namenode results and the files ending
> with
> > dn are the datanode results.
> >
> > Thanks,
> >
> > Edson Ramiro
> >
> >
> > On 6 April 2010 18:19, Todd Lipcon <to...@cloudera.com> wrote:
> >>
> >> Hi Edson,
> >>
> >> Can you please run "jstack" on the daemons in question and paste the
> >> output here?
> >>
> >> -Todd
> >>
> >> On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro <er...@gmail.com>
> wrote:
> >> > Hi all,
> >> >
> >> > I configured the Hadoop in a cluster and the NameNode and JobTracker
> are
> >> > running ok, but the DataNode and TaskTracker Doesn't start, they stop
> >> > and
> >> > keep waiting
> >> > when they are going to start the Jetty
> >> >
> >> > I observed that Jetty can't start the _SelectChannelConnector_
> >> >
> >> > Is there any Jetty configuration that should be changed ?
> >> >
> >> > There is no log message in the NN and JT when I try to start the DN
> and
> >> > TT.
> >> >
> >> > The kernel I'm using is:
> >> > Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64
> GNU/Linux
> >> >
> >> > This is the message when I start the DN. It happens with TT too.
> >> >
> >> > ramiro@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode
> >> > 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG:
> >> > /************************************************************
> >> > STARTUP_MSG: Starting DataNode
> >> > STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115
> >> > STARTUP_MSG: args = []
> >> > STARTUP_MSG: version = 0.20.1+169.56
> >> > STARTUP_MSG: build = -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3;
> >> > compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010
> >> > ************************************************************/
> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Registered
> >> > FSDatasetStatusMBean
> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at 50010
> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is
> 1048576
> >> > bytes/s
> >> > 10/04/06 16:24:14 INFO mortbay.log: Logging to
> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> >> > org.mortbay.log.Slf4jLog
> >> > 10/04/06 16:24:14 INFO http.HttpServer: Port returned by
> >> > webServer.getConnectors()[0].getLocalPort() before open() is -1.
> Opening
> >> > the
> >> > listener on 50075
> >> > 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort()
> returned
> >> > 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
> >> > 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075
> >> > 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14
> >> >
> >> > Thanks in Advance,
> >> >
> >> > Edson Ramiro
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Jetty can't start the SelectChannelConnector
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Edson,
Your attachments did not come through - can you put them on pastebin?
-Todd
On Tue, Apr 6, 2010 at 3:37 PM, Edson Ramiro <er...@gmail.com> wrote:
> Hi Todd,
>
> I'm getting this behavior in another cluster too, there the same thing
> happens.
> and as I don't have a jstack installed in the first cluster and
> I'm not the admin, I'm sending the results of the second cluster.
>
> These are the results:
>
> [erlf07@cohiba ~ ]$ jstack -l 22510
> 22510: well-known file is not secure
> [erlf07@cohiba ~ ]$ jstack -l 3836
> 3836: well-known file is not secure
>
> The jstack -F result is in thread_dump files and the jstack -m result is in
> java_native_frames.
>
> The files ending with nn are the namenode results and the files ending with
> dn are the datanode results.
>
> Thanks,
>
> Edson Ramiro
>
>
> On 6 April 2010 18:19, Todd Lipcon <to...@cloudera.com> wrote:
>>
>> Hi Edson,
>>
>> Can you please run "jstack" on the daemons in question and paste the
>> output here?
>>
>> -Todd
>>
>> On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro <er...@gmail.com> wrote:
>> > Hi all,
>> >
>> > I configured the Hadoop in a cluster and the NameNode and JobTracker are
>> > running ok, but the DataNode and TaskTracker Doesn't start, they stop
>> > and
>> > keep waiting
>> > when they are going to start the Jetty
>> >
>> > I observed that Jetty can't start the _SelectChannelConnector_
>> >
>> > Is there any Jetty configuration that should be changed ?
>> >
>> > There is no log message in the NN and JT when I try to start the DN and
>> > TT.
>> >
>> > The kernel I'm using is:
>> > Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64 GNU/Linux
>> >
>> > This is the message when I start the DN. It happens with TT too.
>> >
>> > ramiro@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode
>> > 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG:
>> > /************************************************************
>> > STARTUP_MSG: Starting DataNode
>> > STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115
>> > STARTUP_MSG: args = []
>> > STARTUP_MSG: version = 0.20.1+169.56
>> > STARTUP_MSG: build = -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3;
>> > compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010
>> > ************************************************************/
>> > 10/04/06 16:24:14 INFO datanode.DataNode: Registered
>> > FSDatasetStatusMBean
>> > 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at 50010
>> > 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is 1048576
>> > bytes/s
>> > 10/04/06 16:24:14 INFO mortbay.log: Logging to
>> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> > org.mortbay.log.Slf4jLog
>> > 10/04/06 16:24:14 INFO http.HttpServer: Port returned by
>> > webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening
>> > the
>> > listener on 50075
>> > 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort() returned
>> > 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
>> > 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075
>> > 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14
>> >
>> > Thanks in Advance,
>> >
>> > Edson Ramiro
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Jetty can't start the SelectChannelConnector
Posted by Edson Ramiro <er...@gmail.com>.
Hi Todd,
I'm getting this behavior in another cluster too, there the same thing
happens.
and as I don't have a jstack installed in the first cluster and
I'm not the admin, I'm sending the results of the second cluster.
These are the results:
[erlf07@cohiba ~ ]$ jstack -l 22510
22510: well-known file is not secure
[erlf07@cohiba ~ ]$ jstack -l 3836
3836: well-known file is not secure
The jstack -F result is in thread_dump files and the jstack -m result is in
java_native_frames.
The files ending with nn are the namenode results and the files ending with
dn are the datanode results.
Thanks,
Edson Ramiro
On 6 April 2010 18:19, Todd Lipcon <to...@cloudera.com> wrote:
> Hi Edson,
>
> Can you please run "jstack" on the daemons in question and paste the
> output here?
>
> -Todd
>
> On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro <er...@gmail.com> wrote:
> > Hi all,
> >
> > I configured the Hadoop in a cluster and the NameNode and JobTracker are
> > running ok, but the DataNode and TaskTracker Doesn't start, they stop and
> > keep waiting
> > when they are going to start the Jetty
> >
> > I observed that Jetty can't start the _SelectChannelConnector_
> >
> > Is there any Jetty configuration that should be changed ?
> >
> > There is no log message in the NN and JT when I try to start the DN and
> TT.
> >
> > The kernel I'm using is:
> > Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64 GNU/Linux
> >
> > This is the message when I start the DN. It happens with TT too.
> >
> > ramiro@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode
> > 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting DataNode
> > STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 0.20.1+169.56
> > STARTUP_MSG: build = -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3;
> > compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010
> > ************************************************************/
> > 10/04/06 16:24:14 INFO datanode.DataNode: Registered FSDatasetStatusMBean
> > 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at 50010
> > 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is 1048576
> > bytes/s
> > 10/04/06 16:24:14 INFO mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 10/04/06 16:24:14 INFO http.HttpServer: Port returned by
> > webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening
> the
> > listener on 50075
> > 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort() returned
> > 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
> > 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075
> > 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14
> >
> > Thanks in Advance,
> >
> > Edson Ramiro
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Jetty can't start the SelectChannelConnector
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Edson,
Can you please run "jstack" on the daemons in question and paste the
output here?
-Todd
On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro <er...@gmail.com> wrote:
> Hi all,
>
> I configured the Hadoop in a cluster and the NameNode and JobTracker are
> running ok, but the DataNode and TaskTracker Doesn't start, they stop and
> keep waiting
> when they are going to start the Jetty
>
> I observed that Jetty can't start the _SelectChannelConnector_
>
> Is there any Jetty configuration that should be changed ?
>
> There is no log message in the NN and JT when I try to start the DN and TT.
>
> The kernel I'm using is:
> Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64 GNU/Linux
>
> This is the message when I start the DN. It happens with TT too.
>
> ramiro@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode
> 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.1+169.56
> STARTUP_MSG: build = -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3;
> compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010
> ************************************************************/
> 10/04/06 16:24:14 INFO datanode.DataNode: Registered FSDatasetStatusMBean
> 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at 50010
> 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is 1048576
> bytes/s
> 10/04/06 16:24:14 INFO mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 10/04/06 16:24:14 INFO http.HttpServer: Port returned by
> webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the
> listener on 50075
> 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort() returned
> 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
> 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075
> 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14
>
> Thanks in Advance,
>
> Edson Ramiro
>
--
Todd Lipcon
Software Engineer, Cloudera