You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by 오석근 <oh...@gmail.com> on 2007/03/16 02:28:52 UTC

Datanode VM crashed

Hi,
My configuration : hadoop-0.11.3
I ran m/r task. I monitoring http://namenode:50070/dfshealth.jsp page.
Then one of datanode dead with followed hs_err log file.
What's wrong?

====hs err log file ====
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# Internal Error (53484152454432554E54494D450E43505001A3), pid=12317,
tid=0xa8dc00
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (diablo-1.5.0_07-b01 mixed
mode)

--------------- T H R E A D ---------------

Current thread (0x0000000000813000): JavaThread
"org.apache.hadoop.dfs.DataNode$DataXceiver@36125b4f" daemon
[_thread_in_Java, id=11066368]

Stack: [0x00007ffffe5eb000,0x00007ffffe6eb000), sp=0x00007ffffe6e7c40,
free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
V [libjvm.so+0x7ddc49]
V [libjvm.so+0x476da6]
V [libjvm.so+0x75a3ff]
V [libjvm.so+0x6f4632]


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
=>0x0000000000813000 JavaThread
"org.apache.hadoop.dfs.DataNode$DataXceiver@36125b4f" daemon
[_thread_in_Java, id=11066368]
0x0000000000813400 JavaThread
"org.apache.hadoop.dfs.DataNode$DataXceiver@68d88d5" daemon
[_thread_in_native, id=11588608]
0x0000000000810800 JavaThread
"org.apache.hadoop.dfs.DataNode$DataXceiver@765971c3" daemon
[_thread_in_native, id=11980800]
0x00000000007c2400 JavaThread
"org.apache.hadoop.dfs.DataNode$DataXceiver@15651229" daemon
[_thread_in_native, id=10022912]
0x000000000082e400 JavaThread
"org.apache.hadoop.dfs.DataNode$DataXceiver@66f877e9" daemon
[_thread_in_native, id=10009600]
0x0000000000621400 JavaThread
"org.apache.hadoop.dfs.DataNode$DataXceiver@73b7a261" daemon
[_thread_in_native, id=10189824]
0x0000000000621000 JavaThread
"org.apache.hadoop.dfs.DataNode$DataXceiver@66b967ed" daemon
[_thread_in_native, id=11063296]
0x0000000000b14400 JavaThread
"org.apache.hadoop.dfs.DataNode$DataXceiveServer@165973ea" daemon
[_thread_in_native, id=11619328]
0x0000000000b0dc00 JavaThread "DataNode:
[/home2/gfs/filesystem/data,/home/gfs/filesystem/data]" daemon
[_thread_blocked, id=11616256]
0x0000000000afb000 JavaThread "SocketListener0-1" [_thread_blocked,
id=11515904]
0x0000000000af9800 JavaThread "SocketListener0-0" [_thread_blocked,
id=11508736]
0x0000000000af9000 JavaThread "Acceptor
ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=50075]"
[_thread_in_native, id=11506688]
0x0000000000b33000 JavaThread "SessionScavenger" daemon
[_thread_blocked, id=11744256]
0x000000000088c800 JavaThread "org.apache.hadoop.io.ObjectWritable
Connection Culler" daemon [_thread_blocked, id=8965120]
0x00000000006f5400 JavaThread "Low Memory Detector" daemon
[_thread_blocked, id=7319552]
0x00000000006ef400 JavaThread "CompilerThread1" daemon [_thread_blocked,
id=7294976]
0x00000000006e9400 JavaThread "CompilerThread0" daemon [_thread_blocked,
id=7270400]
0x00000000006be400 JavaThread "AdapterThread" daemon [_thread_blocked,
id=7245824]
0x00000000006a8400 JavaThread "Signal Dispatcher" daemon
[_thread_blocked, id=7069696]
0x000000000069d800 JavaThread "Finalizer" daemon [_thread_blocked,
id=6979584]
0x000000000069d000 JavaThread "Reference Handler" daemon
[_thread_blocked, id=6935552]
0x0000000000527000 JavaThread "main" [_thread_blocked, id=5332992]

Other Threads:
0x000000000065f600 VMThread [id=6470656]
0x000000000051dc00 WatcherThread [id=7344128]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
PSYoungGen total 2944K, used 2313K [0x0000000836860000,
0x0000000836b70000, 0x000000084b5b0000)
eden space 2752K, 80% used
[0x0000000836860000,0x0000000836a8a4e8,0x0000000836b10000)
from space 192K, 50% used
[0x0000000836b40000,0x0000000836b58000,0x0000000836b70000)
to space 192K, 0% used
[0x0000000836b10000,0x0000000836b10000,0x0000000836b40000)
PSOldGen total 10240K, used 7923K [0x000000080cdb0000,
0x000000080d7b0000, 0x0000000836860000)
object space 10240K, 77% used
[0x000000080cdb0000,0x000000080d56cd60,0x000000080d7b0000)
PSPermGen total 21504K, used 10830K [0x0000000807bb0000,
0x00000008090b0000, 0x000000080cdb0000)
object space 21504K, 50% used
[0x0000000807bb0000,0x0000000808643868,0x00000008090b0000)
PSPermGen total 21504K, used 10830K [0x0000000807bb0000,
0x00000008090b0000, 0x000000080cdb0000)
object space 21504K, 50% used
[0x0000000807bb0000,0x0000000808643868,0x00000008090b0000)

Dynamic libraries:
0x0000000000400000 /home/toolkit/java/bin/java
0x000000080063b000 /lib/libz.so.3
0x000000080074f000 /lib/libpthread.so.2
0x000000080087a000 /lib/libc.so.6
0x0000000800a89000
/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/server/libjvm.so
0x0000000801516000 /usr/lib/libstdc++.so.5
0x000000080170d000 /lib/libm.so.4
0x0000000801829000
/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/native_threads/libhpi.so
0x0000000801935000
/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libverify.so
0x0000000801a44000
/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libjava.so
0x0000000801b6c000
/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libzip.so
0x000000084c6f8000
/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libnet.so
0x000000084c80b000
/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libnio.so
0x000000084c912000
/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libmanagement.so
0x000000080050a000 /libexec/ld-elf.so.1

VM Arguments:
jvm_args: -Xmx1000m -Dhadoop.log.dir=/home/nutch/logs
-Dhadoop.log.file=hadoop-nutch-datanode-nutch1.log
-Dhadoop.home.dir=/home/nutch -Dhadoop.id.str=nutch
-Dhadoop.root.logger=INFO,DRFA
-Djava.library.path=/home/nutch/lib/native/FreeBSD-amd64-64:/home/nutch/lib:/home/toolkit/iconv/lib:/usr/local/lib:/home/nutch/lib:/usr/local/lib:/usr/local/lib:/home/nutch/lib
java_command: org.apache.hadoop.dfs.DataNode
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/home/toolkit/java
PATH=/home/toolkit/java/bin:/home/toolkit/ant/bin:/bin:/usr/local/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/sbin:
LD_LIBRARY_PATH=/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/server:/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64:/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/../lib/amd64:/home/nutch/lib:/usr/local/lib
SHELL=/bin/csh
HOSTTYPE=FreeBSD
OSTYPE=FreeBSD
MACHTYPE=unknown

Signal Handlers:
SIGSEGV: [libjvm.so+0x7de650], sa_mask[0]=0xffffffff, sa_flags=0x00000002
SIGBUS: [libjvm.so+0x7de650], sa_mask[0]=0xffffffff, sa_flags=0x00000002
SIGFPE: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
SIGPIPE: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
SIGILL: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
SIGUSR1: [libjvm.so+0x6f30c0], sa_mask[0]=0x00000000, sa_flags=0x00000040
SIGUSR2: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000002
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x6f1b30], sa_mask[0]=0xffffffff, sa_flags=0x00000002
SIGTERM: [libjvm.so+0x6f1b30], sa_mask[0]=0xffffffff, sa_flags=0x00000002


--------------- S Y S T E M ---------------

OS:FreeBSD
uname:FreeBSD 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Mon Feb 12 20:35:02
KST 2007 root@nutch:/usr/obj/usr/src/sys/NUTCH amd64
rlimit: STACK 524288k, CORE 0k, NOFILE 11095
CPU:total 2 amd64 3dnow

Memory: 4k page, physical 1856136k

vm_info: Java HotSpot(TM) 64-Bit Server VM (diablo-1.5.0_07-b01) for
freebsd-amd64, built on Sep 24 2006 16:09:01 by root with gcc 3.4.4
[FreeBSD] 20050518

Re: Datanode VM crashed

Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
I've had this kind of errors when the machine was broken -- this is a process 
error (JVM error), not program error. Check if your memory is working correctly 
(memtest).

Dawid

오석근 wrote:
> Hi,
> My configuration : hadoop-0.11.3
> I ran m/r task. I monitoring http://namenode:50070/dfshealth.jsp page.
> Then one of datanode dead with followed hs_err log file.
> What's wrong?
> 
> ====hs err log file ====
> #
> # An unexpected error has been detected by HotSpot Virtual Machine:
> #
> # Internal Error (53484152454432554E54494D450E43505001A3), pid=12317,
> tid=0xa8dc00
> #
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (diablo-1.5.0_07-b01 mixed
> mode)
> 
> --------------- T H R E A D ---------------
> 
> Current thread (0x0000000000813000): JavaThread
> "org.apache.hadoop.dfs.DataNode$DataXceiver@36125b4f" daemon
> [_thread_in_Java, id=11066368]
> 
> Stack: [0x00007ffffe5eb000,0x00007ffffe6eb000), sp=0x00007ffffe6e7c40,
> free space=1011k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
> C=native code)
> V [libjvm.so+0x7ddc49]
> V [libjvm.so+0x476da6]
> V [libjvm.so+0x75a3ff]
> V [libjvm.so+0x6f4632]
> 
> 
> --------------- P R O C E S S ---------------
> 
> Java Threads: ( => current thread )
> =>0x0000000000813000 JavaThread
> "org.apache.hadoop.dfs.DataNode$DataXceiver@36125b4f" daemon
> [_thread_in_Java, id=11066368]
> 0x0000000000813400 JavaThread
> "org.apache.hadoop.dfs.DataNode$DataXceiver@68d88d5" daemon
> [_thread_in_native, id=11588608]
> 0x0000000000810800 JavaThread
> "org.apache.hadoop.dfs.DataNode$DataXceiver@765971c3" daemon
> [_thread_in_native, id=11980800]
> 0x00000000007c2400 JavaThread
> "org.apache.hadoop.dfs.DataNode$DataXceiver@15651229" daemon
> [_thread_in_native, id=10022912]
> 0x000000000082e400 JavaThread
> "org.apache.hadoop.dfs.DataNode$DataXceiver@66f877e9" daemon
> [_thread_in_native, id=10009600]
> 0x0000000000621400 JavaThread
> "org.apache.hadoop.dfs.DataNode$DataXceiver@73b7a261" daemon
> [_thread_in_native, id=10189824]
> 0x0000000000621000 JavaThread
> "org.apache.hadoop.dfs.DataNode$DataXceiver@66b967ed" daemon
> [_thread_in_native, id=11063296]
> 0x0000000000b14400 JavaThread
> "org.apache.hadoop.dfs.DataNode$DataXceiveServer@165973ea" daemon
> [_thread_in_native, id=11619328]
> 0x0000000000b0dc00 JavaThread "DataNode:
> [/home2/gfs/filesystem/data,/home/gfs/filesystem/data]" daemon
> [_thread_blocked, id=11616256]
> 0x0000000000afb000 JavaThread "SocketListener0-1" [_thread_blocked,
> id=11515904]
> 0x0000000000af9800 JavaThread "SocketListener0-0" [_thread_blocked,
> id=11508736]
> 0x0000000000af9000 JavaThread "Acceptor
> ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=50075]"
> [_thread_in_native, id=11506688]
> 0x0000000000b33000 JavaThread "SessionScavenger" daemon
> [_thread_blocked, id=11744256]
> 0x000000000088c800 JavaThread "org.apache.hadoop.io.ObjectWritable
> Connection Culler" daemon [_thread_blocked, id=8965120]
> 0x00000000006f5400 JavaThread "Low Memory Detector" daemon
> [_thread_blocked, id=7319552]
> 0x00000000006ef400 JavaThread "CompilerThread1" daemon [_thread_blocked,
> id=7294976]
> 0x00000000006e9400 JavaThread "CompilerThread0" daemon [_thread_blocked,
> id=7270400]
> 0x00000000006be400 JavaThread "AdapterThread" daemon [_thread_blocked,
> id=7245824]
> 0x00000000006a8400 JavaThread "Signal Dispatcher" daemon
> [_thread_blocked, id=7069696]
> 0x000000000069d800 JavaThread "Finalizer" daemon [_thread_blocked,
> id=6979584]
> 0x000000000069d000 JavaThread "Reference Handler" daemon
> [_thread_blocked, id=6935552]
> 0x0000000000527000 JavaThread "main" [_thread_blocked, id=5332992]
> 
> Other Threads:
> 0x000000000065f600 VMThread [id=6470656]
> 0x000000000051dc00 WatcherThread [id=7344128]
> 
> VM state:not at safepoint (normal execution)
> 
> VM Mutex/Monitor currently owned by a thread: None
> 
> Heap
> PSYoungGen total 2944K, used 2313K [0x0000000836860000,
> 0x0000000836b70000, 0x000000084b5b0000)
> eden space 2752K, 80% used
> [0x0000000836860000,0x0000000836a8a4e8,0x0000000836b10000)
> from space 192K, 50% used
> [0x0000000836b40000,0x0000000836b58000,0x0000000836b70000)
> to space 192K, 0% used
> [0x0000000836b10000,0x0000000836b10000,0x0000000836b40000)
> PSOldGen total 10240K, used 7923K [0x000000080cdb0000,
> 0x000000080d7b0000, 0x0000000836860000)
> object space 10240K, 77% used
> [0x000000080cdb0000,0x000000080d56cd60,0x000000080d7b0000)
> PSPermGen total 21504K, used 10830K [0x0000000807bb0000,
> 0x00000008090b0000, 0x000000080cdb0000)
> object space 21504K, 50% used
> [0x0000000807bb0000,0x0000000808643868,0x00000008090b0000)
> PSPermGen total 21504K, used 10830K [0x0000000807bb0000,
> 0x00000008090b0000, 0x000000080cdb0000)
> object space 21504K, 50% used
> [0x0000000807bb0000,0x0000000808643868,0x00000008090b0000)
> 
> Dynamic libraries:
> 0x0000000000400000 /home/toolkit/java/bin/java
> 0x000000080063b000 /lib/libz.so.3
> 0x000000080074f000 /lib/libpthread.so.2
> 0x000000080087a000 /lib/libc.so.6
> 0x0000000800a89000
> /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/server/libjvm.so
> 0x0000000801516000 /usr/lib/libstdc++.so.5
> 0x000000080170d000 /lib/libm.so.4
> 0x0000000801829000
> /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/native_threads/libhpi.so
> 0x0000000801935000
> /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libverify.so
> 0x0000000801a44000
> /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libjava.so
> 0x0000000801b6c000
> /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libzip.so
> 0x000000084c6f8000
> /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libnet.so
> 0x000000084c80b000
> /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libnio.so
> 0x000000084c912000
> /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libmanagement.so
> 0x000000080050a000 /libexec/ld-elf.so.1
> 
> VM Arguments:
> jvm_args: -Xmx1000m -Dhadoop.log.dir=/home/nutch/logs
> -Dhadoop.log.file=hadoop-nutch-datanode-nutch1.log
> -Dhadoop.home.dir=/home/nutch -Dhadoop.id.str=nutch
> -Dhadoop.root.logger=INFO,DRFA
> -Djava.library.path=/home/nutch/lib/native/FreeBSD-amd64-64:/home/nutch/lib:/home/toolkit/iconv/lib:/usr/local/lib:/home/nutch/lib:/usr/local/lib:/usr/local/lib:/home/nutch/lib
> java_command: org.apache.hadoop.dfs.DataNode
> Launcher Type: SUN_STANDARD
> 
> Environment Variables:
> JAVA_HOME=/home/toolkit/java
> PATH=/home/toolkit/java/bin:/home/toolkit/ant/bin:/bin:/usr/local/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/sbin:
> LD_LIBRARY_PATH=/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/server:/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64:/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/../lib/amd64:/home/nutch/lib:/usr/local/lib
> SHELL=/bin/csh
> HOSTTYPE=FreeBSD
> OSTYPE=FreeBSD
> MACHTYPE=unknown
> 
> Signal Handlers:
> SIGSEGV: [libjvm.so+0x7de650], sa_mask[0]=0xffffffff, sa_flags=0x00000002
> SIGBUS: [libjvm.so+0x7de650], sa_mask[0]=0xffffffff, sa_flags=0x00000002
> SIGFPE: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
> SIGPIPE: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
> SIGILL: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
> SIGUSR1: [libjvm.so+0x6f30c0], sa_mask[0]=0x00000000, sa_flags=0x00000040
> SIGUSR2: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
> SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000002
> SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
> SIGQUIT: [libjvm.so+0x6f1b30], sa_mask[0]=0xffffffff, sa_flags=0x00000002
> SIGTERM: [libjvm.so+0x6f1b30], sa_mask[0]=0xffffffff, sa_flags=0x00000002
> 
> 
> --------------- S Y S T E M ---------------
> 
> OS:FreeBSD
> uname:FreeBSD 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Mon Feb 12 20:35:02
> KST 2007 root@nutch:/usr/obj/usr/src/sys/NUTCH amd64
> rlimit: STACK 524288k, CORE 0k, NOFILE 11095
> CPU:total 2 amd64 3dnow
> 
> Memory: 4k page, physical 1856136k
> 
> vm_info: Java HotSpot(TM) 64-Bit Server VM (diablo-1.5.0_07-b01) for
> freebsd-amd64, built on Sep 24 2006 16:09:01 by root with gcc 3.4.4
> [FreeBSD] 20050518
>