You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by 杨杰 <xt...@gmail.com> on 2011/04/11 16:06:18 UTC
What's the matter: problem cleaning system directory: null
HI,
Some mr tasks upon small files had run on our hadoop cluster these
days, with not such high
work load. While when i check it tonight, the cluster refused to
response. So i restart the hdfs
& mapred.
BUT unexcepted exceptions were thrown when starting~ and even the
"hadoop fs -ls" command
could not get any response. The system seems to "hang" there.
btw: at start, there is a piece of message telling "22 port connection
timeout". I checked the
$HADOOP_SSH_OPT in hadoop-env.sh. There, connection time out is set to
1, so i changed it
to 1000. Then the error disappeared, but the following exception goes on~
Thanks for help !
2011-04-11 21:39:57,028 INFO org.apache.hadoop.mapred.JobTracker:
problem cleaning system directory: null
java.io.IOException: Call to job-tracker/192.168.14.210:8020 failed on
local exception: java.io.IOException: Connection reset by peer
at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
at org.apache.hadoop.ipc.Client.call(Client.java:742)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:105)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:177)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1607)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:174)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3528)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:206)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
--
Yang Jie(杨杰)
hi.baidu.com/thinkdifferent
Group of CLOUD, Xi'an Jiaotong University
Department of Computer Science and Technology, Xi’an Jiaotong University
PHONE: 86 1346888 3723
TEL: 86 29 82665263 EXT. 608
MSN: xtyangjie2004@yahoo.com.cn
freedom is not free.
Re: What's the matter: problem cleaning system directory: null
Posted by 杨杰 <xt...@gmail.com>.
I found it similiar to
HADDOP-3027(https://issues.apache.org/jira/browse/HADOOP-3027),
but its error msg is:
org.apache.hadoop.mapred.JobTracker: problem cleaning system
directory: /tmp/hadoop/mapred/system
while mine is:
org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: null
I don't know if its a key issue.
On Mon, Apr 11, 2011 at 10:06 PM, 杨杰 <xt...@gmail.com> wrote:
> HI,
>
> Some mr tasks upon small files had run on our hadoop cluster these
> days, with not such high
> work load. While when i check it tonight, the cluster refused to
> response. So i restart the hdfs
> & mapred.
>
> BUT unexcepted exceptions were thrown when starting~ and even the
> "hadoop fs -ls" command
> could not get any response. The system seems to "hang" there.
>
> btw: at start, there is a piece of message telling "22 port connection
> timeout". I checked the
> $HADOOP_SSH_OPT in hadoop-env.sh. There, connection time out is set to
> 1, so i changed it
> to 1000. Then the error disappeared, but the following exception goes on~
>
> Thanks for help !
>
> 2011-04-11 21:39:57,028 INFO org.apache.hadoop.mapred.JobTracker:
> problem cleaning system directory: null
> java.io.IOException: Call to job-tracker/192.168.14.210:8020 failed on
> local exception: java.io.IOException: Connection reset by peer
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
> at org.apache.hadoop.ipc.Client.call(Client.java:742)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:105)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:177)
> at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1607)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:174)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3528)
> Caused by: java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:206)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.FilterInputStream.read(FilterInputStream.java:116)
> at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> at java.io.DataInputStream.readInt(DataInputStream.java:370)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
>
> --
> Yang Jie(杨杰)
> hi.baidu.com/thinkdifferent
>
> Group of CLOUD, Xi'an Jiaotong University
> Department of Computer Science and Technology, Xi’an Jiaotong University
>
> PHONE: 86 1346888 3723
> TEL: 86 29 82665263 EXT. 608
> MSN: xtyangjie2004@yahoo.com.cn
>
> freedom is not free.
>
--
Yang Jie(杨杰)
hi.baidu.com/thinkdifferent
Group of CLOUD, Xi'an Jiaotong University
Department of Computer Science and Technology, Xi’an Jiaotong University
PHONE: 86 1346888 3723
TEL: 86 29 82665263 EXT. 608
MSN: xtyangjie2004@yahoo.com.cn
freedom is not free.