You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Tom Hall <th...@gmail.com> on 2011/10/31 11:57:20 UTC
Errors on 2 new datanodes
My 2 new datanodes keep getting blacklisted, most of the results on
google come up with
<property>
<name>dfs.datanode.max.xcievers</name>
<value>4096</value>
</property>
<property>
<name>dfs.datanode.handler.count</name>
<value>10</value>
</property>
Can anyone help?
2011-10-31 01:17:22,609 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(192.168.35.117:50010,
storageID=DS-1350419318-127.0.1.1-50010-1317048473534, infoPort=50075,
ipcPort=50020):Failed to transfer blk_-5759512173538110526_48294550 to
192.168.35.118:50010 got java.net.SocketTimeoutException: 485000
millis timeout while waiting for channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/192.168.35.117:50368
remote=/192.168.35.118:50010]
at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:350)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:436)
at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1370)
at java.lang.Thread.run(Thread.java:662)
2011-10-31 01:44:29,541 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
BlockReceiver.lastNodeRun:
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)
at sun.nio.ch.IOUtil.write(IOUtil.java:43)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:793)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
at java.lang.Thread.run(Thread.java:662)
2011-10-31 01:44:29,541 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError:
exception:
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)
at sun.nio.ch.IOUtil.write(IOUtil.java:43)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:793)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
at java.lang.Thread.run(Thread.java:662)
java.io.EOFException: while trying to read 65557 bytes
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:270)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:314)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
Re: Errors on 2 new datanodes
Posted by Tom Hall <th...@gmail.com>.
And on the namenode.
2011-10-31 01:54:11,236 WARN
org.apache.hadoop.hdfs.server.namenode.NameNode: - could not copy
block blk_2319728700078806245_42071514 to /lost+found/tmp/FILENAME
2011-10-31 01:54:11,576 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception for block
blk_4603681002261830265_48295303java.io.IOException: Bad response 1
for block blk_4603681002261830265_48295303 from datanode
192.168.35.117:50010
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2668)
2011-10-31 01:54:11,578 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_4603681002261830265_48295303 bad datanode[2]
192.168.35.117:50010
2011-10-31 01:54:11,578 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_4603681002261830265_48295303 in pipeline
192.168.35.253:50010, 192.168.35.229:50010, 192.168.35.117:50010: bad
datanode 192.168.35.117:50010
2011-10-31 01:54:11,740 WARN org.apache.hadoop.hdfs.server.namenode.NameNode:
- moved corrupted file /tmp/FILENAME to /lost+found
Tom
On Mon, Oct 31, 2011 at 11:25 AM, Tom Hall <th...@gmail.com> wrote:
> Also
> 2011-10-31 11:19:54,728 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(192.168.35.117:50010,
> storageID=DS-1350419318-127.0.1.1-50010-1317048473534, infoPort=50075,
> ipcPort=50020):DataXceiver
> org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException:
> Block blk_-5143210290039847658_37294862 is valid, and cannot be
> written to.
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1314)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:99)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:318)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>
> Is it a permissions issue?
> Why is 127.0.1.1 listed in the path there?
>
> Cheers,
> Tom
>
>
> On Mon, Oct 31, 2011 at 10:57 AM, Tom Hall <th...@gmail.com> wrote:
>> My 2 new datanodes keep getting blacklisted, most of the results on
>> google come up with
>> <property>
>> <name>dfs.datanode.max.xcievers</name>
>> <value>4096</value>
>> </property>
>>
>> <property>
>> <name>dfs.datanode.handler.count</name>
>> <value>10</value>
>> </property>
>> Can anyone help?
>>
>>
>> 2011-10-31 01:17:22,609 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> DatanodeRegistration(192.168.35.117:50010,
>> storageID=DS-1350419318-127.0.1.1-50010-1317048473534, infoPort=50075,
>> ipcPort=50020):Failed to transfer blk_-5759512173538110526_48294550 to
>> 192.168.35.118:50010 got java.net.SocketTimeoutException: 485000
>> millis timeout while waiting for channel to be ready for write. ch :
>> java.nio.channels.SocketChannel[connected local=/192.168.35.117:50368
>> remote=/192.168.35.118:50010]
>> at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
>> at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
>> at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
>> at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:350)
>> at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:436)
>> at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1370)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> 2011-10-31 01:44:29,541 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
>> BlockReceiver.lastNodeRun:
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)
>> at sun.nio.ch.IOUtil.write(IOUtil.java:43)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:793)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>> at java.lang.Thread.run(Thread.java:662)
>> 2011-10-31 01:44:29,541 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError:
>> exception:
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)
>> at sun.nio.ch.IOUtil.write(IOUtil.java:43)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:793)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> java.io.EOFException: while trying to read 65557 bytes
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:270)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:314)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>>
>
Re: Errors on 2 new datanodes
Posted by Tom Hall <th...@gmail.com>.
Also
2011-10-31 11:19:54,728 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(192.168.35.117:50010,
storageID=DS-1350419318-127.0.1.1-50010-1317048473534, infoPort=50075,
ipcPort=50020):DataXceiver
org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException:
Block blk_-5143210290039847658_37294862 is valid, and cannot be
written to.
at org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1314)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:99)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:318)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
Is it a permissions issue?
Why is 127.0.1.1 listed in the path there?
Cheers,
Tom
On Mon, Oct 31, 2011 at 10:57 AM, Tom Hall <th...@gmail.com> wrote:
> My 2 new datanodes keep getting blacklisted, most of the results on
> google come up with
> <property>
> <name>dfs.datanode.max.xcievers</name>
> <value>4096</value>
> </property>
>
> <property>
> <name>dfs.datanode.handler.count</name>
> <value>10</value>
> </property>
> Can anyone help?
>
>
> 2011-10-31 01:17:22,609 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(192.168.35.117:50010,
> storageID=DS-1350419318-127.0.1.1-50010-1317048473534, infoPort=50075,
> ipcPort=50020):Failed to transfer blk_-5759512173538110526_48294550 to
> 192.168.35.118:50010 got java.net.SocketTimeoutException: 485000
> millis timeout while waiting for channel to be ready for write. ch :
> java.nio.channels.SocketChannel[connected local=/192.168.35.117:50368
> remote=/192.168.35.118:50010]
> at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
> at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
> at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
> at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:350)
> at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:436)
> at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1370)
> at java.lang.Thread.run(Thread.java:662)
>
> 2011-10-31 01:44:29,541 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
> BlockReceiver.lastNodeRun:
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)
> at sun.nio.ch.IOUtil.write(IOUtil.java:43)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:793)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
> at java.lang.Thread.run(Thread.java:662)
> 2011-10-31 01:44:29,541 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError:
> exception:
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)
> at sun.nio.ch.IOUtil.write(IOUtil.java:43)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:793)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
> at java.lang.Thread.run(Thread.java:662)
>
> java.io.EOFException: while trying to read 65557 bytes
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:270)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:314)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>