Re: DFS get blocked when writing a file.

[Iván de Prado <iv...@gmail.com>]

Thanks very much for the help. 

I will investigate more about that. 

Iván

El lun, 31-03-2008 a las 11:11 -0700, Raghu Angadi escribió:
> Iván,
> 
> Whether this was expected or an error depends on what happened on the 
> client. This could happen and would not be a bug if client was killed 
> for some other reason for e.g. But if client is also similarly surprised 
>   then its a different case.
> 
> You could grep for this block in NameNode log and client. If you are 
> still interested in looking into this, I would suggest opening a jira.
> 
> Raghu.
> 
> Iván de Prado wrote:
> > Thanks, 
> > 
> > I have tried with the trunk version and now the exception "Trying to
> > change block file offset of block blk_... to ... but actual size of file
> > is ..." has disappeared and the jobs don't seems to get blocked.
> > 
> > But I have another "Broken Pipe" and "EOF" exceptions in the dfs logs.
> > They seems similar to https://issues.apache.org/jira/browse/HADOOP-2042
> > ticket. The Jobs ends but not sure if they are executed smoothly. are
> > these exceptions normal? As example, the exceptions for the block
> > (6801211507359331627) appears in two nodes (I have 2 as replication) and
> > looks like:
> > 
> > hn2: 2008-03-31 05:03:13,736 INFO org.apache.hadoop.dfs.DataNode:
> > Datanode 0 forwarding connect ack to upstream firstbadlink is 
> > hn2: 2008-03-31 05:03:14,507 INFO org.apache.hadoop.dfs.DataNode:
> > Receiving block blk_6801211507359331627 src: /172.16.3.6:38218
> > dest: /172.16.3.6:50010
> > 
> > hn2: 2008-03-31 05:04:14,528 INFO org.apache.hadoop.dfs.DataNode:
> > Exception in receiveBlock for block blk_6801211507359331627
> > java.io.EOFException
> > hn2: 2008-03-31 05:04:14,528 INFO org.apache.hadoop.dfs.DataNode:
> > PacketResponder 0 for block blk_6801211507359331627 Interrupted.
> > hn2: 2008-03-31 05:04:14,528 INFO org.apache.hadoop.dfs.DataNode:
> > PacketResponder 0 for block blk_6801211507359331627 terminating
> > hn2: 2008-03-31 05:04:14,530 INFO org.apache.hadoop.dfs.DataNode:
> > writeBlock blk_6801211507359331627 received exception
> > java.io.EOFException
> > hn2: 2008-03-31 05:04:14,530 ERROR org.apache.hadoop.dfs.DataNode:
> > 172.16.3.4:50010:DataXceiver: java.io.EOFException
> > hn2:    at java.io.DataInputStream.readInt(DataInputStream.java:375)
> > hn2:    at org.apache.hadoop.dfs.DataNode
> > $BlockReceiver.receiveBlock(DataNode.java:2243)
> > hn2:    at org.apache.hadoop.dfs.DataNode
> > $DataXceiver.writeBlock(DataNode.java:1157)
> > hn2:    at org.apache.hadoop.dfs.DataNode
> > $DataXceiver.run(DataNode.java:938)
> > hn2:    at java.lang.Thread.run(Thread.java:619)
> > 
> > hn4: 2008-03-31 05:03:13,590 INFO org.apache.hadoop.dfs.DataNode:
> > Datanode 0 forwarding connect ack to upstream firstbadlink is 
> > hn4: 2008-03-31 05:03:14,506 INFO org.apache.hadoop.dfs.DataNode:
> > Receiving block blk_6801211507359331627 src: /172.16.3.6:41112
> > dest: /172.16.3.6:50010
> > 
> > hn4: 2008-03-31 05:03:26,825 INFO org.apache.hadoop.dfs.DataNode:
> > Exception in receiveBlock for block blk_6801211507359331627
> > java.io.EOFException
> > 
> > hn4: 2008-03-31 05:04:14,524 INFO org.apache.hadoop.dfs.DataNode:
> > PacketResponder blk_6801211507359331627 1 Exception
> > java.net.SocketException: Broken pipe
> > hn4:    at java.net.SocketOutputStream.socketWrite0(Native Method)
> > hn4:    at
> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> > hn4:    at
> > java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> > hn4:    at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
> > hn4:    at org.apache.hadoop.dfs.DataNode
> > $PacketResponder.run(DataNode.java:1825)
> > hn4:    at java.lang.Thread.run(Thread.java:619)
> > hn4: 
> > hn4: 2008-03-31 05:04:14,525 INFO org.apache.hadoop.dfs.DataNode:
> > PacketResponder 1 for block blk_6801211507359331627 terminating
> > hn4: 2008-03-31 05:04:14,525 INFO org.apache.hadoop.dfs.DataNode:
> > writeBlock blk_6801211507359331627 received exception
> > java.io.EOFException
> > hn4: 2008-03-31 05:04:14,526 ERROR org.apache.hadoop.dfs.DataNode:
> > 172.16.3.6:50010:DataXceiver: java.io.EOFException
> > hn4:    at java.io.DataInputStream.readInt(DataInputStream.java:375)
> > hn4:    at org.apache.hadoop.dfs.DataNode
> > $BlockReceiver.receiveBlock(DataNode.java:2243)
> > hn4:    at org.apache.hadoop.dfs.DataNode
> > $DataXceiver.writeBlock(DataNode.java:1157)
> > hn4:    at org.apache.hadoop.dfs.DataNode
> > $DataXceiver.run(DataNode.java:938)
> > hn4:    at java.lang.Thread.run(Thread.java:619)
> > hn4: 
> > 
> > Many thanks, 
> > 
> > Iván de Prado Alonso
> > http://ivandeprado.blogspot.com/