You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sagar Naik <sn...@attributor.com> on 2008/12/15 00:07:41 UTC

occasional createBlockException in Hadoop .18.1


CLIENT EXCEPTION:

2008-12-14 08:41:46,919 [Thread-90] INFO 
org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream 
java.net.SocketTimeoutException: 69000 millis timeout while waiting for 
channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/10.50.80.133:54045 
remote=/10.50.80.108:50010] 2008-12-14 08:41:46,919 [Thread-90] INFO 
org.apache.hadoop.dfs.DFSClient: Abandoning block 
blk_-7364265396616885025_5870078 2008-12-14 08:41:46,920 [Thread-90] 
INFO org.apache.hadoop.dfs.DFSClient: Waiting to find target node: 
10.50.80.108:50010


DATANODE

2008-12-14 08:40:39,215 INFO org.apache.hadoop.dfs.DataNode: Receiving 
block blk_-7364265396616885025_5870078 src: /10.50.80.133:54045 dest: 
/10.50.80.133:50010
.
.
.
.
.  

I occasionally see the datanode as deadnode. When the datanode is 
deadnode, I see the du forked from datanode. The "du" is seen in "D" state



Any pointers to debug this information would help me

-Sagar

Re: occasional createBlockException in Hadoop .18.1

Posted by Sagar Naik <sn...@attributor.com>.
Hi,
Some data points on this issue.
1) "du" runs for 20-30 secs.
2) after some time , I dont see any activity in datanode logs
3) I cant even jstack the datanode (forced it , gave me a 
DebuggerException, double checked the pid), the datanode:50075/stacks 
takes forever to respond

I can telnet to datanode:50010

I think, the disk is bad or something

Pl suggest some pointers to analyze this problem

-Sagar
Sagar Naik wrote:
>
>
> CLIENT EXCEPTION:
>
> 2008-12-14 08:41:46,919 [Thread-90] INFO 
> org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream 
> java.net.SocketTimeoutException: 69000 millis timeout while waiting 
> for channel to be ready for read. ch : 
> java.nio.channels.SocketChannel[connected local=/10.50.80.133:54045 
> remote=/10.50.80.108:50010] 2008-12-14 08:41:46,919 [Thread-90] INFO 
> org.apache.hadoop.dfs.DFSClient: Abandoning block 
> blk_-7364265396616885025_5870078 2008-12-14 08:41:46,920 [Thread-90] 
> INFO org.apache.hadoop.dfs.DFSClient: Waiting to find target node: 
> 10.50.80.108:50010
>
>
> DATANODE
>
> 2008-12-14 08:40:39,215 INFO org.apache.hadoop.dfs.DataNode: Receiving 
> block blk_-7364265396616885025_5870078 src: /10.50.80.133:54045 dest: 
> /10.50.80.133:50010
> .
> .
> .
> .
> . 
> I occasionally see the datanode as deadnode. When the datanode is 
> deadnode, I see the du forked from datanode. The "du" is seen in "D" 
> state
>
>
>
> Any pointers to debug this information would help me
>
> -Sagar