Loading data failed with timeout

[Mayuran Yogarajah <ma...@casalemedia.com>]

Hello, we were importing several TB of data overnight and it seemed one 
of the loads
failed.  We're running Hadoop 0.18.3, and there are 6 nodes in the 
cluster, all are
dual quad core with 6 gigs of ram.  We were using hadoop dfs -put to 
load the data
from both the namenode server and the secondary namenode server in 
parallel.  Space
is not the issue as we have many terabytes of space still remaining.

The load from the namenode is still going, the load from the secondary 
namenode failed.

This is the error we got:

dfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis
timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/x.x.x.x:55748 remote=/x.x.x.x:50010]
09/08/19 07:50:45 INFO dfs.DFSClient: Abandoning block blk_8258931159385721568_6046
09/08/19 07:50:59 INFO dfs.DFSClient: Waiting to find target node: x.x.x.x:50010
09/08/19 07:55:19 INFO dfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/x.x.x.x:47409 remote=/x.x.x.x:50010]
09/08/19 07:55:19 INFO dfs.DFSClient: Abandoning block blk_-6648842835159477749_6046
09/08/19 07:55:41 INFO dfs.DFSClient: Waiting to find target node: x.x.x.x:50010


I thought maybe whatever configuration value set to 69000 was too low, 
but there is nothing in hadoop-site or hadoop-default
using a value of 69000.

Can anyone shed some light on this?

thanks,
M

Re: Loading data failed with timeout

[Raghu Angadi <ra...@yahoo-inc.com>]

There is a config "socket.read.timeout" or "socket.timeout" set to 60000 
(60s). 69000 is based on that.

Mayuran Yogarajah wrote:
> Hello, we were importing several TB of data overnight and it seemed one 
> of the loads
> failed.  We're running Hadoop 0.18.3, and there are 6 nodes in the 
> cluster, all are
> dual quad core with 6 gigs of ram.  We were using hadoop dfs -put to 
> load the data
> from both the namenode server and the secondary namenode server in 
> parallel.  Space
> is not the issue as we have many terabytes of space still remaining.
> 
> The load from the namenode is still going, the load from the secondary 
> namenode failed.
> 
> This is the error we got:
> 
> dfs.DFSClient: Exception in createBlockOutputStream 
> java.net.SocketTimeoutException: 69000 millis
> timeout while waiting for channel to be ready for read. ch : 
> java.nio.channels.SocketChannel[connected local=/x.x.x.x:55748 
> remote=/x.x.x.x:50010]
> 09/08/19 07:50:45 INFO dfs.DFSClient: Abandoning block 
> blk_8258931159385721568_6046
> 09/08/19 07:50:59 INFO dfs.DFSClient: Waiting to find target node: 
> x.x.x.x:50010
> 09/08/19 07:55:19 INFO dfs.DFSClient: Exception in 
> createBlockOutputStream java.net.SocketTimeoutException: 69000 millis 
> timeout while waiting for
> channel to be ready for read. ch : 
> java.nio.channels.SocketChannel[connected local=/x.x.x.x:47409 
> remote=/x.x.x.x:50010]
> 09/08/19 07:55:19 INFO dfs.DFSClient: Abandoning block 
> blk_-6648842835159477749_6046
> 09/08/19 07:55:41 INFO dfs.DFSClient: Waiting to find target node: 
> x.x.x.x:50010
> 
> 
> I thought maybe whatever configuration value set to 69000 was too low, 
> but there is nothing in hadoop-site or hadoop-default
> using a value of 69000.
> 
> Can anyone shed some light on this?
> 
> thanks,
> M