You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "David J. O'Dell" <do...@videoegg.com> on 2009/11/25 20:27:41 UTC

time outs when accessing port 50010

I have 2 clusters:
30 nodes running 0.18.3
and
36 nodes running 0.20.1

I've intermittently seen the following errors on both of my clusters, it 
happens when writing files.
I was hoping this would go away with the new version but I see the same 
behavior on both versions.
The namenode logs don't show any problems, its always on the client and 
datanodes.

Below is any example from this morning, unfortunately I haven't found a 
bug or config that specifically addresses this issue.

Any insight would be greatly appreciated.

Client log:
09/11/25 10:54:15 INFO hdfs.DFSClient: Exception in 
createBlockOutputStream java.net.SocketTimeoutException: 69000 millis 
timeout while waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/10.1.75.11:37852 
remote=/10.1.75.125:50010]
09/11/25 10:54:15 INFO hdfs.DFSClient: Abandoning block 
blk_-105422935413230449_22608
09/11/25 10:54:15 INFO hdfs.DFSClient: Waiting to find target node: 
10.1.75.125:50010

Datanode log:
2009-11-25 10:54:51,170 ERROR 
org.apache.hadoop.hdfs.server.datanode.DataNode: 
DatanodeRegistration(10.1.75.125:50010, 
storageID=DS-1401408597-10.1.75.125-50010-1258737830230, infoPort=50075, 
ipcPort=50020):DataXceiver
java.net.SocketTimeoutException: 120000 millis timeout while waiting for 
channel to be ready for connect. ch : 
java.nio.channels.SocketChannel[connection-pending 
remote=/10.1.75.104:50010]
        at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
        at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
        at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
        at java.lang.Thread.run(Thread.java:619)


Re: time outs when accessing port 50010

Posted by Raghu Angadi <an...@gmail.com>.
On Mon, Dec 21, 2009 at 11:57 AM, dave bayer <da...@cloudfactory.org> wrote:

>
> On Nov 25, 2009, at 11:27 AM, David J. O'Dell wrote:
>
>  I've intermittently seen the following errors on both of my clusters, it
>> happens when writing files.
>> I was hoping this would go away with the new version but I see the same
>> behavior on both versions.
>> The namenode logs don't show any problems, its always on the client and
>> datanodes.
>>
>
> [leaving errors below for reference]
>
> I've seen similar errors on my 0.19.2 cluster when the cluster is decently
> busy. I've traced this more or less to the host in question doing
> verification on its blocks, an operation which seems to take the datanode
> out for upwards of 500 seconds in some cases.
>
>
This issue is fixed in 0.21 (
https://issues.apache.org/jira/browse/HADOOP-4584 )

But the original issue reported is _not_ related to Datanode locking. For
the user, connect() it self times out, which has nothing to do with
FSDataset locking during block report processing.

Raghu.

In 0.19.2, if you look at o.a.h.hdfs.server.datanode.FSDataset.FSVolumeSet,
> you will see that all methods are synchronized. All operations for the
> dataset on the node seem to drop through methods in this class which in turn
> causes a backup when one thread spends a large amount of time locking the
> monitor...
>
> You can grab a few jstacks and use a dump analyzer (like
> https://tda.dev.java.net/) to poke through them to see if you have the
> same behavior.
>
> I have not spent enough time digging into this to understand whether the
> whole dataset really needs to be locked during the operation or if the locks
> could be moved closer to the FSDir operations.
>
> dave bayer
>
> original logs clips included here:
>
>
>> Client log:
>> 09/11/25 10:54:15 INFO hdfs.DFSClient: Exception in
>> createBlockOutputStream java.net.SocketTimeoutException: 69000 millis
>> timeout while waiting for channel to be ready for read. ch :
>> java.nio.channels.SocketChannel[connected local=/10.1.75.11:37852remote=/
>> 10.1.75.125:50010]
>> 09/11/25 10:54:15 INFO hdfs.DFSClient: Abandoning block
>> blk_-105422935413230449_22608
>> 09/11/25 10:54:15 INFO hdfs.DFSClient: Waiting to find target node:
>> 10.1.75.125:50010
>>
>> Datanode log:
>> 2009-11-25 10:54:51,170 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>> 10.1.75.125:50010,
>> storageID=DS-1401408597-10.1.75.125-50010-1258737830230, infoPort=50075,
>> ipcPort=50020):DataXceiver
>> java.net.SocketTimeoutException: 120000 millis timeout while waiting for
>> channel to be ready for connect. ch :
>> java.nio.channels.SocketChannel[connection-pending remote=/
>> 10.1.75.104:50010]
>>      at
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
>>      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>>      at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
>>      at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>>      at java.lang.Thread.run(Thread.java:619)
>>
>
>

Re: time outs when accessing port 50010

Posted by dave bayer <da...@cloudfactory.org>.
On Nov 25, 2009, at 11:27 AM, David J. O'Dell wrote:

> I've intermittently seen the following errors on both of my  
> clusters, it happens when writing files.
> I was hoping this would go away with the new version but I see the  
> same behavior on both versions.
> The namenode logs don't show any problems, its always on the client  
> and datanodes.

[leaving errors below for reference]

I've seen similar errors on my 0.19.2 cluster when the cluster is  
decently busy. I've traced this more or less to the host in question  
doing verification on its blocks, an operation which seems to take the  
datanode out for upwards of 500 seconds in some cases.

In 0.19.2, if you look at  
o.a.h.hdfs.server.datanode.FSDataset.FSVolumeSet, you will see that  
all methods are synchronized. All operations for the dataset on the  
node seem to drop through methods in this class which in turn causes a  
backup when one thread spends a large amount of time locking the  
monitor...

You can grab a few jstacks and use a dump analyzer (like https://tda.dev.java.net/) 
  to poke through them to see if you have the same behavior.

I have not spent enough time digging into this to understand whether  
the whole dataset really needs to be locked during the operation or if  
the locks could be moved closer to the FSDir operations.

dave bayer

original logs clips included here:
>
> Client log:
> 09/11/25 10:54:15 INFO hdfs.DFSClient: Exception in  
> createBlockOutputStream java.net.SocketTimeoutException: 69000  
> millis timeout while waiting for channel to be ready for read. ch :  
> java.nio.channels.SocketChannel[connected local=/10.1.75.11:37852  
> remote=/10.1.75.125:50010]
> 09/11/25 10:54:15 INFO hdfs.DFSClient: Abandoning block  
> blk_-105422935413230449_22608
> 09/11/25 10:54:15 INFO hdfs.DFSClient: Waiting to find target node:  
> 10.1.75.125:50010
>
> Datanode log:
> 2009-11-25 10:54:51,170 ERROR  
> org.apache.hadoop.hdfs.server.datanode.DataNode:  
> DatanodeRegistration(10.1.75.125:50010,  
> storageID=DS-1401408597-10.1.75.125-50010-1258737830230,  
> infoPort=50075, ipcPort=50020):DataXceiver
> java.net.SocketTimeoutException: 120000 millis timeout while waiting  
> for channel to be ready for connect. ch :  
> java.nio.channels.SocketChannel[connection-pending remote=/ 
> 10.1.75.104:50010]
>       at  
> org 
> .apache 
> .hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
>       at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>       at  
> org 
> .apache 
> .hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java: 
> 282)
>       at  
> org 
> .apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: 
> 103)
>       at java.lang.Thread.run(Thread.java:619)


Re: time outs when accessing port 50010

Posted by javateck javateck <ja...@gmail.com>.
right, sorry, my typo

On Mon, Dec 14, 2009 at 10:43 AM, Allen Wittenauer <awittenauer@linkedin.com
> wrote:

> On 12/14/09 9:56 AM, "javateck javateck" <ja...@gmail.com> wrote:
> >   I added following properties in my hadoop-sites.xml:
>
> Did you mean hadoop-site.xml?
>
>
>

Re: time outs when accessing port 50010

Posted by javateck javateck <ja...@gmail.com>.
finally we took out as the file system, I don't feel it's stable enough as
production environment, production is not experiment.

On Mon, Dec 14, 2009 at 9:56 AM, javateck javateck <ja...@gmail.com>wrote:

> Hi Brian,
>
>   Agree, and thanks for the answering.
>
>   Actually in my situation, I also tried telnet to the datanode on 50010 at
> the time it was threw timeout exception, and the telnet was working. And I
> just tried to set the timeout higher and see what's going to happen. And I
> see a lot of time_wait if I do a "netstat -an".
>
>   I added following properties in my hadoop-sites.xml:
>
> <property>
>  <name>dfs.socket.timeout</name>
>  <value>180000</value>
>  <description>dfs socket timeout</description>
> </property>
>
> <property>
>  <name>dfs.datanode.socket.write.timeout</name>
>  <value>3600000</value>
>  <description>datanode write timeout</description>
> </property>
>
>
> On Mon, Dec 14, 2009 at 7:29 AM, Brian Bockelman <bb...@cse.unl.edu>wrote:
>
>>
>> On Dec 14, 2009, at 5:21 AM, javateck javateck wrote:
>>
>> > I have exactly the same issue there.
>> >
>> > Sometimes really feel helpless, maybe very few people use hadoop as a FS
>> at
>> > all. I think this is also why people stop using it, there are so many
>> issues
>> > there, and so few people can help or have the experience.
>>
>> <soapbox>
>> These are the joys of working on a young software project, right?  I would
>> point out that many folks answer many questions every day on the mailing
>> lists.  If you want every question solved every time, you have the option of
>> buying (excellent) support.
>>
>> As far as distributed file systems go, I've got a lot of experience
>> running ones that have more issues and are used by even less folks.  It's
>> not pleasant.  If you just need a 30-40TB filesystem (i.e., not a data
>> processing system) I'd agree that you can probably find more mature systems.
>>  If you use HDFS as a file system only and don't see clear benefits over
>> Lustre, then perhaps you should be using Lustre in the first place.
>> </soapbox>
>>
>> With regards to the error below, I'd guess that it is caused by a
>> networking partition - i.e., it appears that the client couldn't open a
>> socket connection to 10.1.75.125 from 10.1.75.11.  I'd check for routing
>> issues on both nodes.  Does the error happen intermittently for any two
>> nodes or if you look through the past incidents, does it always involve the
>> same node?
>>
>> Brian
>>
>> >
>> >
>> > On Wed, Nov 25, 2009 at 11:27 AM, David J. O'Dell <dodell@videoegg.com
>> >wrote:
>> >
>> >> I have 2 clusters:
>> >> 30 nodes running 0.18.3
>> >> and
>> >> 36 nodes running 0.20.1
>> >>
>> >> I've intermittently seen the following errors on both of my clusters,
>> it
>> >> happens when writing files.
>> >> I was hoping this would go away with the new version but I see the same
>> >> behavior on both versions.
>> >> The namenode logs don't show any problems, its always on the client and
>> >> datanodes.
>> >>
>> >> Below is any example from this morning, unfortunately I haven't found a
>> bug
>> >> or config that specifically addresses this issue.
>> >>
>> >> Any insight would be greatly appreciated.
>> >>
>> >> Client log:
>> >> 09/11/25 10:54:15 INFO hdfs.DFSClient: Exception in
>> createBlockOutputStream
>> >> java.net.SocketTimeoutException: 69000 millis timeout while waiting for
>> >> channel to be ready for read. ch :
>> java.nio.channels.SocketChannel[connected
>> >> local=/10.1.75.11:37852 remote=/10.1.75.125:50010]
>> >> 09/11/25 10:54:15 INFO hdfs.DFSClient: Abandoning block
>> >> blk_-105422935413230449_22608
>> >> 09/11/25 10:54:15 INFO hdfs.DFSClient: Waiting to find target node:
>> >> 10.1.75.125:50010
>> >>
>> >> Datanode log:
>> >> 2009-11-25 10:54:51,170 ERROR
>> >> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>> >> 10.1.75.125:50010,
>> >> storageID=DS-1401408597-10.1.75.125-50010-1258737830230,
>> infoPort=50075,
>> >> ipcPort=50020):DataXceiver
>> >> java.net.SocketTimeoutException: 120000 millis timeout while waiting
>> for
>> >> channel to be ready for connect. ch :
>> >> java.nio.channels.SocketChannel[connection-pending remote=/
>> >> 10.1.75.104:50010]
>> >>      at
>> >>
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
>> >>      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>> >>      at
>> >>
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
>> >>      at
>> >>
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>> >>      at java.lang.Thread.run(Thread.java:619)
>> >>
>> >>
>>
>>
>

Re: time outs when accessing port 50010

Posted by Allen Wittenauer <aw...@linkedin.com>.
On 12/14/09 9:56 AM, "javateck javateck" <ja...@gmail.com> wrote:
>   I added following properties in my hadoop-sites.xml:

Did you mean hadoop-site.xml?



Re: time outs when accessing port 50010

Posted by javateck javateck <ja...@gmail.com>.
Hi Brian,

  Agree, and thanks for the answering.

  Actually in my situation, I also tried telnet to the datanode on 50010 at
the time it was threw timeout exception, and the telnet was working. And I
just tried to set the timeout higher and see what's going to happen. And I
see a lot of time_wait if I do a "netstat -an".

  I added following properties in my hadoop-sites.xml:

<property>
 <name>dfs.socket.timeout</name>
 <value>180000</value>
 <description>dfs socket timeout</description>
</property>

<property>
 <name>dfs.datanode.socket.write.timeout</name>
 <value>3600000</value>
 <description>datanode write timeout</description>
</property>


On Mon, Dec 14, 2009 at 7:29 AM, Brian Bockelman <bb...@cse.unl.edu>wrote:

>
> On Dec 14, 2009, at 5:21 AM, javateck javateck wrote:
>
> > I have exactly the same issue there.
> >
> > Sometimes really feel helpless, maybe very few people use hadoop as a FS
> at
> > all. I think this is also why people stop using it, there are so many
> issues
> > there, and so few people can help or have the experience.
>
> <soapbox>
> These are the joys of working on a young software project, right?  I would
> point out that many folks answer many questions every day on the mailing
> lists.  If you want every question solved every time, you have the option of
> buying (excellent) support.
>
> As far as distributed file systems go, I've got a lot of experience running
> ones that have more issues and are used by even less folks.  It's not
> pleasant.  If you just need a 30-40TB filesystem (i.e., not a data
> processing system) I'd agree that you can probably find more mature systems.
>  If you use HDFS as a file system only and don't see clear benefits over
> Lustre, then perhaps you should be using Lustre in the first place.
> </soapbox>
>
> With regards to the error below, I'd guess that it is caused by a
> networking partition - i.e., it appears that the client couldn't open a
> socket connection to 10.1.75.125 from 10.1.75.11.  I'd check for routing
> issues on both nodes.  Does the error happen intermittently for any two
> nodes or if you look through the past incidents, does it always involve the
> same node?
>
> Brian
>
> >
> >
> > On Wed, Nov 25, 2009 at 11:27 AM, David J. O'Dell <dodell@videoegg.com
> >wrote:
> >
> >> I have 2 clusters:
> >> 30 nodes running 0.18.3
> >> and
> >> 36 nodes running 0.20.1
> >>
> >> I've intermittently seen the following errors on both of my clusters, it
> >> happens when writing files.
> >> I was hoping this would go away with the new version but I see the same
> >> behavior on both versions.
> >> The namenode logs don't show any problems, its always on the client and
> >> datanodes.
> >>
> >> Below is any example from this morning, unfortunately I haven't found a
> bug
> >> or config that specifically addresses this issue.
> >>
> >> Any insight would be greatly appreciated.
> >>
> >> Client log:
> >> 09/11/25 10:54:15 INFO hdfs.DFSClient: Exception in
> createBlockOutputStream
> >> java.net.SocketTimeoutException: 69000 millis timeout while waiting for
> >> channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected
> >> local=/10.1.75.11:37852 remote=/10.1.75.125:50010]
> >> 09/11/25 10:54:15 INFO hdfs.DFSClient: Abandoning block
> >> blk_-105422935413230449_22608
> >> 09/11/25 10:54:15 INFO hdfs.DFSClient: Waiting to find target node:
> >> 10.1.75.125:50010
> >>
> >> Datanode log:
> >> 2009-11-25 10:54:51,170 ERROR
> >> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> >> 10.1.75.125:50010,
> >> storageID=DS-1401408597-10.1.75.125-50010-1258737830230, infoPort=50075,
> >> ipcPort=50020):DataXceiver
> >> java.net.SocketTimeoutException: 120000 millis timeout while waiting for
> >> channel to be ready for connect. ch :
> >> java.nio.channels.SocketChannel[connection-pending remote=/
> >> 10.1.75.104:50010]
> >>      at
> >>
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
> >>      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> >>      at
> >>
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
> >>      at
> >>
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
> >>      at java.lang.Thread.run(Thread.java:619)
> >>
> >>
>
>

Re: time outs when accessing port 50010

Posted by Brian Bockelman <bb...@cse.unl.edu>.
On Dec 14, 2009, at 5:21 AM, javateck javateck wrote:

> I have exactly the same issue there.
> 
> Sometimes really feel helpless, maybe very few people use hadoop as a FS at
> all. I think this is also why people stop using it, there are so many issues
> there, and so few people can help or have the experience.

<soapbox>
These are the joys of working on a young software project, right?  I would point out that many folks answer many questions every day on the mailing lists.  If you want every question solved every time, you have the option of buying (excellent) support.

As far as distributed file systems go, I've got a lot of experience running ones that have more issues and are used by even less folks.  It's not pleasant.  If you just need a 30-40TB filesystem (i.e., not a data processing system) I'd agree that you can probably find more mature systems.  If you use HDFS as a file system only and don't see clear benefits over Lustre, then perhaps you should be using Lustre in the first place.
</soapbox>

With regards to the error below, I'd guess that it is caused by a networking partition - i.e., it appears that the client couldn't open a socket connection to 10.1.75.125 from 10.1.75.11.  I'd check for routing issues on both nodes.  Does the error happen intermittently for any two nodes or if you look through the past incidents, does it always involve the same node?

Brian

> 
> 
> On Wed, Nov 25, 2009 at 11:27 AM, David J. O'Dell <do...@videoegg.com>wrote:
> 
>> I have 2 clusters:
>> 30 nodes running 0.18.3
>> and
>> 36 nodes running 0.20.1
>> 
>> I've intermittently seen the following errors on both of my clusters, it
>> happens when writing files.
>> I was hoping this would go away with the new version but I see the same
>> behavior on both versions.
>> The namenode logs don't show any problems, its always on the client and
>> datanodes.
>> 
>> Below is any example from this morning, unfortunately I haven't found a bug
>> or config that specifically addresses this issue.
>> 
>> Any insight would be greatly appreciated.
>> 
>> Client log:
>> 09/11/25 10:54:15 INFO hdfs.DFSClient: Exception in createBlockOutputStream
>> java.net.SocketTimeoutException: 69000 millis timeout while waiting for
>> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
>> local=/10.1.75.11:37852 remote=/10.1.75.125:50010]
>> 09/11/25 10:54:15 INFO hdfs.DFSClient: Abandoning block
>> blk_-105422935413230449_22608
>> 09/11/25 10:54:15 INFO hdfs.DFSClient: Waiting to find target node:
>> 10.1.75.125:50010
>> 
>> Datanode log:
>> 2009-11-25 10:54:51,170 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>> 10.1.75.125:50010,
>> storageID=DS-1401408597-10.1.75.125-50010-1258737830230, infoPort=50075,
>> ipcPort=50020):DataXceiver
>> java.net.SocketTimeoutException: 120000 millis timeout while waiting for
>> channel to be ready for connect. ch :
>> java.nio.channels.SocketChannel[connection-pending remote=/
>> 10.1.75.104:50010]
>>      at
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
>>      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>>      at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
>>      at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>>      at java.lang.Thread.run(Thread.java:619)
>> 
>> 


Re: time outs when accessing port 50010

Posted by javateck javateck <ja...@gmail.com>.
I have exactly the same issue there.

Sometimes really feel helpless, maybe very few people use hadoop as a FS at
all. I think this is also why people stop using it, there are so many issues
there, and so few people can help or have the experience.


On Wed, Nov 25, 2009 at 11:27 AM, David J. O'Dell <do...@videoegg.com>wrote:

> I have 2 clusters:
> 30 nodes running 0.18.3
> and
> 36 nodes running 0.20.1
>
> I've intermittently seen the following errors on both of my clusters, it
> happens when writing files.
> I was hoping this would go away with the new version but I see the same
> behavior on both versions.
> The namenode logs don't show any problems, its always on the client and
> datanodes.
>
> Below is any example from this morning, unfortunately I haven't found a bug
> or config that specifically addresses this issue.
>
> Any insight would be greatly appreciated.
>
> Client log:
> 09/11/25 10:54:15 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> java.net.SocketTimeoutException: 69000 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/10.1.75.11:37852 remote=/10.1.75.125:50010]
> 09/11/25 10:54:15 INFO hdfs.DFSClient: Abandoning block
> blk_-105422935413230449_22608
> 09/11/25 10:54:15 INFO hdfs.DFSClient: Waiting to find target node:
> 10.1.75.125:50010
>
> Datanode log:
> 2009-11-25 10:54:51,170 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.1.75.125:50010,
> storageID=DS-1401408597-10.1.75.125-50010-1258737830230, infoPort=50075,
> ipcPort=50020):DataXceiver
> java.net.SocketTimeoutException: 120000 millis timeout while waiting for
> channel to be ready for connect. ch :
> java.nio.channels.SocketChannel[connection-pending remote=/
> 10.1.75.104:50010]
>       at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
>       at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>       at java.lang.Thread.run(Thread.java:619)
>
>

Re: time outs when accessing port 50010

Posted by Raghu Angadi <an...@gmail.com>.
You could check whats going on on second datanode (10.1.75.104). The connect
to it from the first datanode took longer than 2 min.

Btw, it looks to be a bug that there is a mismatch in timeout used by
datanode and the client. Ideally client's timeout should be larger than the
timeout used by the datanodes so that the error is detected correctly. Also,
connect timeout should be even smaller.

If you want to try out a work around, change 'timeoutValue' at
DataXceiver.java:282 to some thing like 'timeoutValue/numTagets/2'.

Raghu.

On Wed, Nov 25, 2009 at 11:27 AM, David J. O'Dell <do...@videoegg.com>wrote:

> I have 2 clusters:
> 30 nodes running 0.18.3
> and
> 36 nodes running 0.20.1
>
> I've intermittently seen the following errors on both of my clusters, it
> happens when writing files.
> I was hoping this would go away with the new version but I see the same
> behavior on both versions.
> The namenode logs don't show any problems, its always on the client and
> datanodes.
>
> Below is any example from this morning, unfortunately I haven't found a bug
> or config that specifically addresses this issue.
>
> Any insight would be greatly appreciated.
>
> Client log:
> 09/11/25 10:54:15 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> java.net.SocketTimeoutException: 69000 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/10.1.75.11:37852 remote=/10.1.75.125:50010]
> 09/11/25 10:54:15 INFO hdfs.DFSClient: Abandoning block
> blk_-105422935413230449_22608
> 09/11/25 10:54:15 INFO hdfs.DFSClient: Waiting to find target node:
> 10.1.75.125:50010
>
> Datanode log:
> 2009-11-25 10:54:51,170 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.1.75.125:50010,
> storageID=DS-1401408597-10.1.75.125-50010-1258737830230, infoPort=50075,
> ipcPort=50020):DataXceiver
> java.net.SocketTimeoutException: 120000 millis timeout while waiting for
> channel to be ready for connect. ch :
> java.nio.channels.SocketChannel[connection-pending remote=/
> 10.1.75.104:50010]
>       at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
>       at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
>       at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>       at java.lang.Thread.run(Thread.java:619)
>
>