You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "sysolver (JIRA)" <ji...@apache.org> on 2011/05/10 04:59:03 UTC
[jira] [Commented] (HDFS-148) timeout when writing dfs file causes
infinite loop when closing the file
[ https://issues.apache.org/jira/browse/HDFS-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030995#comment-13030995 ]
sysolver commented on HDFS-148:
-------------------------------
I have met the same problem on JobTracker(CDH3) as well. When the JobTracker try to finish the history log file(stored on HDFS), one of the datanode is dead so that to cause this file "corrupted" (the actual replication number is lower than the minReplication). In such kind of scenario, the file cannot be completed and cause the dfsclient trapped in an infinite-loop, which blocks the JobTracker finally.
Viewing the HDFS-278’s patch, it can help to solve the problem to some extent. However, in such kind of case, the problem is not happening with the client side or the connection between server and client, the root cause is the "corrupted" file. With the current implementation, it is hard to tell the real situation, and even misleading. I wonder if it is possible to throw some IO Exception on server(NameNode) side to tell the file is corrupted.
> timeout when writing dfs file causes infinite loop when closing the file
> ------------------------------------------------------------------------
>
> Key: HDFS-148
> URL: https://issues.apache.org/jira/browse/HDFS-148
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.2
> Reporter: Nigel Daley
> Assignee: Sameer Paranjpye
> Priority: Critical
>
> If, when writing to a dfs file, I get a timeout exception:
> 06/11/29 11:16:05 WARN fs.DFSClient: Error while writing.
> java.net.SocketTimeoutException: timed out waiting for rpc response
> at org.apache.hadoop.ipc.Client.call(Client.java:469)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:164)
> at org.apache.hadoop.dfs.$Proxy0.reportWrittenBlock(Unknown Source)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.internalClose(DFSClient.java:1220)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:1175)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.flush(DFSClient.java:1121)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.write(DFSClient.java:1103)
> at org.apache.hadoop.examples.NNBench2.createWrite(NNBench2.java:107)
> at org.apache.hadoop.examples.NNBench2.main(NNBench2.java:247)
> then the close() operation on the file appears to go into an infinite loop of retrying:
> 06/11/29 13:11:19 INFO fs.DFSClient: Could not complete file, retrying...
> 06/11/29 13:11:20 INFO fs.DFSClient: Could not complete file, retrying...
> 06/11/29 13:11:21 INFO fs.DFSClient: Could not complete file, retrying...
> 06/11/29 13:11:23 INFO fs.DFSClient: Could not complete file, retrying...
> 06/11/29 13:11:24 INFO fs.DFSClient: Could not complete file, retrying...
> ...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira