You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2008/02/04 19:09:08 UTC
[jira] Created: (HADOOP-2777) DFSClient more robust if the namenode
is busy doing GC
DFSClient more robust if the namenode is busy doing GC
------------------------------------------------------
Key: HADOOP-2777
URL: https://issues.apache.org/jira/browse/HADOOP-2777
Project: Hadoop Core
Issue Type: Improvement
Components: dfs
Reporter: dhruba borthakur
In the current code, if the client (writer) encounters an RPC error while fetching a new block id from the namenode, it does not retry. It throws an exception to the application. This becomes especially bad if the namenode is in the middle of a GC and does not respond in time. The reason the client throws an exception is because it does not know whether the namenode successfully allocated a block for this file.
One possible enhancement would be to make the client retry the addBlock RPC if needed. The client can send the block list that it currently has. The namenode can match the block list send by the client with what it has in its own metadata and then send back a new blockid (or a previously allocated blockid that the client had not yet received because the earlier RPC timedout). This will make the client more robust!
This works even when we support Appends because the namenode will *always* verify that the client has the lease for the file in question.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2777) DFSClient more robust if the
namenode is busy doing GC
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12565462#action_12565462 ]
Raghu Angadi commented on HADOOP-2777:
--------------------------------------
Are two simultaneous writers possible with appends?
> The client can send the block list that it currently has.
Client needs to send only the last block it knows, I think.
Before this fix goes in, should we increase client.ipc.timeout to something like 10-15 minutes? It does not have any perf penalty and current value of 1 min is too short (since we are actually going to remove this timeout, HADOOP-2188).
> DFSClient more robust if the namenode is busy doing GC
> ------------------------------------------------------
>
> Key: HADOOP-2777
> URL: https://issues.apache.org/jira/browse/HADOOP-2777
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
>
> In the current code, if the client (writer) encounters an RPC error while fetching a new block id from the namenode, it does not retry. It throws an exception to the application. This becomes especially bad if the namenode is in the middle of a GC and does not respond in time. The reason the client throws an exception is because it does not know whether the namenode successfully allocated a block for this file.
> One possible enhancement would be to make the client retry the addBlock RPC if needed. The client can send the block list that it currently has. The namenode can match the block list send by the client with what it has in its own metadata and then send back a new blockid (or a previously allocated blockid that the client had not yet received because the earlier RPC timedout). This will make the client more robust!
> This works even when we support Appends because the namenode will *always* verify that the client has the lease for the file in question.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.