You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Colin Patrick McCabe (JIRA)" <ji...@apache.org> on 2014/07/25 20:17:39 UTC
[jira] [Commented] (HDFS-6755) Make DFSOutputStream more efficient
[ https://issues.apache.org/jira/browse/HDFS-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074717#comment-14074717 ]
Colin Patrick McCabe commented on HDFS-6755:
--------------------------------------------
The code here is using "exponential backoff" to wait for the {{NameNode}} to be available. Getting rid of the sleep won't make anything more efficient... it will just increase the number of cases where a temporary network issue between a client and a NameNode causes a file close to fail.
> Make DFSOutputStream more efficient
> -----------------------------------
>
> Key: HDFS-6755
> URL: https://issues.apache.org/jira/browse/HDFS-6755
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.6.0
> Reporter: Mit Desai
> Assignee: Mit Desai
>
> Following code in DFSOutputStream may have an unnecessary sleep.
> {code}
> try {
> Thread.sleep(localTimeout);
> if (retries == 0) {
> throw new IOException("Unable to close file because the last block"
> + " does not have enough number of replicas.");
> }
> retries--;
> localTimeout *= 2;
> if (Time.now() - localstart > 5000) {
> DFSClient.LOG.info("Could not complete " + src + " retrying...");
> }
> } catch (InterruptedException ie) {
> DFSClient.LOG.warn("Caught exception ", ie);
> }
> {code}
> Currently, the code sleeps before throwing an exception which should not be the case.
> The sleep time gets doubled on every iteration, which can make a significant effect if there are more than one iterations and it would sleep just to throw an exception. We need to move the sleep down after decrementing retries.
--
This message was sent by Atlassian JIRA
(v6.2#6252)