[jira] [Commented] (HADOOP-12827) WebHdfs socket timeouts should be configurable

["Xiaoyu Yao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-12827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157243#comment-15157243 ] 

Xiaoyu Yao commented on HADOOP-12827:
-------------------------------------

Thanks [~and1000] for reporting the issue and propose the fix. We have seen similar customer issues recently and think about approach similar to the one you proposed here. We proceed with an existing webhdfs retry mechanism introduced by  HDFS-5219/HDFS-5122 to work around it without code changes, which is orthogonal to approach proposed here. 

The patch looks good to me overall. I like the usage of time unit with suffix instead of hard-code time duration to ms or seconds.  Below are some of my suggestions:

1. We need to document the new keys in hdfs-default.xml?
2. Your manual test result looks good to me. Can you add them to unit tests by enhancing TestWebHdfsTimeouts with the new configuration keys?


> WebHdfs socket timeouts should be configurable
> ----------------------------------------------
>
>                 Key: HADOOP-12827
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12827
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>         Environment: all
>            Reporter: Austin Donnelly
>            Assignee: Austin Donnelly
>              Labels: easyfix, newbie
>         Attachments: HADOOP-12827.001.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> WebHdfs client connections use sockets with fixed timeouts of 60 seconds to connect, and 60 seconds for reads.
> This is a problem because I am trying to use WebHdfs to access an archive storage system which can take minutes to hours to return the requested data over WebHdfs.
> The fix is to add new configuration file options to allow these 60s defaults to be customised in hdfs-site.xml.
> If the new configuration options are not present, the behavior is unchanged from before.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)