You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-issues@hadoop.apache.org by "Allen Wittenauer (JIRA)" <ji...@apache.org> on 2013/08/19 22:39:47 UTC

[jira] [Commented] (HADOOP-9884) Hadoop calling du -sk is expensive

    [ https://issues.apache.org/jira/browse/HADOOP-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744184#comment-13744184 ] 

Allen Wittenauer commented on HADOOP-9884:
------------------------------------------

We need to tread carefully here.  Replacing the du call has the potential to break distributed cache (and probably other things), especially for non-HDFS-based systems.
                
> Hadoop calling du -sk is expensive
> ----------------------------------
>
>                 Key: HADOOP-9884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9884
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Alex Newman
>
> On numerous occasions we've had customers worry about slowness while hadoop calls du -sk underneath the hood. For most of these users getting the information from df would be sufficient and much faster. In fact there is a hack going around, that is quiet common that replaces df with du. Sometimes people have to tune the vcache. What if we just allowed users to use the df information instead of the du information with a patch and config setting. I'd be glad to code it up

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira