You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2011/02/25 23:59:21 UTC
[jira] Created: (HDFS-1658) A less expensive way to figure out
directory size
A less expensive way to figure out directory size
-------------------------------------------------
Key: HDFS-1658
URL: https://issues.apache.org/jira/browse/HDFS-1658
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and counts its child size. This is an expensive operation if a directory is huge.
On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-1658:
--------------------------------
Description:
Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and get the number of its children. This is an expensive operation especially when a directory has many children because it may require multiple RPCs.
On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
was:
Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and counts its child size. This is an expensive operation if a directory is huge.
On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and get the number of its children. This is an expensive operation especially when a directory has many children because it may require multiple RPCs.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999626#comment-12999626 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1658:
----------------------------------------------
> Currently in order to figure out a directory size, we have to list a directory ...
We may use {{FileSystem.getContentSummary(Path)}} or "fs -count".
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and counts its child size. This is an expensive operation if a directory is huge.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019687#comment-13019687 ]
Hairong Kuang commented on HDFS-1658:
-------------------------------------
I want to discuss if we could pursue option #1. Right now if a path is a directory, FileStatus.length is actually undefined. It happens that we chose to put 0 there. I think my proposal is just to enhance the semantics, strictly speaking not an incompatible change.
> It is unnecessary to call getFileInfo first
The problem is that most applications work with FileStatus. For example getFileSplits in MapReduce has to get FileStatus for all files by traversing the input directories by calling getFileInfo and listStatus. If we can check a directory is empty by looking at its FileStatus, we can avoid issue a listStatus call to list its children.
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and get the number of its children. This is an expensive operation especially when a directory has many children because it may require multiple RPCs.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999665#comment-12999665 ]
Hairong Kuang commented on HDFS-1658:
-------------------------------------
FileSystem.getContentSummary seems to traverse the whole subtree, does't it?
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and counts its child size. This is an expensive operation if a directory is huge.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999666#comment-12999666 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1658:
----------------------------------------------
How would you define directory size then?
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and counts its child size. This is an expensive operation if a directory is huge.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020030#comment-13020030 ]
Hairong Kuang commented on HDFS-1658:
-------------------------------------
Nicholas and Sanjay, thank for your time on giving a thought on this. I really like Nicholas' proposal. This sounds great to me!
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and get the number of its children. This is an expensive operation especially when a directory has many children because it may require multiple RPCs.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Hairong Kuang (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang reassigned HDFS-1658:
-----------------------------------
Assignee: Weiyan Wang (was: Hairong Kuang)
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Weiyan Wang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and get the number of its children. This is an expensive operation especially when a directory has many children because it may require multiple RPCs.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020021#comment-13020021 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1658:
----------------------------------------------
- Here is an imaginary case unfavorable to option #1: a user program might first put {{FileStatus}}'s, including directories and files, on a list. Then, iterate all the entries and sum up the lengths.
- For {{FileSystem}}, how about add a new method {{FileStatus.getChildrenCount()}} instead of overloading {{FileStatus.getLen()}}?
- For {{FileContext}}, how about remove {{FileStatus.getLen()}} and add {{FileStatus.getBytes()}}, {{FileStatus.getChildrenCount()}}?
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and get the number of its children. This is an expensive operation especially when a directory has many children because it may require multiple RPCs.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999680#comment-12999680 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1658:
----------------------------------------------
I missed that. Thanks for clarifying it.
How about add a new parameter, _depth_, to {{FileSystem.getContentSummary(..)}} so that it will traverse the subtree up to the depth?
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and get the number of its children. This is an expensive operation especially when a directory has many children because it may require multiple RPCs.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019970#comment-13019970 ]
Sanjay Radia commented on HDFS-1658:
------------------------------------
Having the length be number of directory entries makes sense.
However, it is an incompatible (semantic) change and may break some applications. I don't have a good sense of which, if any, applications it may break.
BTW I am planning to add a new method for FileContext that returns FileStatusInfo (see HADOOP-7018).
There I was hoping to make the length be the number of directory entries. Perhaps we could add the same new method to FileSystem.
I will probably have a patch out for review probably next week.
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and get the number of its children. This is an expensive operation especially when a directory has many children because it may require multiple RPCs.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000414#comment-13000414 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1658:
----------------------------------------------
> ... But still an application needs to call getFileInfo first. After it figures out the path is a directory, ...
It is unnecessary to call getFileInfo first since whether a path is a directory can be determined by the file/directory counts returned by {{getContentSummary(..)}}.
We have two solutions:
# Change {{FileSystem.getFileInfo(..)}} to return the number of children in {{FileStatus.length}}.
# Add a new {{FileSystem.getContentSummary(..)}} method with _depth_ as a parameter.
\\
#1 is a semantic change but #2 is not. I am afraid that there may be user codes relying on the fact that {{FileStatus.length == 0}} when the path is a directory.
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and get the number of its children. This is an expensive operation especially when a directory has many children because it may require multiple RPCs.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999667#comment-12999667 ]
Hairong Kuang commented on HDFS-1658:
-------------------------------------
The number of children in a directory. :)
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and counts its child size. This is an expensive operation if a directory is huge.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1658) A less expensive way to figure out
directory size
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000099#comment-13000099 ]
Hairong Kuang commented on HDFS-1658:
-------------------------------------
Adding parameter depth works. But still an application needs to call getFileInfo first. After it figures out the path is a directory, it then has to call getContentSummary to get the directory size.
If we change FileStatus.length to represent directory size, one call getFileInfo is enough.
> A less expensive way to figure out directory size
> -------------------------------------------------
>
> Key: HDFS-1658
> URL: https://issues.apache.org/jira/browse/HDFS-1658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
>
> Currently in order to figure out a directory size, we have to list a directory by calling RPC getListing and get the number of its children. This is an expensive operation especially when a directory has many children because it may require multiple RPCs.
> On the other hand when fetching the status of a path (i.e. calling RPC getFileInfo), the length field of FileStatus is set to be 0 if the path is a directory.
> I am thinking to change this field (FileStatus#length) to be the directory size when the path is a directory. So we can call getFileInfo to get the directory size. This call is much less expensive and simpler than getListing.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira