You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2010/02/03 21:41:27 UTC
[jira] Created: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
NameNode should not return full path name when lisitng a diretory or getting the status of a file
-------------------------------------------------------------------------------------------------
Key: HDFS-946
URL: https://issues.apache.org/jira/browse/HDFS-946
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Hairong Kuang
Fix For: 0.22.0
FSDirectory#getListring(String src) has the following code:
int i = 0;
for (INode cur : contents) {
listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
i++;
}
So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
FSDirectory#getFileInfo(String) does not need to return the file name either.
Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834396#action_12834396 ]
Suresh Srinivas commented on HDFS-946:
--------------------------------------
# TestDFSShell.java not sure why the methods are named starting with caps. Also is the change to this file needed?
# FSDirectory.createFileStatus - consider moving isDirectory check outside. Also current code extends beyond 80 columns.
# HDFSFileStatus
#* consider naming it HdfsFileStatus
#* final static public should public static final
#* since this if for HDFS, comments in the code about different notions in the FS is not required in methods getPermission(), getOwner(), getGroup(),
#* Some of the method parameters and other variables could be declared final
# getFulName() - without unnecessary else code is more readable. Same for getFullPath()
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: HDFSFileStatus.patch
This patch does the following:
1. create a new file status in HDFS, called HDFSFileStatus, for over-the-wire transfer, in which the path contains only the local name of a path, not the full path. Also the path is represented as is a byte array in Java UTF8 encoding just the same as the one stored in each inode.
2. change ClientProtocol getListing to return HDFSFileStatus[] and getFileInfo to return HDFSFileStatus.
a. The path in the return value of getFileInfo is always an empty byte array.
b. If listStatus is called on a file, the path in the only HDFSFileStatus returned also is an empty byte array;
c. If listStatus is called on a directory, the path in each HDFSFileStatus in the returned array contains the local name of the directory entry.
3. FileSystem#getFileStatus and FileSystem#listStatus still see FileStatus which contains the full path name.
4. Unit tests are added to TestFileStatus to verify the change.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: (was: HdfsFileStatus-yahoo20.patch)
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HdfsFileStatus-yahoo20.patch, HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: (was: HDFSFileStatus.patch)
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang reassigned HDFS-946:
----------------------------------
Assignee: Hairong Kuang
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: HDFSFileStatus1.patch
This fixed a bug in the listing of the root.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832745#action_12832745 ]
Hairong Kuang commented on HDFS-946:
------------------------------------
> If you are proposing that the object that is sent over-the-wire is different from FileStatus. If so, please consider the requirement of HDFS-878 too.
This jira tries to reduce the cost of getFileInfo and listing a directory, where HDFS-878 adds cost to these two operations.. So I will not implement HDFS-878 in this jira. Since we are having so many problems with getFileInfo and list a directory, we should be very cautious about adding anything to FileStatus in hdfs unless it is absolutely necessary.
I have conducted some experiments with my patch. I write an application that spawns 100 threads, each of which lists a directory of size 1300 for 200 times. I use yourKit to profile the NameNode while the application is running. Without the patch, NameNode's CPU utilization is 20~26% and time spent on GC is 3~5%. With the patch, NameNode's CPU utilization drops to 12~17% and the time spent on GS is mostly 0% but occasionally becomes 1 or 2%.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: (was: HdfsFileStatus-yahoo20.patch)
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: HdfsFileStatusProxy-Yahoo20.patch
This patch additionally has the change to hdfsproxy in Yahoo!'s security branch.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HdfsFileStatus-yahoo20.patch, HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch, HdfsFileStatusProxy-Yahoo20.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Hadoop Flags: [Incompatible change]
Status: Patch Available (was: Open)
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: HdfsFileStatus-yahoo20.patch
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HdfsFileStatus-yahoo20.patch, HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829250#action_12829250 ]
Doug Cutting commented on HDFS-946:
-----------------------------------
This sounds reasonable. But the client would still return fully-qualified paths, no?
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Fix For: 0.22.0
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853659#action_12853659 ]
Hudson commented on HDFS-946:
-----------------------------
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/])
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HdfsFileStatus-yahoo20.patch, HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch, HdfsFileStatusProxy-Yahoo20.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: HdfsFileStatus4.patch
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: HdfsFileStatus-yahoo20.patch
Here is the patch that will be committed to Yahoo! Hadoop 20.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HdfsFileStatus-yahoo20.patch, HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829564#action_12829564 ]
dhruba borthakur commented on HDFS-946:
---------------------------------------
Client's should continue to get the full path name in a FileStatus object, isn't it? Otherwise many many existing client applications will break.
If you are proposing that the object that is sent over-the-wire is different from FileStatus. If so, please consider the requirement of HDFS-878 too.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Fix For: 0.22.0
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Status: Open (was: Patch Available)
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836984#action_12836984 ]
Hudson commented on HDFS-946:
-----------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #197 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/197/])
. NameNode should not return full path name when lisitng a diretory or getting the status of a file. Contributed by Hairong Kuang.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: HdfsFileStatus-yahoo20.patch
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HdfsFileStatus-yahoo20.patch, HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829269#action_12829269 ]
Hairong Kuang commented on HDFS-946:
------------------------------------
For this jira, the client will still return fully-qualified paths.
But I am thinking that even at the FileContext level it is not necessary to return fully-qualified paths. However this is a user-facing incompatible change. I would prefer to discuss it in a different jira.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Fix For: 0.22.0
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: HDFSFileStatus.patch
This patch does the following:
1. create a new file status in HDFS, called HDFSFileStatus, for over-the-wire transfer, in which the path contains only the local name of a path, not the full path. Also the path is represented as is a byte array in Java UTF8 encoding just the same as the one stored in each inode.
2. change ClientProtocol getListing to return HDFSFileStatus[] and getFileInfo to return HDFSFileStatus.
a. The path in the return value of getFileInfo is always an empty byte array.
b. If listStatus is called on a file, the path in the only HDFSFileStatus returned also is an empty byte array;
c. If listStatus is called on a directory, the path in each HDFSFileStatus in the returned array contains the local name of the directory entry.
3. FileSystem#getFileStatus and FileSystem#listStatus still see FileStatus which contains the full path name.
4. Unit tests are added to TestFileStatus to verify the change.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834719#action_12834719 ]
Hadoop QA commented on HDFS-946:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12436085/HdfsFileStatus3.patch
against trunk revision 910760.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/234/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/234/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/234/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/234/console
This message is automatically generated.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: HdfsFileStatus3.patch
This patch incorporates all review comments except for "2. consider move isDirectory outside". In addition, it
1. removes the method getPath(), which returns the byte array, from HdfsFileStatus. This makes the byte array immutable, which makes the change safer.
2. fixes a few more bugs in Jsp code which misuses getPath().
3. adds comments to INode#name that reminds that if this encoding is changed, the ClientProtocol is changed and the decoding of HdfsFileStatus#name should change too.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: (was: HdfsFileStatus4.patch)
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853958#action_12853958 ]
Hudson commented on HDFS-946:
-----------------------------
Integrated in Hadoop-Hdfs-trunk #275 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/])
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HdfsFileStatus-yahoo20.patch, HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch, HdfsFileStatusProxy-Yahoo20.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835876#action_12835876 ]
Suresh Srinivas commented on HDFS-946:
--------------------------------------
+1 the patch looks good.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853868#action_12853868 ]
Hudson commented on HDFS-946:
-----------------------------
Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/])
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HdfsFileStatus-yahoo20.patch, HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch, HdfsFileStatusProxy-Yahoo20.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Hadoop Flags: [Incompatible change, Reviewed] (was: [Incompatible change])
Status: Patch Available (was: Open)
Looks that contribute tests did not get to run. Resubmitting the patch...
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Attachment: HdfsFileStatus4.patch
This patch synced with the trunk and fixed a bug in ListPathsServlet.
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-946) NameNode should not return full path
name when lisitng a diretory or getting the status of a file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-946:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Hudson seems to have problem running the test. I ran "ant test-core" locally and tests were passed.
I've committed this!
> NameNode should not return full path name when lisitng a diretory or getting the status of a file
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-946
> URL: https://issues.apache.org/jira/browse/HDFS-946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: HdfsFileStatus-yahoo20.patch, HDFSFileStatus.patch, HDFSFileStatus1.patch, HdfsFileStatus3.patch, HdfsFileStatus4.patch
>
>
> FSDirectory#getListring(String src) has the following code:
> int i = 0;
> for (INode cur : contents) {
> listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
> i++;
> }
> So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.
> FSDirectory#getFileInfo(String) does not need to return the file name either.
> Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.