You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Brian Bockelman (JIRA)" <ji...@apache.org> on 2008/10/08 01:24:44 UTC
[jira] Created: (HADOOP-4368) Superuser privileges required to do
"df"
Superuser privileges required to do "df"
----------------------------------------
Key: HADOOP-4368
URL: https://issues.apache.org/jira/browse/HADOOP-4368
Project: Hadoop Core
Issue Type: Wish
Components: contrib/fuse-dfs, dfs
Affects Versions: 0.18.1
Reporter: Brian Bockelman
Priority: Minor
super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654457#action_12654457 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
------------------------------------------------
> I want to connect to Hadoop as superuser/supergroup, ...
This sounds impossible or should not be allowed. Otherwise, it is very easy to hack the system.
Currently, the superusers are either the user starting the namenode or the users who belong to the group specified by the conf property dfs.permissions.supergroup (the default is "supergroup"). There is no way for clients to get the superuser accounts in runtime.
See also http://hadoop.apache.org/core/docs/r0.19.0/hdfs_permissions_guide.html#The+Super-User
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653517#action_12653517 ]
Pete Wyckoff commented on HADOOP-4368:
--------------------------------------
I will look at those, but what about authorization and future change plans? integration with other systems - nis, ldap, ...
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Attachment: hadoop4368.fsstatus.v3.patch
Updated patch attached, addressing Nicholas' and Doug's comments.
@Nicholas: I have added a -df command to FsShell, which has similar output to GNUs core-utils df command. I left the DFSAdmin output, as I thought this was (a) useful, and (b) gives more interpretations on the reported figures for the HDFS scenario than FsShell#df() should.
NB: The java.io.File getTotalSpace() etc API works find for normally mounted local and NFS partitions on linux, but for partitions that are mounted by autofs, I found it always gave 0s. This seems to be a problem with these API methods that are new in Java 6. I couldn't find reference to such a problem anywhere on the web.
Two questions:
1. Should I provide ant-patch output myself?
2. Should I fix libhdfs in this issue as well?
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638929#action_12638929 ]
Pete Wyckoff commented on HADOOP-4368:
--------------------------------------
hi brian,
do you know why they (who?) added this check?
pete
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639042#action_12639042 ]
Brian Bockelman commented on HADOOP-4368:
-----------------------------------------
No clue - I suspect that it was checked in when permissions were added to the FSNamesystem.
I emailed the core-user list, but no dice.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Attachment: hadoop4368.fsstatus.v2.patch
Address Doug's comments. Also made FsStatus Writable, because FileStatus is also writable.
For testing, I tried asserting that capacity = free + used, but this doesn't seem to be the case for HDFS, and could feasibly be not true on other filesystems implementation, so I omitted this from the test.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Bockelman updated HADOOP-4368:
------------------------------------
Attachment: fuse_statfs_trunk.patch
Patch against trunk, post-Pete's reorganization of the fuse-dfs source.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668191#action_12668191 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
------------------------------------------------
>It appears that getCapacityTotal(), getCapacityUsed(), getCapacityRemaining are all exported in the FSNamesystemMBean interface.
How about we use getStats() in FSNamesystemMBean?
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641977#action_12641977 ]
Konstantin Shvachko commented on HADOOP-4368:
---------------------------------------------
shv> opening getStats for all users means moving it to the FSShell command group.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Release Note: Adds API support for unprivileged users to see filesystem statistics (capacity, used, free). Adds a -df option to fs shell utility that allows unprivileged users to see filesystem statistics. Changes libhdfs and namenode jsps to use new filesystem statistics API.
Added release note.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch, hadoop4368.fsstatus.v7.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Attachment: hadoop4368.fsstatus.v5.patch
New patch attached, all working, all tests passing. This patch is ready for review.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668235#action_12668235 ]
Craig Macdonald commented on HADOOP-4368:
-----------------------------------------
Here's what I propose:
* getCapacityTotal(), getCapacityUsed(), getCapacityRemaining() remain in FSNamesystem, but implement these using the return from getStats().
* I don't known JMX, but I think deprecating and ultimately removing the above getCapacity*() methods in favour of getStats() would make the use of the relevant statistics more difficult?
* Other changes to DistributedFileSystem and DFSClient remain as in your original review.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662196#action_12662196 ]
Brian Bockelman commented on HADOOP-4368:
-----------------------------------------
Craig,
Thank you very much for helping out. Regrettably, I have not have the time to contribute/write this patch myself.
Brian
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644080#action_12644080 ]
Doug Cutting commented on HADOOP-4368:
--------------------------------------
> Could this class be promoted to FileSystem?
Yes. It would be better renamed FsStatus, and 'dfsUsed' renamed just 'used', as indicated above. The default implementation might return Long.MAX_VALUE for capacity and free, and zero for used.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667982#action_12667982 ]
Craig Macdonald commented on HADOOP-4368:
-----------------------------------------
@Nicholas: It appears that getCapacityTotal(), getCapacityUsed(), getCapacityRemaining are all exported in the FSNamesystemMBean interface. This is no longer in the public javadoc, but still exists.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654206#action_12654206 ]
Brian Bockelman commented on HADOOP-4368:
-----------------------------------------
Nah - I meant, how do we determine what the name of the superuser should be. I want to connect to Hadoop as superuser/supergroup, but you said hardcoding "superuser"/"supergroup" would not work in all cases.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653443#action_12653443 ]
Raghu Angadi commented on HADOOP-4368:
--------------------------------------
my 2c: This policy should not be worked around in fuse_dfs. It should be fixed in HDFS. fuse should handle errors (including permissions errors) for any commands normally.
+1 for not requiring superuser for df/stats in HDFS.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653325#action_12653325 ]
Brian Bockelman commented on HADOOP-4368:
-----------------------------------------
Hey all,
This has gotten a bit worse under 0.19.0. In 0.19.0, "df" simply doesn't report anything for Hadoop as a normal user but works as root. I.e., for normal user:
[brian@red ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 254G 34G 208G 14% /
/dev/sda1 99M 14M 81M 15% /boot
none 7.9G 0 7.9G 0% /dev/shm
nfs03:/mnt/raid 4.1T 1.7T 2.4T 41% /mnt/nfs03
nfs04:/mnt/raid 3.2T 1.1T 2.1T 34% /mnt/nfs04
dcache-pnfs.unl.edu:/fs
391M 79M 278M 22% /pnfs/unl.edu
For root,
[brian@red ~]$ sudo df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 254G 34G 208G 14% /
/dev/sda1 99M 14M 81M 15% /boot
none 7.9G 0 7.9G 0% /dev/shm
nfs03:/mnt/raid 4.1T 1.7T 2.4T 41% /mnt/nfs03
nfs04:/mnt/raid 3.2T 1.1T 2.1T 34% /mnt/nfs04
dcache-pnfs.unl.edu:/fs
391M 79M 278M 22% /pnfs/unl.edu
fuse 140T 122T 140T 47% /mnt/hadoop
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666672#action_12666672 ]
Doug Cutting commented on HADOOP-4368:
--------------------------------------
> 1. Should I provide ant-patch output myself?
Sure, it never hurts to do this.
> 2. Should I fix libhdfs in this issue as well?
If you like, but I don't see it as a requirement for this issue.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639114#action_12639114 ]
Doug Cutting commented on HADOOP-4368:
--------------------------------------
>I suspect that it was checked in when permissions were added to the FSNamesystem.
Did you verify this? 'svn blame' tells all.
http://tinyurl.com/42uf5d
which leads to HADOOP-2659, indicating the folks you might direct your question towards. core-dev might be the more appropriate list at this point.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Status: Open (was: Patch Available)
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch, hadoop4368.fsstatus.v7.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669344#action_12669344 ]
Hadoop QA commented on HADOOP-4368:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12399212/hadoop4368.fsstatus.v7.patch
against trunk revision 739416.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3788/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3788/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3788/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3788/console
This message is automatically generated.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch, hadoop4368.fsstatus.v7.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653397#action_12653397 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
------------------------------------------------
doConnectAsSuperuser(..) won't work very well since the constant "superuser" is not used in HDFS and the constant "supergroup" is a configurable parameter.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669377#action_12669377 ]
Craig Macdonald commented on HADOOP-4368:
-----------------------------------------
The contrib test that failed is: org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.TestStartAtOffset.testStartAfterOffset (by timeout), not related to this patch.
I think this patch is ready.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch, hadoop4368.fsstatus.v7.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669235#action_12669235 ]
Hadoop QA commented on HADOOP-4368:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12399122/hadoop4368.fsstatus.v6.patch
against trunk revision 739416.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 46 javac compiler warnings (more than the trunk's current 2563 warnings).
-1 findbugs. The patch appears to cause Findbugs to fail.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3785/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3785/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3785/console
This message is automatically generated.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641970#action_12641970 ]
Pete Wyckoff commented on HADOOP-4368:
--------------------------------------
bq. In general DFSAdmin commands should require superuser privileges.
but, not for commands where in the posix world super user is not required - I think the default should be to go with posix requirements unless there's a compelling reason in HDFS to require more.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Component/s: (was: contrib/fuse-dfs)
libhdfs
Changed components to dfs, libhdfs
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653370#action_12653370 ]
Craig Macdonald commented on HADOOP-4368:
-----------------------------------------
We cannot add a method for FsStatus for FileSystem in 0.19 branch, however, we can do in trunk. In this case, I vote +1 for accepting Brian's patch for the 0.19 branch and inclusion in the 0.19.1 release.
For trunk, the FsStatus should be added to FileSystem
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643663#action_12643663 ]
Craig Macdonald commented on HADOOP-4368:
-----------------------------------------
@Allen
The capacity operations on the namenode being discussed here are all lightweight, simply fetching three instance variables from the NameNode.
@Doug
So there is an existing DiskStatus already in DistributedFileSystem, which has a very similar API. Could this class be promoted to FileSystem?
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4368:
-------------------------------------------
Issue Type: New Feature (was: Wish)
Hadoop Flags: [Reviewed]
Craig, please add release note since this is a new feature (a new FsShell command).
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch, hadoop4368.fsstatus.v7.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Status: Patch Available (was: Open)
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch, hadoop4368.fsstatus.v7.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668256#action_12668256 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
------------------------------------------------
> I don't known JMX, but I think deprecating and ultimately removing the above getCapacity*() methods in favour of getStats() would make the use of the relevant statistics more difficult?
I see your points. Let's keep these methods.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669078#action_12669078 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
------------------------------------------------
- FsStatus.java is not included in hadoop4368.fsstatus.v6.patch
- For deprecating methods, please also add the annotation tag @Deprecated.
Everything else looks good.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653382#action_12653382 ]
Konstantin Shvachko commented on HADOOP-4368:
---------------------------------------------
I am confused. Does that mean that anybody will be able to connect to HDFS as a superuser via fuse?
It seamed like a rather simple move of a command from one class to another. Why would you want to cut corners like that?
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Status: Patch Available (was: Open)
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653511#action_12653511 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
------------------------------------------------
The main JIRA for permissions is HADOOP-1298. There is also a user guide in the forrest doc.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667353#action_12667353 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
------------------------------------------------
The patch looks good to me. The only problem I can see is that FSNamesystem.getStats() should not be declared with "throw IOException" anymore. Similarly, NameNode.getStats() should not be declared with "throw IOException".
Some minor suggestions:
- remove getCapacityTotal(), getCapacityUsed(), getCapacityUsedPercent(), getCapacityUsedNonDFS() and getCapacityRemaining() in FSNamesystem. These methods are mostly used in a test. All the uses of these methods can be replaced by getStats().
- deprecated getRawCapacity() and getRawUsed() in DistributedFileSystem and remove totalRawCapacity() and totalRawUsed() in DFSClient.
It would be great if we can remove DiskStatus instead of deprecated it. However, it seems we can't because it is public.
Thank you for working on this, Craig.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661112#action_12661112 ]
Craig Macdonald commented on HADOOP-4368:
-----------------------------------------
@Doug
Will make these changes, however before I do so do you have any comment on the last issue I raised for LocalFileSystem etc?
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639270#action_12639270 ]
Pete Wyckoff commented on HADOOP-4368:
--------------------------------------
bq. core-dev might be the more appropriate list at this
I emailed core-dev about this with the subject "Requiring super user for getStats"
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12673984#action_12673984 ]
Hudson commented on HADOOP-4368:
--------------------------------
Integrated in Hadoop-trunk #756 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/756/])
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch, hadoop4368.fsstatus.v7.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Status: Open (was: Patch Available)
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653988#action_12653988 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
------------------------------------------------
> Nicholas, what's the best way to determine the user / group I should be using?
We currently use shell commands "whoami" and "bash -c groups" to get the username and group information. See org.apache.hadoop.util.Shell.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting reassigned HADOOP-4368:
------------------------------------
Assignee: Craig Macdonald
This is looking good. A few nits:
- FsStatus should not be nested, but rather a standalone class.
- FsStatus needs Javadoc.
- We should add a unit test for this new public API (you're welcome, Nigel). This should be as new test methods in existing unit test classes.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653388#action_12653388 ]
Craig Macdonald commented on HADOOP-4368:
-----------------------------------------
bq. Does that mean that anybody will be able to connect to HDFS as a superuser via fuse?
No. Brian's patch allows the statfs call (as represented by the df command) to work correctly, by calling the appropriate libhdfs methods as superuser, before dropping the connection.
The moving of the appropriate API from DistributedFileSystem to FileSystem is indeed fairly straightforward. I guess I was raising the question as to whether that would be permitted for 0.19.1, as I obviously view the current proposed patch as an interim fix.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642987#action_12642987 ]
Allen Wittenauer commented on HADOOP-4368:
------------------------------------------
I argue that anything that doesn't require super user shouldn't be under dfsadmin to make the distinction clear.
One thing that has me concerned... what sort of load on the name node does generating the data for -report generate? If it is significant, then it should remained privileged.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640612#action_12640612 ]
Konstantin Shvachko commented on HADOOP-4368:
---------------------------------------------
You are talking about DFSAdmin -report command, right?
In general DFSAdmin commands should require superuser privileges. If -report should be opened to regular users it should not be a DFSAdmin command but rather a regular shell command. So opening getStats for all users means moving it to the FSShell command group.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662486#action_12662486 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
------------------------------------------------
We should also move the command from DFSAdmin to FsShell as mentioned in [a previous comment|https://issues.apache.org/jira/browse/HADOOP-4368?focusedCommentId=12641989#action_12641989].
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Bockelman updated HADOOP-4368:
------------------------------------
Attachment: fuse_statfs.patch
Attached a patch for 0.19.0.
This patch adds a "connect as superuser" method which statfs uses.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653979#action_12653979 ]
Brian Bockelman commented on HADOOP-4368:
-----------------------------------------
Hm - one of our sysadmins pointed out that the patch doesn't compile correctly because I submitted the wrong version of it.
Nicholas, what's the best way to determine the user / group I should be using?
Might as well get the patch right when I resubmit it.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662494#action_12662494 ]
Doug Cutting commented on HADOOP-4368:
--------------------------------------
Craig> any comment on the last issue I raised for LocalFileSystem etc?
Hmm. I guess it makes sense to make it FileSystem#getStats(Path), so that LocalFileSystem can report different stats for different volumes. Should we also define FileSystem#getStats() to call this with the root path by default as a convenience?
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Attachment: hadoop4368.fsstatus.v4.patch
@Doug: Brian, the original poster, was concerned about lookups from fuse_dfs. In actual fact the problem is with libhdfs and also the provision of an adequate Java API. In this case, fixing libhdfs as well as the Java API should resolve the entire issue.
Attached work-in-progress patch. This attempts to add libhdfs support for FsStatus API. If anyone knows JNI, perhaps they can explain why it is calling append(Path f, int bufferSize) instead of getStatus()?? I'm mystified.
{noformat}
$ ant -Dcompile.c++=true -Dlibhdfs=true clean compile test-c++-libhdfs
<snip>
[exec] Exception in thread "main" java.lang.NullPointerException
[exec] at org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:89)
[exec] at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:154)
[exec] at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:185)
[exec] at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:531)
[exec] Call to org.apache.hadoop.fs.FsStatus::getCapacity failed!
[exec] hdfsGetCapacity: -1
[exec] Exception in thread "main" java.lang.NullPointerException
[exec] at org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:89)
[exec] at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:154)
[exec] at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:185)
[exec] at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:531)
[exec] Call to org.apache.hadoop.fs.FsStatus::getUsed failed!
<snip>
{noformat}
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641989#action_12641989 ]
Doug Cutting commented on HADOOP-4368:
--------------------------------------
Pete, no one disagrees with your thesis, that getStats should not be restricted to superuser. The solution is to move it from DFSAdmin to FSShell. Unfortunately it's not quite that simple, since this is currently DFS-specific, and FSShell is generic. But capacity is a fairly generic operation. So a proper fix might also add a FileSsytem method like:
{code}
public FsStatus FileSystem#getStatus()
public class FsStatus {
public long getCapacity();
public long getFree();
public long getUsed();
}
{code}
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Attachment: hadoop4368.fsstatus.v6.patch
Updated patch addressing agreed changes.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640627#action_12640627 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
------------------------------------------------
> In general DFSAdmin commands should require superuser privileges.
+1
When I worked on HADOOP-2659, I added checks for superuser for all DFSAdmin commands.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Status: Patch Available (was: Open)
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640634#action_12640634 ]
Brian Bockelman commented on HADOOP-4368:
-----------------------------------------
Hey Konstantin, Nicholas,
The DFSAdmin -report command gives you a lot more information than what getStats does. getStats only returns the value of capacity/free/used; these are kept in internal counters, and not recomputed when the function is executed. Traditionally, knowing how large the file system is is not a protected operation in Unix.
I guess it would make sense to include the getStats command in the FSShell command group.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653478#action_12653478 ]
Pete Wyckoff commented on HADOOP-4368:
--------------------------------------
More generally, is there a design doc or jira for where permissions are going in hdfs? so we can see how these jive with fuse-dfs?
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Attachment: hadoop4368.fsstatus.v7.patch
Ignore the HadoopQA results, patch did not include all files. Improved patch is now attached (v7).
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch, hadoop4368.fsstatus.v7.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Attachment: hadoop4368.fsstatus.patch
This patch in effect corrects the Java API:
* adds FsStatus in FileSystem, and made DFSClient use this
* DiskStatus in DistributedFileSystem is deprecate, but extends FsStatus.
* FSNamesystem#getStats() no longer requires super user privelege - all it does is return the values of three variables, so does not add considerable load to the name node.
* As suggested by Doug, the default FileSystem implementation returns Long.MAX_VALUE for free and capacity, and 0 for used. This could be fixed for LocalFileSystem using the java.io.File api, however, it depends on which partition of the local filesystem that you want to query.
** In this case, perhaps getStats(Path p) would be more appropriate? DistributedFileSystem would ignore the Path parameter
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/fuse-dfs, dfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Status: Patch Available (was: Open)
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-4368:
------------------------------------
Status: Open (was: Patch Available)
Canceling patch until I address Nicholas' (straightforward) comments.
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: Wish
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4368) Superuser privileges required to do
"df"
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4368:
-------------------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Craig!
> Superuser privileges required to do "df"
> ----------------------------------------
>
> Key: HADOOP-4368
> URL: https://issues.apache.org/jira/browse/HADOOP-4368
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs, libhdfs
> Affects Versions: 0.18.1
> Reporter: Brian Bockelman
> Assignee: Craig Macdonald
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch, hadoop4368.fsstatus.v7.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
> As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
> To fix this, simply comment out the privilege check in the getStats method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.