You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Bikas Saha (Created) (JIRA)" <ji...@apache.org> on 2012/03/31 00:47:27 UTC
[jira] [Created] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Support file permissions and ownership on Windows for RawLocalFileSystem
------------------------------------------------------------------------
Key: HADOOP-8235
URL: https://issues.apache.org/jira/browse/HADOOP-8235
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: 1.1.0
Reporter: Bikas Saha
Assignee: Bikas Saha
The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated HADOOP-8235:
-------------------------------
Assignee: Ivan Mitic (was: Bikas Saha)
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Ivan Mitic
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285879#comment-13285879 ]
Bikas Saha commented on HADOOP-8235:
------------------------------------
{code}
+ // If on windows domain, token format is DOMAIN\\user and we want to
+ // extract only the user name
+ if (Shell.WINDOWS) {
+ int i = owner.indexOf('\
{code}
Why not remove the domain in the output of winutils itself. Does any other part of the code use the domain information?
Do you see any issues in user names clashing when the domain is stripped off? If yes, then whats the solution. At least a comment would help.
{code}
+ // FIXME: Group names could have spaces on Windows
{code}
Is there a solution in mind? What is the jira tracking this?
{code}
fs.delete(systemDir, true);
if (FileSystem.mkdirs(fs, systemDir,
new FsPermission(SYSTEM_DIR_PERMISSION))) {
+ if (Shell.WINDOWS) {
+ // Explicitly set ownership on Windows, as in some scenarios
+ // Administrators group would end up being the owner what is
+ // currently not supported by the Hadoop security model.
+ fs.setOwner(systemDir, getMROwner().getShortUserName(), null);
+ }
break;
}
{code}
Any issues in doing setOwner regardless of OS? That would make sure expected ownership is always set
Same for the other places this has been done.
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Chuan Liu
> Attachments: HADOOP-8235-branch-1-win.patch
>
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Chuan Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chuan Liu updated HADOOP-8235:
------------------------------
Status: Patch Available (was: Open)
In this patch, we create a native Windows console program to emulate certain Linux command line utilities that are used by Hadoop to set file permissions, ownership, and query file status. The Windows program does not support full command line semantic and functionalities as the equivalent Linux commands. We only provide enough support for the Hadoop usage. The source code can be built on Windows with all versions of Visual Studio 10, including Visual Studio Express 10. The solution file and project file are also generated in Visual Studio 10. (Note for Visual Studio Express: Visual Studio Express does not include compiler for X64 platform. To build on X64 platform, Windows SDK is needed.)
Corresponding Java code, e.g. Shell.java, are also modified for Hadoop to pick up the correct commands on Windows. There are also a few file permission related test fixes due to different OS behaviors.
This is just an initial patch for review and collecting feedbacks from community.
We will later adding tests for the Windows program and make changes based on the feedbacks.
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Chuan Liu
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Chuan Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286743#comment-13286743 ]
Chuan Liu commented on HADOOP-8235:
-----------------------------------
I have created the following Jiras to track issues Bikas has raised, as well as some known problems.
HADOOP-8454 Fix the ‘chmod =[perm]’ bug in winutils
HADOOP-8455 Address user name format on domain joined Windows machines
HADOOP-8456 Support spaces in user names and group names.
HADOOP-8457 Address file ownership issue for users in Administrators group on Windows.
HADOOP-8453 Add unit tests for winutils
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Chuan Liu
> Attachments: HADOOP-8235-2-branch-1-win.patch, HADOOP-8235-branch-1-win.patch
>
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Chuan Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chuan Liu updated HADOOP-8235:
------------------------------
Attachment: HADOOP-8235-branch-1-win.patch
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Chuan Liu
> Attachments: HADOOP-8235-branch-1-win.patch
>
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Bikas Saha (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246714#comment-13246714 ]
Bikas Saha commented on HADOOP-8235:
------------------------------------
The proposal is to emulate Hadoop read(r), write(w), execute(x) permissions on top of the Windows ACL based access control. RawLocalFileSystem could be improved to map rwx to Windows permissions and set them for the users and groups using the “icacls” Windows command. Similar commands would need to found for other operations. E.g. we can use “takeown” command for changing the ownership of a file/dir.
There is no concept of “others” on Windows but that could be emulated using either the built-in “Everyone” or “AuthenticatedUsers” Windows groups. RawLocalFileStatus would map “others” to one of these groups when running on Windows. Alternatively, Hadoop daemons could always assign no permissions to "others" on Windows or make other permissions == group permissions.
When a user creates a file on Unix then it gets group permissions based on the default group of the user. On Windows, such a file could inherit permissions ACL's from multiple groups (say machine Administrators etc) and we might need to see how the default group permissions can be made to work.
Lastly, Hadoop uses java setWritable/setReadable etc commands which dont work in some cases for Windows.
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286074#comment-13286074 ]
Hadoop QA commented on HADOOP-8235:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530270/HADOOP-8235-2-branch-1-win.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1061//console
This message is automatically generated.
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Chuan Liu
> Attachments: HADOOP-8235-2-branch-1-win.patch, HADOOP-8235-branch-1-win.patch
>
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated HADOOP-8235:
-------------------------------
Assignee: Chuan Liu (was: Ivan Mitic)
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Chuan Liu
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-8235:
---------------------------------
Resolution: Fixed
Target Version/s: HADOOP-1-Windows (was: 1.1.0, 0.24.0)
Status: Resolved (was: Patch Available)
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Chuan Liu
> Attachments: HADOOP-8235-2-branch-1-win.patch, HADOOP-8235-branch-1-win.patch
>
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-8235:
---------------------------------
Attachment: HADOOP-8235-2-branch-1-win.patch
Please use dos2unix to convert patch files to unix format - it deals with the ^Ms
Uploaded converted patch.
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Chuan Liu
> Attachments: HADOOP-8235-2-branch-1-win.patch, HADOOP-8235-branch-1-win.patch
>
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8235) Support file permissions and
ownership on Windows for RawLocalFileSystem
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286256#comment-13286256 ]
Sanjay Radia commented on HADOOP-8235:
--------------------------------------
Please file jiras to addresses the issues Bikas has raised if needed. In the meantime I will commit the patch to the branch because this patch fixes a large number of tests.
> Support file permissions and ownership on Windows for RawLocalFileSystem
> ------------------------------------------------------------------------
>
> Key: HADOOP-8235
> URL: https://issues.apache.org/jira/browse/HADOOP-8235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
> Affects Versions: 1.1.0
> Reporter: Bikas Saha
> Assignee: Chuan Liu
> Attachments: HADOOP-8235-2-branch-1-win.patch, HADOOP-8235-branch-1-win.patch
>
>
> The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira