You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Chuan Liu (JIRA)" <ji...@apache.org> on 2012/08/14 01:31:38 UTC
[jira] [Created] (HADOOP-8694) Create true symbolic links on
Windows
Chuan Liu created HADOOP-8694:
---------------------------------
Summary: Create true symbolic links on Windows
Key: HADOOP-8694
URL: https://issues.apache.org/jira/browse/HADOOP-8694
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
Fix For: 1-win
In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Mitic updated HADOOP-8694:
-------------------------------
Fix Version/s: (was: 1-win)
Status: Patch Available (was: Open)
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Attachments: HADOOP-8694-branch-1-win-2.patch, HADOOP-8694-branch-1-win.patch, secpol.png
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450980#comment-13450980 ]
Ivan Mitic commented on HADOOP-8694:
------------------------------------
Thanks Chuan, updated patch looks good, +1
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Fix For: 1-win
>
> Attachments: HADOOP-8694-branch-1-win-2.patch, HADOOP-8694-branch-1-win.patch, secpol.png
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Chuan Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chuan Liu updated HADOOP-8694:
------------------------------
Attachment: HADOOP-8694-branch-1-win-2.patch
Attach a new patch.
>1. FileUtil#symlink: Should we log a warning on ExitCodeException to be consistent with the previous behavior?
Add the same warning message in the new patch.
>2. symlink.c: IMO, we should remove CreateJunctionPointW if it is not used in the codebase.
Removed.
>3. Would be good to add a few test cases for the symlink functionality.
Add a new test case in TestFileUtil. [~enis] contributed this.
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Fix For: 1-win
>
> Attachments: HADOOP-8694-branch-1-win-2.patch, HADOOP-8694-branch-1-win.patch, secpol.png
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Chuan Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chuan Liu updated HADOOP-8694:
------------------------------
Attachment: secpol.png
Attach a screenshot of Local Security Policy console (secpol.msc), where users can add symbolic links privilege to their accounts.
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Fix For: 1-win
>
> Attachments: HADOOP-8694-branch-1-win.patch, secpol.png
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Chuan Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chuan Liu updated HADOOP-8694:
------------------------------
Attachment: HADOOP-8694-branch-1-win.patch
Attach a patch that will create symbolic links on Windows instead of copying files. Users need to add symbolic privilege to the account running Hadoop. If the user is in Administrators group, Hadoop need to be run in elevated mode.
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Fix For: 1-win
>
> Attachments: HADOOP-8694-branch-1-win.patch
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450700#comment-13450700 ]
Ivan Mitic commented on HADOOP-8694:
------------------------------------
Thanks Chuan for the patch. It's great to see symlink functionality on Windows.
A few minor comments:
1. FileUtil#symlink: Should we log a warning on {{ExitCodeException}} to be consistent with the previous behavior?
2. symlink.c: IMO, we should remove {{CreateJunctionPointW}} if it is not used in the codebase.
3. Would be good to add a few test cases for the symlink functionality.
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Fix For: 1-win
>
> Attachments: HADOOP-8694-branch-1-win.patch, secpol.png
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454385#comment-13454385 ]
Hadoop QA commented on HADOOP-8694:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12544286/HADOOP-8694-branch-1-win-2.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1446//console
This message is automatically generated.
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Attachments: HADOOP-8694-branch-1-win-2.patch, HADOOP-8694-branch-1-win.patch, secpol.png
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467151#comment-13467151 ]
Suresh Srinivas commented on HADOOP-8694:
-----------------------------------------
+1 for the patch. I do not see any DOS mode characters in the patch.
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Attachments: HADOOP-8694-branch-1-win-2.patch, HADOOP-8694-branch-1-win.patch, secpol.png
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454560#comment-13454560 ]
Bikas Saha commented on HADOOP-8694:
------------------------------------
+1 looks good.
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Attachments: HADOOP-8694-branch-1-win-2.patch, HADOOP-8694-branch-1-win.patch, secpol.png
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-8694:
------------------------------------
Resolution: Fixed
Fix Version/s: 1-win
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I committed the patch. Thank you Chuan.
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Fix For: 1-win
>
> Attachments: HADOOP-8694-branch-1-win-2.patch, HADOOP-8694-branch-1-win.patch, secpol.png
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8694) Create true symbolic links on
Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454562#comment-13454562 ]
Bikas Saha commented on HADOOP-8694:
------------------------------------
The vcproj changes look like all lines have edits? Is it some line ending issue? Could you run this through dos2unix?
> Create true symbolic links on Windows
> -------------------------------------
>
> Key: HADOOP-8694
> URL: https://issues.apache.org/jira/browse/HADOOP-8694
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Chuan Liu
> Assignee: Chuan Liu
> Attachments: HADOOP-8694-branch-1-win-2.patch, HADOOP-8694-branch-1-win.patch, secpol.png
>
>
> In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links.
>
> In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop.
> This JIRA is created to tracking symlink support on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira