You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Raghu Angadi (JIRA)" <ji...@apache.org> on 2007/09/11 02:27:29 UTC
[jira] Created: (HADOOP-1873) User permissions for Map/Reduce
User permissions for Map/Reduce
-------------------------------
Key: HADOOP-1873
URL: https://issues.apache.org/jira/browse/HADOOP-1873
Project: Hadoop
Issue Type: Improvement
Reporter: Raghu Angadi
HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530335 ]
Raghu Angadi commented on HADOOP-1873:
--------------------------------------
Keeping the ticket it self in the job conf implies that the file should be world readable and is going to be copied to local disk on many machines. Is that ok?
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529730 ]
Raghu Angadi commented on HADOOP-1873:
--------------------------------------
MR may not be able to renew user tickets since that would usually require access to users secret/password. But they should have their own tickets and renew them.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi reassigned HADOOP-1873:
------------------------------------
Assignee: Tsz Wo (Nicholas), SZE (was: Raghu Angadi)
Assigning to Nicholas since he is looking into permission issues.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Tsz Wo (Nicholas), SZE
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533884 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-1873:
------------------------------------------------
For preventing file/directory name guessing, we could append large random numbers to file names. Suppose we want to store foo. We use the name foo2561346125341261461254 instead. Then, jobtracker could first get the file name by "ls foo*" and then read the file. Other users without directory lookup permission are not possible to guess the name.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557058#action_12557058 ]
Hadoop QA commented on HADOOP-1873:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372730/mapred7.patch
against trunk revision .
@author +1. The patch does not contain any @author tags.
javadoc -1. The javadoc tool appears to have generated messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1517/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1517/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1517/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1517/console
This message is automatically generated.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Affects Versions: 0.15.1
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Fix For: 0.16.0
>
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch, mapred7.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545569 ]
Owen O'Malley commented on HADOOP-1873:
---------------------------------------
To address Doug's concerns about people stuff big stuff into the xml, how about:
{code}
public class Ticket {
...
public static void setTicket(Configuration conf, String attributeName, Ticket ticket);
public static Ticket getTicket(Configuration conf, String attributeName);
}
{code}
so that we can hide how the tickets are stored in the job conf.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi reassigned HADOOP-1873:
------------------------------------
Assignee: Raghu Angadi
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555729#action_12555729 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
It's unfortunate that we have to alter the configuration to set the protections we want.
To make this safer, we could either:
- restore the umask in a 'finally' clause, so it's restored on exceptions; or
- set the umask in a copy of the configuration, so that it doesn't need to be restored.
In either case, it might be cleaner to have utility methods that perform file create and mkdir with a modified umask.
Also, in JobClient, the umask is restored after the job is serialized, so the jobtracker will see the modified value, not the original.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556026#action_12556026 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
> The cost of copying a configuration might be too big, so the patch takes the first option.
I like the utility methods you've added, but I still worry that they're not thread-safe. File creation in another thread could get the wrong umask.
The cost of copying a configuration should not be too great, since it mostly just copies the Properties. Even that could be eliminated by, in Configuration's copy constructor, replacing 'properties.clone()' with 'new Properties(properties)'. This is safe, since we do not permit removing configuration properties nor setting them to null.
Also, you didn't use the new utility method in one of the calls to mkdirs(). Is that intentional?
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch, mapred3.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-1873:
----------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thank you Hairong.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Affects Versions: 0.15.1
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Fix For: 0.16.0
>
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch, mapred7.patch, mapred8.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529695 ]
Allen Wittenauer commented on HADOOP-1873:
------------------------------------------
Yes, Kerberos tickets have timeouts. In fact, it is ultra-important that MR jobs be able to autorenew any renewable tickets they are passed.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555700#action_12555700 ]
Hairong Kuang commented on HADOOP-1873:
---------------------------------------
There was a typo in my previous comment. Here is the right version.
If not, the mapred system directory should be under a directory which is writable by the user who starts the JOBTRACKER.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530337 ]
Allen Wittenauer commented on HADOOP-1873:
------------------------------------------
No. World readable means that anyone on the node will be able to grab the ticket and impersonate that user. The keytab cache file needs to be protected.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Attachment: mapred3.patch
The cost of copying a configuration might be too big, so the patch takes the first option.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch, mapred3.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558235#action_12558235 ]
Hudson commented on HADOOP-1873:
--------------------------------
Integrated in Hadoop-Nightly #363 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/363/])
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Affects Versions: 0.15.1
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Fix For: 0.16.0
>
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch, mapred7.patch, mapred8.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546512 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
I agree that transmitting the job.xml data via RPC is probably the best long-term solution. The job.jar file is also a potential issue. Perhaps we can move all access to it to the task jvm, instead of having the tasktracker unpack it? But, again, the short-term solution of placing both of these in a world-writable, system-readable directory works fine.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546365 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
> Setting job files to be worldwide readable but owner-writable;
Will we permit per-file permissions? Wasn't there some discussion that we should only support per-directory permissions?
There are two entities which must be able to read job files: the jobtracker/tasktracker and the submitter. It seems to me this can be handled by setting the group to be the jobtracker/tasktracker group, and the owner to be the submitter. Then they should be readable by user and group but not world. The group of jobtracker/tasktracker could perhaps be returned by a JobClient method.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Status: Open (was: Patch Available)
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Affects Versions: 0.15.1
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Fix For: 0.16.0
>
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch, mapred7.patch, mapred8.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546459 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
> We only need a single group.
Sorry. I was mistaken. I assumed that the owner could change the group to a group that they're not a member of, but in fact that's not permitted. Oops.
So perhaps the system dir can be world-writable but not world-readable or executable? Then users can add files to it when they're submitting jobs, but other users cannot read those files? Could that work? My hope is to avoid having the jobtracker and tasktracker assume the user's identity if possible, but still make it so that a users job submissions are not publicly readable.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-1873) User permissions for
Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553482 ]
hairong edited comment on HADOOP-1873 at 12/19/07 12:43 PM:
------------------------------------------------------------------
This patch makes the following changes to map/reduce:
1. JobTracker removes all the files/subdirectories under the map/reduce system instead of removing the mapreduce system directory. If the system directory does not exist, it creates it with the permission \-wx\-wx\-wx. This requires that the runner of the job tracker is the same as the namenode.
2. JobClient creates job direoctories using the permission \-wx\-wx\-wx and creates job files using the permission rw\-r\-\-r\-\-. It also stores its user id in the job configuration.
was (Author: hairong):
This patch makes the following changes to map/reduce:
1. JobTracker removes all the files/subdirectories under the map/reduce system instead of removing the mapreduce system directory. If the system directory does not exist, it creates it with the right permission -wx-wx-wx. This requires that the runner of the job tracker is the same as the namenode.
2. JobClient creates job direoctories using the permission -wx-wx-wx and creates job files using the permission rw-r--r--. It also stores its user id in the job configuration.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557856#action_12557856 ]
Hadoop QA commented on HADOOP-1873:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372826/mapred8.patch
against trunk revision r610978.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1540/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1540/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1540/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1540/console
This message is automatically generated.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Affects Versions: 0.15.1
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Fix For: 0.16.0
>
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch, mapred7.patch, mapred8.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546500 ]
Hairong Kuang commented on HADOOP-1873:
---------------------------------------
This sounds good to me.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Attachment: mapred5.patch
This patch also reflects Nicholas's comment.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546504 ]
Hairong Kuang commented on HADOOP-1873:
---------------------------------------
As Owen said, in the long run we should let a jobclient to transfer job files to the jobtracker and let the jobtracker to upload the files to dfs. This way, we do not need to worry about the permission problem and it is more secure. But let's do it in the next release.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556969#action_12556969 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
Another option: In the FileSystem.create and FileSystem.mkdirs static utility methods, we might create the file or directory first, then set the protection. This has the disadvantage of making two RPC calls, but it has the advantage of being thread safe. In the current case (job submission) the performance impact of these extra RPCs would be negligible, no?
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530585 ]
Raghu Angadi commented on HADOOP-1873:
--------------------------------------
It is not clear to me when very generic "pluggable/kerberos" authentication will be supported very well in HDFS yet. I know that it will be supported in future. Also this is my first time working with Map/Reduce part of Hadoop at more than superficial level. I would like to propose modest goals for this jira and not add major new requirements for for HADOOP-1298 :
# The system is not less secure than it currently is. easy to to do :).
# No chages to either map-reduce config or map reduce code should be required.
# System directory is world writable (at least to create directories).
# use (restrictive?) umask when it is supported in 'fs.create()/mkdirs()'.
# Pass only the "user name" from job client. Either option of jobconf file or rpc is fine. Jobconf file seems simpler and intuitive, except that conf file should be world readable, I think it is ok for the first version.
# MapReduce creates a "SimpleUserTicket" from the user name uses it when ever it is doing file io on behalf of user's job/task.
# Not sure yet what user the mapReduce itself uses. The user that starts it seems ok for now.
Note that all of this is transparent to the user and improving the implementation would not break external interfaces. I think this simpler approach itself add a very useful functionality. I would like to think of this as making MapReduce work with HDFS permissions and not as a major security overhaul of Map/Reduce.
I mentioned about the expiring tickets etc earlier more as something "we care about but not implementing yet".
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Attachment: mapred7.patch
This sounds good to me. I guess this is how "mkdir -m" is implemented in Unix. Here is a new patch.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch, mapred7.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Christophe Taton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533887 ]
Christophe Taton commented on HADOOP-1873:
------------------------------------------
Just a small warning: these kind of file access restrictions work under fully compliant POSIX permissions. As it has been discussed in HADOOP-1298, the current design is to add permissions on directories only, which would actually not allow this kind of restriction directly.
We could achieve this by inserting sub-directories to allow any access restriction scheme, or else we'll need to redesign H-1298 to provide full-POSIX permissions.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546415 ]
Hairong Kuang commented on HADOOP-1873:
---------------------------------------
> Will we permit per-file permissions? Wasn't there some discussion that we should only support per-directory permissions?
Yes, we do support per-file permission. The only things that we do not support are sticky bits and excutable file permission.
> There are two entities which must be able to read job files: the jobtracker/tasktracker and the submitter. It seems to me this can be handled by setting the group to be the jobtracker/tasktracker group, and the owner to be the submitter. Then they should be readable by user and group but not world. The group of jobtracker/tasktracker could perhaps be returned by a JobClient method.
Most of dfs users are also map/reduce users. So using jobtracker/tasktracker group is almost equivalent to world-wide readable. Secondly dfs does not support creating groups. All the groups and user names are fetched from unix. So it is quite a burden to create a jobtracker/tasktracker group for every mapreduce user in Unix.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553988 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-1873:
------------------------------------------------
I introduce a new static method FsPermission.createImmutable(short) in HADOOP-1298. Please use it for the constants.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Marco Nicosia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marco Nicosia updated HADOOP-1873:
----------------------------------
Description:
HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
was:
HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
hairong> It's recommended that the jobtracker is run as the
hairong> same user as the namenode.
User code cannot run as the same user as the HDFS daemons. If they run as same, then user code has the ability to kill the HDFS daemons, which we've seen in the past.
Additionally, we've reasoned that to be truly "data secure" the user code cannot run as the same user as the owner of the HDFS hadoop/dfs/data directories. User code could scan the host's filesystem (ufs, ext3, etc) directories to read all user data.
Hadoop doesn't currently switch users before running user code. Even if it did, that'd mean we'd have to run everything (NameNode, JobTracker) as root, which would have a big impact as well.
hairong> If not, mapred system directory should be under a
hairong> directory which is writable by the user who starts
hairong> the namenode.
Wouldn't this result in a security hole?
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Attachment: mapred4.patch
This patch incorporates Doug's comments. It also fixed a bug in Configuration.getProps(), which does not load the properties in overlay correctly.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Attachment: mapred6.patch
Mmm...Copying configuration still does not work. Any change to the copied configuration is not visible to the file system handler. In order for the change to take effect, we have to close the file system handle (so it is removed from FileSystem cache and then refetch it. So the mkdirs code looks like:
{noformat}
public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
throws IOException {
Configuration conf = new Configuration(fs.getConf());
conf.setInt(FsPermission.UMASK_LABEL, 0);
fs.close();
fs = FileSystem.get(conf);
return fs.mkdirs(dir, permission);
}
{noformat}
But closing the file handler is not acceptable and refeching a new file system handler is too expensive. The attached patch reverts to option 1.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530499 ]
Owen O'Malley commented on HADOOP-1873:
---------------------------------------
Actually, I think we should add into Configuration:
{code}
void setWritable(String key, Writable value) throws IOException;
Writable getWritable(String key) throws IOException;
{code}
which puts a generic writable into a Configuration with stringification/quoting and use that to put the tickets into the JobConf.
Another issue that needs to be addressed for map/reduce is the usage of the system directory, which is related to the job conf security. Obviously the current usage doesn't work at all once we have permissions. I'm not sure what the right solution is. We either need to move the job.xml and the input splits via rpc or some other solution. The easiest solution would be to give the JobTracker super-user dfs permissions, but that doesn't map well into have dynamic map/reduce clusters. Clearly the job.xml (and input splits) should be protected for general reading/writing.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529585 ]
Raghu Angadi commented on HADOOP-1873:
--------------------------------------
The current proposal is to just a ticket through {{submitJob()}} interface and propegate it to tasks. It may not work if a ticket has a timout (kerberos?). This implementation might have to when there is support for short lived timeout. In that case, HDFS could support a special ticket that can impersonate other users.:
- when Job tracker accepts a job it first validates the user. If the user is allowed to run a job, it creates a new ticket (using the configured security module) and renews the tickets over time (Just like DFSClient would review).
- for Namenode RPCs, it uses a wrapper/nested ticket that includes the user it wants to impersonate. Namenode verifies the actual user and if that users is allowed to impersonate others, then executes the RPC as the intended user.
I think for now, we can just pass the JobClient's ticket around.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530332 ]
Raghu Angadi commented on HADOOP-1873:
--------------------------------------
Owen says kerberos tickets are mostly renewable without private information.
Also, Owen suggested we should propegate the ticket through job conf file. This sounds good to me. It implies we should be able to write and read objects as entries in the conf file. I will try adding putWritable(), getWritable() interface.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546358 ]
Hairong Kuang commented on HADOOP-1873:
---------------------------------------
Per discussion with Owen, I'd like to adress the problem by
1. Setting map/reduce system directory to be world wide readable and writable;
2. Setting job files to be worldwide readable but owner-writable;
3. User/groups information is saved in configuration file as a string but the detail is encapsulated in methods saveToConf/readFromConf as discussed in HADOOP-2229.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530485 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
> It implies we should be able to write and read objects as entries in the conf file. [ ... ]
Why can't we just convert tickets to strings? Configuration#set(String name, Object value) calls the toString() method, so all that we really need is a ticket constructor or factory method that takes a string, no?
> Keeping the ticket it self in the job conf implies that the file should be world readable [...]
Not necessarily. The job conf file should be made readable in HDFS only by its creator and by the mapreduce system. The local copy of job.xml on a jobtracker node should also not be visible to other jobs, but that might be harder to implement...
I'm still on the fence about whether we should store tickets in the job or handle them separately. It would certainly be more convenient to keep them in the job configuration, since that already travels with the job, and, arguably, the other parts of a job configuration should also only be viewable by the job's owner and the mapreduce system. The alternative I see would be to never write tickets to disk, but to only pass them in RPCs, from client to jobtracker, from jobtracker to tasktracker, and from tasktracker to task. That would certainly be more secure, but it might also be overkill.
A hybrid approach that occurs to me would be to never write job configurations to disk, but rather to keep them in memory and pass them via RPC. The job.jar would still be on disk, and potentially be viewable by other jobs, but job.xml would no longer exist. Is that viable?
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556989#action_12556989 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
+1 this looks good to me. Thanks for your patience in working this out!
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch, mapred7.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546425 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
> Most of dfs users are also map/reduce users. So using jobtracker/tasktracker group is almost equivalent to world-wide readable.
I don't follow this. We should have a group called something like "mapred-daemon" that most users do not belong to but that the user running the mapred daemons does belong to. That group should be able to read the submitted job files. Won't that work?
> All the groups and user names are fetched from unix. So it is quite a burden to create a jobtracker/tasktracker group for every mapreduce user in Unix.
We only need a single group. If adding a new group for this is impractical then we could use the standard unix group named "daemon", and run all daemons as "daemon". We could change our launch scripts to use 'sudo -u daemon'. The default user and group could be "daemon", optionally overridden by an environment variable in conf/hadoop-env.sh.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530462 ]
Allen Wittenauer commented on HADOOP-1873:
------------------------------------------
BTW, I would expect the key cache to have not just the tickets for the MR job but any other tickets submitted with the job. For example, if my MR job needed to write to a Kerberized NFS mount, those tickets should go with the job, be renewed as needed, etc.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Fix Version/s: 0.16.0
Affects Version/s: 0.15.1
Status: Patch Available (was: Open)
Thank Doug for working with me to get it done.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Affects Versions: 0.15.1
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Fix For: 0.16.0
>
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch, mapred7.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Attachment: mapred8.patch
This patch fixed the javadoc warning.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Affects Versions: 0.15.1
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Fix For: 0.16.0
>
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch, mapred7.patch, mapred8.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Attachment: mapred.patch
This patch makes the following changes to map/reduce:
1. JobTracker removes all the files/subdirectories under the map/reduce system instead of removing the mapreduce system directory. If the system directory does not exist, it creates it with the right permission -wx-wx-wx. This requires that the runner of the job tracker is the same as the namenode.
2. JobClient creates job direoctories using the permission -wx-wx-wx and creates job files using the permission rw-r--r--. It also stores its user id in the job configuration.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545571 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
+1
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530575 ]
Owen O'Malley commented on HADOOP-1873:
---------------------------------------
> That's not clear to me. Can't we make the system directory world-writable but readable only by
> the jobtracker/tasktracker user?
I assume you mean that the system directory is rwx-wx-wx so that you can create directories in it, but not scan what is there? Even so, the job directory under the system dir would need to be world readable and writable so that the job tracker can read it and delete it when the job is done. But you don't want the users to be able to guess the directory names and grab each other's tickets by looking at their job.xmls. I suspect that we just need to send the JobConf and input splits via rpc.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546366 ]
Allen Wittenauer commented on HADOOP-1873:
------------------------------------------
(I totally admit that I'm only half paying attention this and I feel guilt about that... but... )
> Will we permit per-file permissions? Wasn't there some discussion that we should only support per-directory permissions?
If you don't do per-file permissions, directories like /tmp become a major battle ground as, at least in our experiences, everybody expects to be able to write there... so then the question becomes, how do you protect users from overwriting each other in what are essentially public directories?
[Although, this does beg another question: should hadoop have the equiv. of mkstemp() and friends?]
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530501 ]
Doug Cutting commented on HADOOP-1873:
--------------------------------------
> void setWritable(String key, Writable value) throws IOException;
Yikes. I guess we can go that way, but it's going to encourage putting big binary stuff into XML, which is not optimal.
> Obviously the current usage doesn't work at all once we have permissions.
That's not clear to me. Can't we make the system directory world-writable but readable only by the jobtracker/tasktracker user?
> give the JobTracker super-user dfs permissions
I hope we don't have to do that. We should have a user id for the job submitter and a user id for the jobtracker/tasktracker. For a dynamic, single-user mapred cluster, these could be the same. For a static mapred cluster, these could be different. In either case, other users should not be able to read one's job files.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang reassigned HADOOP-1873:
-------------------------------------
Assignee: Hairong Kuang (was: Tsz Wo (Nicholas), SZE)
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Status: Patch Available (was: Open)
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Affects Versions: 0.15.1
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Fix For: 0.16.0
>
> Attachments: mapred.patch, mapred2.patch, mapred3.patch, mapred4.patch, mapred5.patch, mapred6.patch, mapred7.patch, mapred8.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1873) User permissions for Map/Reduce
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1873:
----------------------------------
Attachment: mapred2.patch
After some thought, I revert JobTracker's old behavior, i.e., removing the mapreduce system directory at the start up time. It's recommended that the jobtracker is run as the same user as the namenode. If not, mapred system directory should be under a directory which is writable by the user who starts the namenode. I will incorporates Nicholas's comment after HADOOP-1298 is committed.
> User permissions for Map/Reduce
> -------------------------------
>
> Key: HADOOP-1873
> URL: https://issues.apache.org/jira/browse/HADOOP-1873
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Attachments: mapred.patch, mapred2.patch
>
>
> HADOOP-1298 and HADOOP-1701 add permissions and pluggable security for DFS files and DFS accesses. Same users permission should work for Map/Reduce jobs as well.
> User persmission should propegate from client to map/reduce tasks and all the file operations should be subject to user permissions. This is transparent to the user (i.e. no changes to user code should be required).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.