[jira] Updated: (MAPREDUCE-1288) DistributedCache localizes only once per cache URI

["Devaraj Das (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated MAPREDUCE-1288:
-----------------------------------

    Attachment: MR-1288-bp20-1.patch

This bug surfaced on one of the secure Yahoo clusters. This is the scenario:
1. There is a file "/a/b/c/1.txt" on the hdfs which is private (one of the directories in the path leading up to the hdfs file does not have EXECUTE permissions for OTHERS).
2. A user "foo" uses this file in his job as a DistributedCache file, and the TTs localizes this file in a location owned by user "foo" (since this file is private it lands up in the protected place).
3. A second user "bar" also tries to use the same file in his job. Both users belong to the same unix group.
4. Assume some TT that localized "/a/b/c/1.txt" file before, while running foo's task, got a task of bar's job. It concludes the file was already localized since the mapping has an entry for /a/b/c/1.txt (mapping refers to the mapping between the Cache URIs and the CacheStatus objects, maintained by TT). 
5. The TT doesn't localize this file again. It instead points the tasks to the file that was localized in step (2). Since the directory where the file was localized is not readable by anyone other than "foo", the tasks of "bar"'s job fails.

I guess earlier this issue didn't arise earlier (pre-security) since the distributed cache files, even if they were private, were getting localized in directories that were readable by all users.

Attaching a patch for Y20S that addresses the issue.

> DistributedCache localizes only once per cache URI
> --------------------------------------------------
>
>                 Key: MAPREDUCE-1288
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1288
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: distributed-cache, security, tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Devaraj Das
>            Priority: Critical
>         Attachments: MR-1288-bp20-1.patch
>
>
> As part of the file localization the distributed cache localizer creates a copy of the file in the corresponding user's private directory. The localization in DistributedCache assumes the key as the URI of the cachefile and if it already exists in the map, the localization is not done again. This means that another user cannot access the same distributed cache file. We should change the key to include the username so that localization is done for every user.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.