[jira] [Assigned] (HADOOP-15571) After HADOOP-13440, multiple filesystems/file-contexts created with the same Configuration object are forced to have the same umask

["Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod Kumar Vavilapalli reassigned HADOOP-15571:
------------------------------------------------

    Assignee: Vinod Kumar Vavilapalli

Thanks for the review [~xyao]!
{quote}FileContext assume to be a per process wide settings. If uri1 and uri2 are under different context, should they use different Configuration objects? This way, the existing logic will be able to handle it properly.
{quote}
That was my first approach too, but there are too many uses of FileContext (in YARN and potentially as well as in downstream projects) and forcing all of them to create a new Configuration object is not right. This creation of such a new Config object per URI was not needed in 2.x.

> After HADOOP-13440, multiple filesystems/file-contexts created with the same Configuration object are forced to have the same umask
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-15571
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15571
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>            Priority: Critical
>         Attachments: HADOOP-15571.txt
>
>
> Ran into a super hard-to-debug issue due to this. [Edit: Turns out the same issue as YARN-5749 that [~Tao Yang] ran into]
> h4. Issue
> Configuration conf = new Configuration();
>  fc1 = FileContext.getFileContext(uri1, conf);
>  fc2 = FileContext.getFileContext(uri2, conf);
>  fc.setUMask(umask_for_fc1); // Screws up umask for fc2 also!
> This was not the case before HADOOP-13440.
> h4. Symptoms:
> h5. Scenario I ran into
> When trying to localize a HDFS directory (hdfs:///my/dir/1.txt), NodeManager tries to replicate the directory structure on the local file-system ($yarn-local-dirs/filecache/my/dir/1.txt).
> Now depending on whether NM has ever done a log-aggregation (completely unrelated code that sets umask to be 137 for its own files on HDFS), the directories /my and /my/dir on local-fs may have different permissions. In the specific case where NM did log-aggregation, /my/dir was created with 137 umask and so localization of 1.txt completely failed due to absent directory executable permissions!
> h5. Previous scenarios:
> We ran into this before in test-cases and instead of fixing the root-cause, we just fixed the test-cases: YARN-5679 / YARN-5749



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org