[jira] [Commented] (YARN-5727) Improve YARN shared cache support for LinuxContainerExecutor

["Chris Trezzo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578701#comment-16578701 ] 

Chris Trezzo commented on YARN-5727:
------------------------------------

As stated above, I no longer think that the v1 design attached is the right idea. It takes the approach of assuming that the permissions issue is a problem at the YARN layer. In fact, I think this is more a problem with the way map reduce supports the shared cache. Currently the shared cache only supports public resources (i.e. all resources uploaded to the shared cache will be world readable). The problem is that MapReduce is localizing all of the job resources into the user cache instead of a public one. YARN is then put in a place where it is essentially changing permissions for some resources. Ideally, MapReduce would initially localize resources intended for the shared cache into the public cache to begin with. This would allow the shared cache uploader to checksum and upload the resources even in the case of linux container executor.

> Improve YARN shared cache support for LinuxContainerExecutor
> ------------------------------------------------------------
>
>                 Key: YARN-5727
>                 URL: https://issues.apache.org/jira/browse/YARN-5727
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Chris Trezzo
>            Priority: Major
>         Attachments: YARN-5727-Design-v1.pdf
>
>
> When running LinuxContainerExecutor in a secure mode ({{yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users}} set to {{false}}), all localized files are owned by the user that owns the container which localized the resource. This presents a problem for the shared cache when a YARN application requests a resource to be uploaded to the shared cache that has a non-public visibility. The shared cache uploader (running as the node manager user) does not have access to the localized files and can not compute the checksum of the file or upload it to the cache. The solution should ideally satisfy the following three requirements:
> # Localized files should still be safe/secure. Other users that run containers should not be able to modify, or delete the publicly localized files of others.
> # The node manager user should be able to access these files for the purpose of checksumming and uploading to the shared cache without being a privileged user.
> # The solution should avoid making unnecessary copies of the localized files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org