You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Eric Badger (Jira)" <ji...@apache.org> on 2021/02/02 00:58:00 UTC

[jira] [Updated] (YARN-10607) User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH

     [ https://issues.apache.org/jira/browse/YARN-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Badger updated YARN-10607:
-------------------------------
    Description: 
When using the tarball approach to ship relevant Hadoop jars to containers, it is helpful to set {{mapreduce.admin.user.env}} to something like {{PATH=./hadoop-tarball:\{\{PATH\}\}}} to make sure that all of the Hadoop binaries are on the PATH. This way you can call {{hadoop}} instead of {{./hadoop-tarball/hadoop}}. The intention here is to force prepend {{./hadoop-tarball}} and then append the set {{PATH}} afterwards. But if a user would like to override the appended portion of {{PATH}} in their environment, they are unable to do so. This is because {{PATH}} ends up getting parsed twice. Initially it is set via {{mapreduce.admin.user.env}} to {{PATH=./hadoop-tarball:$SYS_PATH}}}. In this case {{SYS_PATH}} is what I'll refer to as the normal system path. E.g. {{/usr/local/bin:/usr/bin}}, etc.

After this, the user env parsing happens. For example, let's say the user sets their {{PATH}} to {{PATH=.:$PATH}}. We have already parsed {{PATH}} from the admin.user.env. Then we go to parse the user environment and find the user also specified {{PATH}}. So {{$PATH}} ends up getting getting expanded to {{./hadoop-tarball:$SYS_PATH}}, which leads to the user's {{PATH}} being {{PATH=.:./hadoop-tarball:$SYS_PATH}}. We then append this to {{PATH}}, which has already been set in the environment map via the admin.user.env. So we finally end up with {{PATH=./hadoop-tarball:$SYS_PATH:.:./hadoop-tarball:$SYS_PATH}}

This normally isn't a huge deal, but if you want to override the version of python, perl, etc. that is on the host or in the container, you need to prepend it to your {{PATH}} or you will need to refer to it by its full path. This is a pain and it should be possible to prepend to {{PATH}} to override the system/container {{PATH}} even if you want to prepend a {{PATH}} for you hadoop tarball.

  was:
When using the tarball approach to ship relevant Hadoop jars to containers, it is helpful to set {{mapreduce.admin.user.env}} to something like {{PATH=./hadoop-tarball:\{\{PATH\}\}}} to make sure that all of the Hadoop binaries are on the PATH. This way you can call {{hadoop}} instead of {{./hadoop-tarball/hadoop}}. The intention here is to force prepend {{./hadoop-tarball}} and then append the set {{PATH}} afterwards. But if a user would like to override the appended portion of {{PATH}} in their environment, they are unable to do so. This is because {{PATH}} ends up getting parsed twice. Initially it is set via {{mapreduce.admin.user.env}} to {{PATH=./hadoop-tarball:$SYS_PATH}}}. In this case {{SYS_PATH}} is what I'll refer to as the normal system path. E.g. {{/usr/local/bin:/usr/bin}}, etc.

After this, the user env parsing happens. For example, let's say the user sets their {{PATH}} to {{PATH=.:$PATH}}. We have already parsed {{PATH}} from the admin.user.env. Then we go to parse the user environment and find the user also specified {{PATH}}. So {{$PATH}} ends up getting getting expanded to {{./hadoop-tarball:$SYS_PATH}}, which leads to the user's {{PATH}} being {{PATH=.:./hadoop-tarball:$SYS_PATH}}. We then append this to {{PATH}}, which has already been set in the environment map via the admin.user.env. So we finally end up with {{PATH=./hadoop-tarball:$SYS_PATH:.:./hadoop-tarball:$SYS_PATH}}

This normally isn't a huge deal, but if you want to override the version of python, perl, etc. that is on the host or in the container, you need to prepend it to your {{PATH}} or you will need to refer to it by its full path. This is a pain and it should be possible to prepend to {{PATH}} to override the system/container {{PATH}} even if you have admin.user.env set to prepend a {{PATH}}


> User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH
> ---------------------------------------------------------------------------------------
>
>                 Key: YARN-10607
>                 URL: https://issues.apache.org/jira/browse/YARN-10607
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Eric Badger
>            Assignee: Eric Badger
>            Priority: Major
>
> When using the tarball approach to ship relevant Hadoop jars to containers, it is helpful to set {{mapreduce.admin.user.env}} to something like {{PATH=./hadoop-tarball:\{\{PATH\}\}}} to make sure that all of the Hadoop binaries are on the PATH. This way you can call {{hadoop}} instead of {{./hadoop-tarball/hadoop}}. The intention here is to force prepend {{./hadoop-tarball}} and then append the set {{PATH}} afterwards. But if a user would like to override the appended portion of {{PATH}} in their environment, they are unable to do so. This is because {{PATH}} ends up getting parsed twice. Initially it is set via {{mapreduce.admin.user.env}} to {{PATH=./hadoop-tarball:$SYS_PATH}}}. In this case {{SYS_PATH}} is what I'll refer to as the normal system path. E.g. {{/usr/local/bin:/usr/bin}}, etc.
> After this, the user env parsing happens. For example, let's say the user sets their {{PATH}} to {{PATH=.:$PATH}}. We have already parsed {{PATH}} from the admin.user.env. Then we go to parse the user environment and find the user also specified {{PATH}}. So {{$PATH}} ends up getting getting expanded to {{./hadoop-tarball:$SYS_PATH}}, which leads to the user's {{PATH}} being {{PATH=.:./hadoop-tarball:$SYS_PATH}}. We then append this to {{PATH}}, which has already been set in the environment map via the admin.user.env. So we finally end up with {{PATH=./hadoop-tarball:$SYS_PATH:.:./hadoop-tarball:$SYS_PATH}}
> This normally isn't a huge deal, but if you want to override the version of python, perl, etc. that is on the host or in the container, you need to prepend it to your {{PATH}} or you will need to refer to it by its full path. This is a pain and it should be possible to prepend to {{PATH}} to override the system/container {{PATH}} even if you want to prepend a {{PATH}} for you hadoop tarball.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org