[jira] [Comment Edited] (HADOOP-11022) User replaced functions get lost 2-3 levels deep (e.g., sbin)

["Allen Wittenauer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-11022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114606#comment-14114606 ] 

Allen Wittenauer edited comment on HADOOP-11022 at 8/28/14 11:45 PM:
---------------------------------------------------------------------

The basic problem is the export of HADOOP_ENV_PROCESSED.  This is used to prevent hadoop-env.sh from being re-read.  This protection makes sense if you have, e.g., FOO_OPT="$FOO_OPTS blah" in this file.  But with the curren state of the world, there's quite a few things going wrong here:

1. We don't protect the other *-env.sh files like we do hadoop-env.sh.  i.e., yarn-env.sh will still get re-read multiple times.
2. This only works correctly because the vars in hadoop-env.sh are also exported.... which is probably the wrong thing to do as well.
3. Now that there are more than just vars in *-env.sh, the system really does need to re-read *-env.sh to pick those up.

Some potential solutions worth exploring:

* What happens if we no longer export anything in *-env.sh and re-read them at every command run like we do with the other files?  (This brings Hadoop in line with most other OS utilities.)  One side effect: users upgrading to trunk would need to also remove the export lines as part of upgrading.
* If we protect all the *-env.sh and move user defined functions to a new file?  This has a lot of potential!  One big problem is what to do about YARN_OPTS. 


was (Author: aw):
The basic problem is the export of HADOOP_ENV_PROCESSED.  This is used to prevent hadoop-env.sh from being re-read.  This protection makes sense if you have, e.g., FOO_OPT="${FOO_OPTS} blah" in this file.  But with the curren state of the world, there's quite a few things going wrong here:

1. We don't protect the other *-env.sh files like we do hadoop-env.sh.  i.e., yarn-env.sh will still get re-read multiple times.
2. This only works correctly because the vars in hadoop-env.sh are also exported.... which is probably the wrong thing to do as well.
3. Now that there are more than just vars in *-env.sh, the system really does need to re-read *-env.sh to pick those up.

Some potential solutions worth exploring:

* What happens if we no longer export anything in *-env.sh and re-read them at every command run like we do with the other files?  (This brings Hadoop in line with most other OS utilities.)  One side effect: users upgrading to trunk would need to also remove the export lines as part of upgrading.
* If we protect all the *-env.sh and move user defined functions to a new file?  This has a lot of potential!  One big problem is what to do about YARN_OPTS. 

> User replaced functions get lost 2-3 levels deep (e.g., sbin)
> -------------------------------------------------------------
>
>                 Key: HADOOP-11022
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11022
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 3.0.0
>            Reporter: Allen Wittenauer
>            Priority: Critical
>
> The code that protects hadoop-env.sh from being re-executed is also causing functions that the user replaced to get overridden with the defaults.  This typically happens when running commands that nest, such as most of the content in sbin.  Just running stuff out of bin (e.g., bin/hdfs --daemon start namenode) does not trigger this behavior.



--
This message was sent by Atlassian JIRA
(v6.2#6252)