You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Jay Finger (JIRA)" <ji...@apache.org> on 2012/06/01 18:39:23 UTC

[jira] [Created] (HADOOP-8464) hadoop-env.sh is included twice: once via hadoop-config.sh the again explicitly via scripts

Jay Finger created HADOOP-8464:
----------------------------------

             Summary: hadoop-env.sh is included twice: once via hadoop-config.sh the again explicitly via scripts
                 Key: HADOOP-8464
                 URL: https://issues.apache.org/jira/browse/HADOOP-8464
             Project: Hadoop Common
          Issue Type: Bug
          Components: conf
    Affects Versions: 2.0.0-alpha
            Reporter: Jay Finger


Many (most? all?) scripts include hadoop-env.sh twice.  The first time is indirectly:  scripts do ". hadoop-config.sh" and hadoop-config.sh then does ". hadoop-env.sh".  Then scripts will later directly do ". hadoop-env.sh".

This double-include is error prone.

Note several Jira's reporting issues with HADOOP_LOG_DIR set to /user/user (e.g. /root/root, /hadoop/hadop, etc).  This is because hadoop-env.sh has done "HADOOP_LOG_DIR=$HADOOP_LOG_DIR/$user".  Many fixes for those issues are around addressing that one line, but it seems that this is going to be a recurring theme of breakages.  It's probably also why services are routinely started with many command line arguments repeated multiple times.

I'm not familiar enough with the "design" of all the scripts, so not sure what the solution should be.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8464) hadoop-env.sh is included twice: once via hadoop-config.sh the again explicitly via scripts

Posted by "Vinay (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290014#comment-13290014 ] 

Vinay commented on HADOOP-8464:
-------------------------------

Ideally calling hadoop-env.sh multiple times should not create any problems.

Because hadoop-config.sh itself called from so many files. Ex: hadoop-daemon.sh, slaves.sh, hdfs, yarn, etc.

We can remove the explicit call of hadoop-env.sh from scripts other than hadoop-config.sh

HADOOP-8476 will resolve all the issues.
                
> hadoop-env.sh is included twice: once via hadoop-config.sh the again explicitly via scripts
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8464
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8464
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 2.0.0-alpha
>            Reporter: Jay Finger
>
> Many (most? all?) scripts include hadoop-env.sh twice.  The first time is indirectly:  scripts do ". hadoop-config.sh" and hadoop-config.sh then does ". hadoop-env.sh".  Then scripts will later directly do ". hadoop-env.sh".
> This double-include is error prone.
> Note several Jira's reporting issues with HADOOP_LOG_DIR set to /user/user (e.g. /root/root, /hadoop/hadop, etc).  This is because hadoop-env.sh has done "HADOOP_LOG_DIR=$HADOOP_LOG_DIR/$user".  Many fixes for those issues are around addressing that one line, but it seems that this is going to be a recurring theme of breakages.  It's probably also why services are routinely started with many command line arguments repeated multiple times.
> I'm not familiar enough with the "design" of all the scripts, so not sure what the solution should be.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8464) hadoop-env.sh is included twice: once via hadoop-config.sh the again explicitly via scripts

Posted by "Vinay (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290019#comment-13290019 ] 

Vinay commented on HADOOP-8464:
-------------------------------

And one more thing is, variables configured in hadoop-env.sh should not append to its own initial value. It should be absolute. Then calling multiple times will not be a problem at all.
                
> hadoop-env.sh is included twice: once via hadoop-config.sh the again explicitly via scripts
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8464
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8464
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 2.0.0-alpha
>            Reporter: Jay Finger
>
> Many (most? all?) scripts include hadoop-env.sh twice.  The first time is indirectly:  scripts do ". hadoop-config.sh" and hadoop-config.sh then does ". hadoop-env.sh".  Then scripts will later directly do ". hadoop-env.sh".
> This double-include is error prone.
> Note several Jira's reporting issues with HADOOP_LOG_DIR set to /user/user (e.g. /root/root, /hadoop/hadop, etc).  This is because hadoop-env.sh has done "HADOOP_LOG_DIR=$HADOOP_LOG_DIR/$user".  Many fixes for those issues are around addressing that one line, but it seems that this is going to be a recurring theme of breakages.  It's probably also why services are routinely started with many command line arguments repeated multiple times.
> I'm not familiar enough with the "design" of all the scripts, so not sure what the solution should be.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8464) hadoop-env.sh is included twice: once via hadoop-config.sh the again explicitly via scripts

Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290156#comment-13290156 ] 

Daryn Sharp commented on HADOOP-8464:
-------------------------------------

Won't we lose the ability to override values if it does an absolute set?  I'd think the script should only set the value if unset.  Ie. something like: {code}${KEY:=<default value or expression>}{code}

I agree that technically it would be safe to include hadoop-env multiple times although it would be good to avoid redundant and unnecessary includes where possible.
                
> hadoop-env.sh is included twice: once via hadoop-config.sh the again explicitly via scripts
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8464
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8464
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 2.0.0-alpha
>            Reporter: Jay Finger
>
> Many (most? all?) scripts include hadoop-env.sh twice.  The first time is indirectly:  scripts do ". hadoop-config.sh" and hadoop-config.sh then does ". hadoop-env.sh".  Then scripts will later directly do ". hadoop-env.sh".
> This double-include is error prone.
> Note several Jira's reporting issues with HADOOP_LOG_DIR set to /user/user (e.g. /root/root, /hadoop/hadop, etc).  This is because hadoop-env.sh has done "HADOOP_LOG_DIR=$HADOOP_LOG_DIR/$user".  Many fixes for those issues are around addressing that one line, but it seems that this is going to be a recurring theme of breakages.  It's probably also why services are routinely started with many command line arguments repeated multiple times.
> I'm not familiar enough with the "design" of all the scripts, so not sure what the solution should be.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira