You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Radim Kolar (Created) (JIRA)" <ji...@apache.org> on 2012/04/03 20:00:29 UTC
[jira] [Created] (MAPREDUCE-4101) nodemanager depends on /bin/bash
nodemanager depends on /bin/bash
--------------------------------
Key: MAPREDUCE-4101
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: nodemanager
Affects Versions: 0.23.1
Environment: FreeBSD 8.2 / 64 bit
Reporter: Radim Kolar
Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
i found 2 cases:
1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Bikas Saha (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247739#comment-13247739 ]
Bikas Saha commented on MAPREDUCE-4101:
---------------------------------------
so we should work on fixing it and not adding new ones, right?
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4101) nodemanager depends on /bin/bash
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar updated MAPREDUCE-4101:
-----------------------------------
Status: Open (was: Patch Available)
patch is probably wrong
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4101) nodemanager depends on /bin/bash
Posted by "Radim Kolar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar updated MAPREDUCE-4101:
-----------------------------------
Attachment: bash-replace-by-sh.txt
Replace Bash by standard /bin/sh. Currently used functionality is bash -c "command" which is supported by standard sh too
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262437#comment-13262437 ]
Radim Kolar commented on MAPREDUCE-4101:
----------------------------------------
There is already configurable shell feature - MRJobConfig.MAPRED_ADMIN_USER_SHELL it is just not used for anything except putting into environment in TaskAttemptImpl.
You want to use this setting for every container launch?
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Bikas Saha (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247672#comment-13247672 ]
Bikas Saha commented on MAPREDUCE-4101:
---------------------------------------
Security is a beast by itself :P
Leaving it as sh lets users setup their installations as long as required stuff is in the PATH. Making it /bin/sh forces installation node to have /bin etc. Some of these things may be difficult, say on Windows.
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268622#comment-13268622 ]
Ahmed Radwan commented on MAPREDUCE-4101:
-----------------------------------------
Sorry for the late reply. I meant using configurations instead of hardcoding. Most of the patch is changing the hardcoded "bash" to "sh". It will be cleaner to have it as a value for a configuration property and have the default set to "sh" for example.
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Allen Wittenauer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249679#comment-13249679 ]
Allen Wittenauer commented on MAPREDUCE-4101:
---------------------------------------------
Given that other parts of the system use ${BASH_SOURCE}, is it actually worth the effort to change this?
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267736#comment-13267736 ]
Radim Kolar commented on MAPREDUCE-4101:
----------------------------------------
i had on plan to rework your patch into trunk version with my improvements as i stated in jira.
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246326#comment-13246326 ]
Hadoop QA commented on MAPREDUCE-4101:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521313/bash-replace-by-sh.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2146//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2146//console
This message is automatically generated.
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247655#comment-13247655 ]
Radim Kolar commented on MAPREDUCE-4101:
----------------------------------------
/bin/sh is always better then plain sh for security purposes
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247903#comment-13247903 ]
Radim Kolar commented on MAPREDUCE-4101:
----------------------------------------
I am not sure if shell script can be launched without #! /abs path.
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247705#comment-13247705 ]
Radim Kolar commented on MAPREDUCE-4101:
----------------------------------------
Nodemanager is creating script with #!/bin/bash, so every node has to have /bin
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267256#comment-13267256 ]
Radim Kolar commented on MAPREDUCE-4101:
----------------------------------------
I will rework this patch but first i need to work on pluggable process tree
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252795#comment-13252795 ]
Ahmed Radwan commented on MAPREDUCE-4101:
-----------------------------------------
It think it is cleaner to avoid hardcoding the commands and have it configurable with the default in *-default.xml.
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Bikas Saha (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245623#comment-13245623 ]
Bikas Saha commented on MAPREDUCE-4101:
---------------------------------------
I agree on using sh.
It would be great if you could post a patch to fix it. Thanks!
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267763#comment-13267763 ]
Bikas Saha commented on MAPREDUCE-4101:
---------------------------------------
I assume you mean to do that when that patch makes its way into trunk from branch-1-win. Otherwise merging branch-1-win into trunk will create a mess.
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267638#comment-13267638 ]
Bikas Saha commented on MAPREDUCE-4101:
---------------------------------------
Do you have a jira for the pluggable process tree? It would sound like a dup of MAPREDUCE-4204.
Can you please wait until it gets merged back into mainline from branch-1-win? And then make the changes you still think are necessary after that refactoring.
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4101) nodemanager depends on /bin/bash
Posted by "Radim Kolar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar updated MAPREDUCE-4101:
-----------------------------------
Labels: patch (was: )
Affects Version/s: trunk
Status: Patch Available (was: Open)
patch is against trunk
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261914#comment-13261914 ]
Radim Kolar commented on MAPREDUCE-4101:
----------------------------------------
You want configurable shell? I think it is overkill.
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on
/bin/bash
Posted by "Bikas Saha (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247526#comment-13247526 ]
Bikas Saha commented on MAPREDUCE-4101:
---------------------------------------
Could you check if the test failure is related to your changes?
Comments
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
@@ -379,7 +379,7 @@ public class TaskLog {
}
}
- private static final String bashCommand = "bash";
+ private static final String bashCommand = "sh";
Now you might want to rename this to shellCommand.
+++ b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
@@ -169,7 +169,7 @@ public class DefaultContainerExecutor extends ContainerExecutor {
ContainerExecutor.TASK_LAUNCH_SCRIPT_PERMISSION);
// Setup command to run
- String[] command = {"bash", "-c",
+ String[] command = {"/bin/sh", "-c",
Why the extra /bin/ ?
> nodemanager depends on /bin/bash
> --------------------------------
>
> Key: MAPREDUCE-4101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.1, trunk
> Environment: FreeBSD 8.2 / 64 bit
> Reporter: Radim Kolar
> Labels: patch
> Attachments: bash-replace-by-sh.txt
>
>
> Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough.
> i found 2 cases:
> 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin)
> 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_000012/default_container_executor.sh]
> this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira