You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Hitesh Shah (Commented) (JIRA)" <ji...@apache.org> on 2011/10/07 03:42:29 UTC
[jira] [Commented] (MAPREDUCE-3123) Symbolic links with special
chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122483#comment-13122483 ]
Hitesh Shah commented on MAPREDUCE-3123:
----------------------------------------
Vinod/Thomas/Arun - a query on task.sh - the final command i.e. "exec setsid /bin/bash -c ..." encapsulates the user provided command with "". The assumption here seems to be they can use environment variables with the command for expansion and that the onus of correctly escaping the command i.e escaping any special chars is on the users themselves. Would that be correct?
For example, creating the symlink above is pretty simply by just quoting the file name but if the symlink is something that will be executed then special chars like "&" or even basic things like spaces can create problems if not quoted and escaped properly.
> Symbolic links with special chars causing container/task.sh to fail
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3123
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
>
> the following job throws an exception when you have the special characters in it.
> hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* -Dmapreduce.job.queuename=queue1 -files file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat -output Streaming/streaming-980/Output -jobconf mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=*
> Exception:
> 2011-09-27 20:58:48,903 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: launchContainer:
> [container-executor, hadoopuser, 1, application_1317077272567_0239,
> container_1317077272567 0239_01_000001,
> tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_000001,
> tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 000001/task.sh,
> tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_000001/container_1317077272567_0239_01_000001.tokens]1109221111-tests.jar:hadoop-mapreduce-p2011-09-27
> 20:58:48,944 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container is : 2
> 2011-09-27
> 20:58:48,946 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception from container-launch :
>
> org.apache.hadoop.util.Shell$ExitCodeException:
> /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_000001/task.sh:
> line 26: syntax error near unexpected token `-_+='
> /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_000001/task.sh:
> line 26: `ln -sf /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir test
> ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:261)
>
> at org.apache.hadoop.util.Shell.run(Shell.java:188)
>
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>
> at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174)
>
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197)
>
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-27 20:58:48,951 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
> 2011-09-27 20:58:48,951 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Processing
> container_1317077272567_0239_01_000001 of type UPDATE_DIAGNOSTICS_MSG
> 2011-09-27 20:58:48,951 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Container exited with a non-zero exit code 2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira