You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Shane Kumpf (JIRA)" <ji...@apache.org> on 2018/07/02 21:59:00 UTC
[jira] [Comment Edited] (YARN-8485) Priviledged container app
launch is failing intermittently
[ https://issues.apache.org/jira/browse/YARN-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530482#comment-16530482 ]
Shane Kumpf edited comment on YARN-8485 at 7/2/18 9:58 PM:
-----------------------------------------------------------
{quote}by checking /usr/bin/sudo is good enough{quote}
I agree this should be enough for now and is the least risky change. We can open a follow on effort to make this configurable if we find an operating system where this is needed. +1 on the latest patch, pending pre-commit.
was (Author: shanekumpf@gmail.com):
{code}by checking /usr/bin/sudo is good enough{code}
I agree this should be enough for now and is the least risky change. We can open a follow on effort to make this configurable if we find an operating system where this is needed. +1 on the latest patch, pending pre-commit.
> Priviledged container app launch is failing intermittently
> ----------------------------------------------------------
>
> Key: YARN-8485
> URL: https://issues.apache.org/jira/browse/YARN-8485
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
> Environment: Debian
> Reporter: Yesha Vora
> Assignee: Eric Yang
> Priority: Major
> Attachments: YARN-8485.001.patch, YARN-8485.002.patch
>
>
> Privileged application fails intermittently
> {code:java}
> yarn jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar -shell_command "sleep 30" -num_containers 1 -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=xxx -shell_env YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=true -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar{code}
> Here, container launch fails with 'Privileged containers are disabled' even though Docker privilege container is enabled in the cluster
> {code:java|title=nm log}
> 2018-06-28 21:21:15,647 INFO runtime.DockerLinuxContainerRuntime (DockerLinuxContainerRuntime.java:allowPrivilegedContainerExecution(664)) - All checks pass. Launching privileged container for : container_e01_1530220647587_0001_01_000002
> 2018-06-28 21:21:15,665 WARN nodemanager.LinuxContainerExecutor (LinuxContainerExecutor.java:handleExitCode(593)) - Exit code from container container_e01_1530220647587_0001_01_000002 is : 29
> 2018-06-28 21:21:15,666 WARN nodemanager.LinuxContainerExecutor (LinuxContainerExecutor.java:handleExitCode(599)) - Exception from container-launch with container ID: container_e01_1530220647587_0001_01_000002 and exit code: 29
> org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:958)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
> at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
> at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:494)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:306)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Exception from container-launch.
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Container id: container_e01_1530220647587_0001_01_000002
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Exit code: 29
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Exception message: Launch container failed
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Shell error output: check privileges failed for user: hrt_qa, error code: 0
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Privileged containers are disabled for user: hrt_qa
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Error constructing docker command, docker error code=11, error message='Privileged containers are disabled'
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) -
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Shell output: main : command provided 4
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - main : run as user is hrt_qa
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - main : requested yarn user is hrt_qa
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Creating script paths...
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Creating local dirs...
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Creating script paths...
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Creating local dirs...
> 2018-06-28 21:21:15,693 WARN launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(598)) - Container launch failed : Container exited with a non-zero exit code 29.
> 2018-06-28 21:21:15,693 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(623)) - Failed to get tail of the container's prelaunch error log file
> java.io.FileNotFoundException: File /grid/0/hadoop/yarn/log/application_1530220647587_0001/container_e01_1530220647587_0001_01_000002/prelaunch.err does not exist
> at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:609)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:575)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:340)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-06-28 21:21:15,704 INFO container.ContainerImpl (ContainerImpl.java:handle(2093)) - Container container_e01_1530220647587_0001_01_000002 transitioned from RUNNING to EXITED_WITH_FAILURE{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org