You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by sam liu <sa...@gmail.com> on 2014/08/12 11:18:06 UTC
How to use docker in Hadoop, with patch of YARN-1964?
Hi Experts,
I am very interesting that Hadoop could work with Docker and doing some
trial on patch of YARN-1964.
I applied patch yarn-1964-branch-2.2.0-docker.patch of jira YARN-1964 on
branch 2.2 and am going to install a Hadoop cluster using the new generated
tarball including the patch.
Then, I think I can use DockerContainerExecutor, but I do not know much
details on the usage and have following questions:
*1. After installation, What's the detailed config steps to adopt
DockerContainerExecutor?*
*2. How to verify whether a MR task is really launched in Docker container
not Yarn container?*
*3. Which hadoop branch will officially include Docker support?*
Thanks a lot!
Re: How to use docker in Hadoop, with patch of YARN-1964?
Posted by sam liu <sa...@gmail.com>.
After applying this patch, I added following config in yarn-site.xml:
<property>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor</value>
</property>
Then I can start NodeManager with enabling DockerContainerExecutor. But
failed to execute a simple mr job, and the exception is* 'Cannot connect to
the Docker daemon. Is 'docker -d' running on this host?'*. Below are the
full exception info and any body could give me some hints?
14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 running in
uber mode : false
14/08/12 20:06:14 INFO mapreduce.Job: map 0% reduce 0%
14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 failed
with state FAILED due to: Application application_1407899030909_0002
failed 2 times due to AM Container for appattempt_1407899030909_0002_000002
exited with exitCode: 1 due to: Exception from container-launc h:
org.apache.hadoop.util.Shell$ExitCodeException: Warning: '-rm' is
deprecated, it will be replaced by '--rm' soon. See usage.
Warning: '-name' is deprecated, it will be replaced by '--name' soon. See
usage.
2014/08/12 20:06:13 Cannot connect to the Docker daemon. Is 'docker -d'
running on this host?
at org.apache.hadoop.util.Shell.runCommand(Shell.java:444)
at org.apache.hadoop.util.Shell.run(Shell.java:359)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:569)
at
org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor.launchContainer(DockerContainerExecutor.java:174)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Thanks!
2014-08-12 2:18 GMT-07:00 sam liu <sa...@gmail.com>:
> Hi Experts,
>
> I am very interesting that Hadoop could work with Docker and doing some
> trial on patch of YARN-1964.
>
> I applied patch yarn-1964-branch-2.2.0-docker.patch of jira YARN-1964 on
> branch 2.2 and am going to install a Hadoop cluster using the new generated
> tarball including the patch.
>
> Then, I think I can use DockerContainerExecutor, but I do not know much
> details on the usage and have following questions:
>
>
>
> *1. After installation, What's the detailed config steps to adopt
> DockerContainerExecutor? *
>
>
> *2. How to verify whether a MR task is really launched in Docker container
> not Yarn container?*
> *3. Which hadoop branch will officially include Docker support?*
>
> Thanks a lot!
>
Re: How to use docker in Hadoop, with patch of YARN-1964?
Posted by sam liu <sa...@gmail.com>.
After applying this patch, I added following config in yarn-site.xml:
<property>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor</value>
</property>
Then I can start NodeManager with enabling DockerContainerExecutor. But
failed to execute a simple mr job, and the exception is* 'Cannot connect to
the Docker daemon. Is 'docker -d' running on this host?'*. Below are the
full exception info and any body could give me some hints?
14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 running in
uber mode : false
14/08/12 20:06:14 INFO mapreduce.Job: map 0% reduce 0%
14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 failed
with state FAILED due to: Application application_1407899030909_0002
failed 2 times due to AM Container for appattempt_1407899030909_0002_000002
exited with exitCode: 1 due to: Exception from container-launc h:
org.apache.hadoop.util.Shell$ExitCodeException: Warning: '-rm' is
deprecated, it will be replaced by '--rm' soon. See usage.
Warning: '-name' is deprecated, it will be replaced by '--name' soon. See
usage.
2014/08/12 20:06:13 Cannot connect to the Docker daemon. Is 'docker -d'
running on this host?
at org.apache.hadoop.util.Shell.runCommand(Shell.java:444)
at org.apache.hadoop.util.Shell.run(Shell.java:359)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:569)
at
org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor.launchContainer(DockerContainerExecutor.java:174)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Thanks!
2014-08-12 2:18 GMT-07:00 sam liu <sa...@gmail.com>:
> Hi Experts,
>
> I am very interesting that Hadoop could work with Docker and doing some
> trial on patch of YARN-1964.
>
> I applied patch yarn-1964-branch-2.2.0-docker.patch of jira YARN-1964 on
> branch 2.2 and am going to install a Hadoop cluster using the new generated
> tarball including the patch.
>
> Then, I think I can use DockerContainerExecutor, but I do not know much
> details on the usage and have following questions:
>
>
>
> *1. After installation, What's the detailed config steps to adopt
> DockerContainerExecutor? *
>
>
> *2. How to verify whether a MR task is really launched in Docker container
> not Yarn container?*
> *3. Which hadoop branch will officially include Docker support?*
>
> Thanks a lot!
>
Re: How to use docker in Hadoop, with patch of YARN-1964?
Posted by sam liu <sa...@gmail.com>.
After applying this patch, I added following config in yarn-site.xml:
<property>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor</value>
</property>
Then I can start NodeManager with enabling DockerContainerExecutor. But
failed to execute a simple mr job, and the exception is* 'Cannot connect to
the Docker daemon. Is 'docker -d' running on this host?'*. Below are the
full exception info and any body could give me some hints?
14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 running in
uber mode : false
14/08/12 20:06:14 INFO mapreduce.Job: map 0% reduce 0%
14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 failed
with state FAILED due to: Application application_1407899030909_0002
failed 2 times due to AM Container for appattempt_1407899030909_0002_000002
exited with exitCode: 1 due to: Exception from container-launc h:
org.apache.hadoop.util.Shell$ExitCodeException: Warning: '-rm' is
deprecated, it will be replaced by '--rm' soon. See usage.
Warning: '-name' is deprecated, it will be replaced by '--name' soon. See
usage.
2014/08/12 20:06:13 Cannot connect to the Docker daemon. Is 'docker -d'
running on this host?
at org.apache.hadoop.util.Shell.runCommand(Shell.java:444)
at org.apache.hadoop.util.Shell.run(Shell.java:359)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:569)
at
org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor.launchContainer(DockerContainerExecutor.java:174)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Thanks!
2014-08-12 2:18 GMT-07:00 sam liu <sa...@gmail.com>:
> Hi Experts,
>
> I am very interesting that Hadoop could work with Docker and doing some
> trial on patch of YARN-1964.
>
> I applied patch yarn-1964-branch-2.2.0-docker.patch of jira YARN-1964 on
> branch 2.2 and am going to install a Hadoop cluster using the new generated
> tarball including the patch.
>
> Then, I think I can use DockerContainerExecutor, but I do not know much
> details on the usage and have following questions:
>
>
>
> *1. After installation, What's the detailed config steps to adopt
> DockerContainerExecutor? *
>
>
> *2. How to verify whether a MR task is really launched in Docker container
> not Yarn container?*
> *3. Which hadoop branch will officially include Docker support?*
>
> Thanks a lot!
>
Re: How to use docker in Hadoop, with patch of YARN-1964?
Posted by sam liu <sa...@gmail.com>.
After applying this patch, I added following config in yarn-site.xml:
<property>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor</value>
</property>
Then I can start NodeManager with enabling DockerContainerExecutor. But
failed to execute a simple mr job, and the exception is* 'Cannot connect to
the Docker daemon. Is 'docker -d' running on this host?'*. Below are the
full exception info and any body could give me some hints?
14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 running in
uber mode : false
14/08/12 20:06:14 INFO mapreduce.Job: map 0% reduce 0%
14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 failed
with state FAILED due to: Application application_1407899030909_0002
failed 2 times due to AM Container for appattempt_1407899030909_0002_000002
exited with exitCode: 1 due to: Exception from container-launc h:
org.apache.hadoop.util.Shell$ExitCodeException: Warning: '-rm' is
deprecated, it will be replaced by '--rm' soon. See usage.
Warning: '-name' is deprecated, it will be replaced by '--name' soon. See
usage.
2014/08/12 20:06:13 Cannot connect to the Docker daemon. Is 'docker -d'
running on this host?
at org.apache.hadoop.util.Shell.runCommand(Shell.java:444)
at org.apache.hadoop.util.Shell.run(Shell.java:359)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:569)
at
org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor.launchContainer(DockerContainerExecutor.java:174)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Thanks!
2014-08-12 2:18 GMT-07:00 sam liu <sa...@gmail.com>:
> Hi Experts,
>
> I am very interesting that Hadoop could work with Docker and doing some
> trial on patch of YARN-1964.
>
> I applied patch yarn-1964-branch-2.2.0-docker.patch of jira YARN-1964 on
> branch 2.2 and am going to install a Hadoop cluster using the new generated
> tarball including the patch.
>
> Then, I think I can use DockerContainerExecutor, but I do not know much
> details on the usage and have following questions:
>
>
>
> *1. After installation, What's the detailed config steps to adopt
> DockerContainerExecutor? *
>
>
> *2. How to verify whether a MR task is really launched in Docker container
> not Yarn container?*
> *3. Which hadoop branch will officially include Docker support?*
>
> Thanks a lot!
>