You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Yves Weissig <we...@uni-mainz.de> on 2014/03/04 16:24:33 UTC
Meaning of messages in log and debugging
Hello list,
I'm currently debugging my Hadoop MR application and I have some general
questions to the messages in the log and the debugging process.
- What does "Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143" mean? What does 143 stand
for?
- I also see the following exception in the log: "Exception from
container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)". What does this mean? It
originates from a "Diagnostics report" from a container and the log4j
message level is set to INFO.
- Are there any related links which describe the life cycle of a container?
- Is there a "golden rule" to debug a Hadoop MR application?
- My application is very memory intense... is there any way to profile
the memory consumption of a single container?
Thanks!
Best regards
Yves
Re: Meaning of messages in log and debugging
Posted by Zhijie Shen <zs...@hortonworks.com>.
bq. Container killed by the ApplicationMaster. Container killed on request.
Exit code is 143" mean? What does 143 stand for?
It's the diagnostic message generated by YARN, which indicates the
container is killed by MR's ApplicationMaster. 143 is a exit code of an
YARN container, which indicates the termination of a container.
bq. Are there any related links which describe the life cycle of a
container?
This is what I found online:
http://diggerk.wordpress.com/2013/09/19/lifecycle-of-yarn-resource-manager-containers/.
Otherwise, you can have a look at ContainerImpl.java if you want to know
the detail.
bq. My application is very memory intense... is there any way to profile the
memory consumption of a single container?
You can find the metrics info RM and NM web UI, or you
can programmatically access the RESTful APIs.
- Zhijie
On Tue, Mar 4, 2014 at 7:24 AM, Yves Weissig <we...@uni-mainz.de> wrote:
> Hello list,
>
> I'm currently debugging my Hadoop MR application and I have some general
> questions to the messages in the log and the debugging process.
>
> - What does "Container killed by the ApplicationMaster.
> Container killed on request. Exit code is 143" mean? What does 143 stand
> for?
>
> - I also see the following exception in the log: "Exception from
> container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)". What does this mean? It
> originates from a "Diagnostics report" from a container and the log4j
> message level is set to INFO.
>
> - Are there any related links which describe the life cycle of a container?
>
> - Is there a "golden rule" to debug a Hadoop MR application?
>
> - My application is very memory intense... is there any way to profile
> the memory consumption of a single container?
>
> Thanks!
> Best regards
> Yves
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Meaning of messages in log and debugging
Posted by Zhijie Shen <zs...@hortonworks.com>.
bq. Container killed by the ApplicationMaster. Container killed on request.
Exit code is 143" mean? What does 143 stand for?
It's the diagnostic message generated by YARN, which indicates the
container is killed by MR's ApplicationMaster. 143 is a exit code of an
YARN container, which indicates the termination of a container.
bq. Are there any related links which describe the life cycle of a
container?
This is what I found online:
http://diggerk.wordpress.com/2013/09/19/lifecycle-of-yarn-resource-manager-containers/.
Otherwise, you can have a look at ContainerImpl.java if you want to know
the detail.
bq. My application is very memory intense... is there any way to profile the
memory consumption of a single container?
You can find the metrics info RM and NM web UI, or you
can programmatically access the RESTful APIs.
- Zhijie
On Tue, Mar 4, 2014 at 7:24 AM, Yves Weissig <we...@uni-mainz.de> wrote:
> Hello list,
>
> I'm currently debugging my Hadoop MR application and I have some general
> questions to the messages in the log and the debugging process.
>
> - What does "Container killed by the ApplicationMaster.
> Container killed on request. Exit code is 143" mean? What does 143 stand
> for?
>
> - I also see the following exception in the log: "Exception from
> container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)". What does this mean? It
> originates from a "Diagnostics report" from a container and the log4j
> message level is set to INFO.
>
> - Are there any related links which describe the life cycle of a container?
>
> - Is there a "golden rule" to debug a Hadoop MR application?
>
> - My application is very memory intense... is there any way to profile
> the memory consumption of a single container?
>
> Thanks!
> Best regards
> Yves
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Meaning of messages in log and debugging
Posted by Zhijie Shen <zs...@hortonworks.com>.
bq. Container killed by the ApplicationMaster. Container killed on request.
Exit code is 143" mean? What does 143 stand for?
It's the diagnostic message generated by YARN, which indicates the
container is killed by MR's ApplicationMaster. 143 is a exit code of an
YARN container, which indicates the termination of a container.
bq. Are there any related links which describe the life cycle of a
container?
This is what I found online:
http://diggerk.wordpress.com/2013/09/19/lifecycle-of-yarn-resource-manager-containers/.
Otherwise, you can have a look at ContainerImpl.java if you want to know
the detail.
bq. My application is very memory intense... is there any way to profile the
memory consumption of a single container?
You can find the metrics info RM and NM web UI, or you
can programmatically access the RESTful APIs.
- Zhijie
On Tue, Mar 4, 2014 at 7:24 AM, Yves Weissig <we...@uni-mainz.de> wrote:
> Hello list,
>
> I'm currently debugging my Hadoop MR application and I have some general
> questions to the messages in the log and the debugging process.
>
> - What does "Container killed by the ApplicationMaster.
> Container killed on request. Exit code is 143" mean? What does 143 stand
> for?
>
> - I also see the following exception in the log: "Exception from
> container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)". What does this mean? It
> originates from a "Diagnostics report" from a container and the log4j
> message level is set to INFO.
>
> - Are there any related links which describe the life cycle of a container?
>
> - Is there a "golden rule" to debug a Hadoop MR application?
>
> - My application is very memory intense... is there any way to profile
> the memory consumption of a single container?
>
> Thanks!
> Best regards
> Yves
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Meaning of messages in log and debugging
Posted by Zhijie Shen <zs...@hortonworks.com>.
bq. Container killed by the ApplicationMaster. Container killed on request.
Exit code is 143" mean? What does 143 stand for?
It's the diagnostic message generated by YARN, which indicates the
container is killed by MR's ApplicationMaster. 143 is a exit code of an
YARN container, which indicates the termination of a container.
bq. Are there any related links which describe the life cycle of a
container?
This is what I found online:
http://diggerk.wordpress.com/2013/09/19/lifecycle-of-yarn-resource-manager-containers/.
Otherwise, you can have a look at ContainerImpl.java if you want to know
the detail.
bq. My application is very memory intense... is there any way to profile the
memory consumption of a single container?
You can find the metrics info RM and NM web UI, or you
can programmatically access the RESTful APIs.
- Zhijie
On Tue, Mar 4, 2014 at 7:24 AM, Yves Weissig <we...@uni-mainz.de> wrote:
> Hello list,
>
> I'm currently debugging my Hadoop MR application and I have some general
> questions to the messages in the log and the debugging process.
>
> - What does "Container killed by the ApplicationMaster.
> Container killed on request. Exit code is 143" mean? What does 143 stand
> for?
>
> - I also see the following exception in the log: "Exception from
> container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)". What does this mean? It
> originates from a "Diagnostics report" from a container and the log4j
> message level is set to INFO.
>
> - Are there any related links which describe the life cycle of a container?
>
> - Is there a "golden rule" to debug a Hadoop MR application?
>
> - My application is very memory intense... is there any way to profile
> the memory consumption of a single container?
>
> Thanks!
> Best regards
> Yves
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.