You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Nitin Mathur <nt...@gmail.com> on 2015/04/17 00:11:12 UTC
Container exited with a non-zero exit code 134
Hi All,
Production job failing with Container exited with a non-zero exit code 134.
Any ideas what could be causing this as I am not able to see anything
straight forward here.
- Nitin
Re: Container exited with a non-zero exit code 134
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hello Nitin,
When I see an exit code greater than 128, then that makes me think the process didn't exit normally and instead it was terminated unexpectedly by a signal. On most of the systems I've used, if a process gets killed, then the reported exit status is 128 + the signal number. 134 - 128 = 6, which is SIGABRT. That would indicate something very unexpected happened inside the process (i.e. memory corruption) and it chose to call the abort function.
You might try looking for other evidence of abnormal program termination. If this was a MapReduce job or some other kind of Java process, then you could look for hs_err_pid files that show the state of the JVM before the crash. If the container was running native code, then you might see a core dump.
Hope this helps.
Chris Nauroth
Hortonworks
http://hortonworks.com/
From: Nitin Mathur <nt...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, April 16, 2015 at 3:11 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Container exited with a non-zero exit code 134
Hi All,
Production job failing with Container exited with a non-zero exit code 134. Any ideas what could be causing this as I am not able to see anything straight forward here.
- Nitin
Re: Container exited with a non-zero exit code 134
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hello Nitin,
When I see an exit code greater than 128, then that makes me think the process didn't exit normally and instead it was terminated unexpectedly by a signal. On most of the systems I've used, if a process gets killed, then the reported exit status is 128 + the signal number. 134 - 128 = 6, which is SIGABRT. That would indicate something very unexpected happened inside the process (i.e. memory corruption) and it chose to call the abort function.
You might try looking for other evidence of abnormal program termination. If this was a MapReduce job or some other kind of Java process, then you could look for hs_err_pid files that show the state of the JVM before the crash. If the container was running native code, then you might see a core dump.
Hope this helps.
Chris Nauroth
Hortonworks
http://hortonworks.com/
From: Nitin Mathur <nt...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, April 16, 2015 at 3:11 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Container exited with a non-zero exit code 134
Hi All,
Production job failing with Container exited with a non-zero exit code 134. Any ideas what could be causing this as I am not able to see anything straight forward here.
- Nitin
Re: Container exited with a non-zero exit code 134
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hello Nitin,
When I see an exit code greater than 128, then that makes me think the process didn't exit normally and instead it was terminated unexpectedly by a signal. On most of the systems I've used, if a process gets killed, then the reported exit status is 128 + the signal number. 134 - 128 = 6, which is SIGABRT. That would indicate something very unexpected happened inside the process (i.e. memory corruption) and it chose to call the abort function.
You might try looking for other evidence of abnormal program termination. If this was a MapReduce job or some other kind of Java process, then you could look for hs_err_pid files that show the state of the JVM before the crash. If the container was running native code, then you might see a core dump.
Hope this helps.
Chris Nauroth
Hortonworks
http://hortonworks.com/
From: Nitin Mathur <nt...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, April 16, 2015 at 3:11 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Container exited with a non-zero exit code 134
Hi All,
Production job failing with Container exited with a non-zero exit code 134. Any ideas what could be causing this as I am not able to see anything straight forward here.
- Nitin
Re: Container exited with a non-zero exit code 134
Posted by Ted Yu <yu...@gmail.com>.
Can you provide a bit more information please (such as hadoop release) ?
Using the following command you would get more clue on the cause (plug in
your app Id and container Id):
yarn logs -applicationId application_1386639398517_0007 -containerId
container_1386639398517_0007_01_000019
Cheers
On Thu, Apr 16, 2015 at 3:11 PM, Nitin Mathur <nt...@gmail.com> wrote:
> Hi All,
>
> Production job failing with Container exited with a non-zero exit code
> 134. Any ideas what could be causing this as I am not able to see anything
> straight forward here.
>
>
> - Nitin
>
>
>
Re: Container exited with a non-zero exit code 134
Posted by Ted Yu <yu...@gmail.com>.
Can you provide a bit more information please (such as hadoop release) ?
Using the following command you would get more clue on the cause (plug in
your app Id and container Id):
yarn logs -applicationId application_1386639398517_0007 -containerId
container_1386639398517_0007_01_000019
Cheers
On Thu, Apr 16, 2015 at 3:11 PM, Nitin Mathur <nt...@gmail.com> wrote:
> Hi All,
>
> Production job failing with Container exited with a non-zero exit code
> 134. Any ideas what could be causing this as I am not able to see anything
> straight forward here.
>
>
> - Nitin
>
>
>
Re: Container exited with a non-zero exit code 134
Posted by Ted Yu <yu...@gmail.com>.
Can you provide a bit more information please (such as hadoop release) ?
Using the following command you would get more clue on the cause (plug in
your app Id and container Id):
yarn logs -applicationId application_1386639398517_0007 -containerId
container_1386639398517_0007_01_000019
Cheers
On Thu, Apr 16, 2015 at 3:11 PM, Nitin Mathur <nt...@gmail.com> wrote:
> Hi All,
>
> Production job failing with Container exited with a non-zero exit code
> 134. Any ideas what could be causing this as I am not able to see anything
> straight forward here.
>
>
> - Nitin
>
>
>
Re: Container exited with a non-zero exit code 134
Posted by Ted Yu <yu...@gmail.com>.
Can you provide a bit more information please (such as hadoop release) ?
Using the following command you would get more clue on the cause (plug in
your app Id and container Id):
yarn logs -applicationId application_1386639398517_0007 -containerId
container_1386639398517_0007_01_000019
Cheers
On Thu, Apr 16, 2015 at 3:11 PM, Nitin Mathur <nt...@gmail.com> wrote:
> Hi All,
>
> Production job failing with Container exited with a non-zero exit code
> 134. Any ideas what could be causing this as I am not able to see anything
> straight forward here.
>
>
> - Nitin
>
>
>
Re: Container exited with a non-zero exit code 134
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hello Nitin,
When I see an exit code greater than 128, then that makes me think the process didn't exit normally and instead it was terminated unexpectedly by a signal. On most of the systems I've used, if a process gets killed, then the reported exit status is 128 + the signal number. 134 - 128 = 6, which is SIGABRT. That would indicate something very unexpected happened inside the process (i.e. memory corruption) and it chose to call the abort function.
You might try looking for other evidence of abnormal program termination. If this was a MapReduce job or some other kind of Java process, then you could look for hs_err_pid files that show the state of the JVM before the crash. If the container was running native code, then you might see a core dump.
Hope this helps.
Chris Nauroth
Hortonworks
http://hortonworks.com/
From: Nitin Mathur <nt...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, April 16, 2015 at 3:11 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Container exited with a non-zero exit code 134
Hi All,
Production job failing with Container exited with a non-zero exit code 134. Any ideas what could be causing this as I am not able to see anything straight forward here.
- Nitin