You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jian Fang <ji...@gmail.com> on 2013/08/30 19:03:23 UTC
"bad interpreter: Text file busy" and other errors in Hadoop 2.1.0-beta
Hi,
I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
messages as follows.
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException: bash:
/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh:
/bin/bash: bad interpreter: Text file busy
at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
at org.apache.hadoop.util.Shell.run(Shell.java:373)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
cleanup failed for container container_1377823589199_0002_01_000214 :
org.apache.hadoop.yarn.exceptions.YarnException: Container
container_1377823589199_0002_01_000214 is not handled by this NodeManager
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Any thing wrong here?
Thanks,
John
Re: "bad interpreter: Text file busy" and other errors in Hadoop 2.1.0-beta
Posted by Jian Fang <ji...@gmail.com>.
For the first error, looked like one thread/process was writing the
following shell file,
/var/lib/hadoop/tmp/nm-local-
dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh
but another thread/process was trying to execute it at the same time, which
was why the error "bad interpreter: Text file busy" was thrown.
Is there any possibility that 2.1.0-beta introduced some race condition
here?
I will look more into the second error.
Thank,
John
On Sat, Aug 31, 2013 at 12:42 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi John
>
> This exception should indicate error from the container process. If the
> container process exits with non-zero exit code, it will be logged.
> In case of such errors, you'd better look at the per-container log see
> what's happening there.
>
> Jian
>
>
> On Fri, Aug 30, 2013 at 10:03 AM, Jian Fang <jian.fang.subscribe@gmail.com
> > wrote:
>
>> Hi,
>>
>> I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
>> messages as follows.
>>
>> Exception from container-launch:
>> org.apache.hadoop.util.Shell$ExitCodeException: bash:
>> /var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh:
>> /bin/bash: bad interpreter: Text file busy
>>
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
>> at org.apache.hadoop.util.Shell.run(Shell.java:373)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>> cleanup failed for container container_1377823589199_0002_01_000214 :
>> org.apache.hadoop.yarn.exceptions.YarnException: Container
>> container_1377823589199_0002_01_000214 is not handled by this NodeManager
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> at
>> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
>> at
>> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
>> at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210)
>> at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Any thing wrong here?
>>
>> Thanks,
>>
>> John
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: "bad interpreter: Text file busy" and other errors in Hadoop 2.1.0-beta
Posted by Jian Fang <ji...@gmail.com>.
For the first error, looked like one thread/process was writing the
following shell file,
/var/lib/hadoop/tmp/nm-local-
dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh
but another thread/process was trying to execute it at the same time, which
was why the error "bad interpreter: Text file busy" was thrown.
Is there any possibility that 2.1.0-beta introduced some race condition
here?
I will look more into the second error.
Thank,
John
On Sat, Aug 31, 2013 at 12:42 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi John
>
> This exception should indicate error from the container process. If the
> container process exits with non-zero exit code, it will be logged.
> In case of such errors, you'd better look at the per-container log see
> what's happening there.
>
> Jian
>
>
> On Fri, Aug 30, 2013 at 10:03 AM, Jian Fang <jian.fang.subscribe@gmail.com
> > wrote:
>
>> Hi,
>>
>> I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
>> messages as follows.
>>
>> Exception from container-launch:
>> org.apache.hadoop.util.Shell$ExitCodeException: bash:
>> /var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh:
>> /bin/bash: bad interpreter: Text file busy
>>
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
>> at org.apache.hadoop.util.Shell.run(Shell.java:373)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>> cleanup failed for container container_1377823589199_0002_01_000214 :
>> org.apache.hadoop.yarn.exceptions.YarnException: Container
>> container_1377823589199_0002_01_000214 is not handled by this NodeManager
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> at
>> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
>> at
>> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
>> at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210)
>> at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Any thing wrong here?
>>
>> Thanks,
>>
>> John
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: "bad interpreter: Text file busy" and other errors in Hadoop 2.1.0-beta
Posted by Jian Fang <ji...@gmail.com>.
For the first error, looked like one thread/process was writing the
following shell file,
/var/lib/hadoop/tmp/nm-local-
dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh
but another thread/process was trying to execute it at the same time, which
was why the error "bad interpreter: Text file busy" was thrown.
Is there any possibility that 2.1.0-beta introduced some race condition
here?
I will look more into the second error.
Thank,
John
On Sat, Aug 31, 2013 at 12:42 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi John
>
> This exception should indicate error from the container process. If the
> container process exits with non-zero exit code, it will be logged.
> In case of such errors, you'd better look at the per-container log see
> what's happening there.
>
> Jian
>
>
> On Fri, Aug 30, 2013 at 10:03 AM, Jian Fang <jian.fang.subscribe@gmail.com
> > wrote:
>
>> Hi,
>>
>> I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
>> messages as follows.
>>
>> Exception from container-launch:
>> org.apache.hadoop.util.Shell$ExitCodeException: bash:
>> /var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh:
>> /bin/bash: bad interpreter: Text file busy
>>
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
>> at org.apache.hadoop.util.Shell.run(Shell.java:373)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>> cleanup failed for container container_1377823589199_0002_01_000214 :
>> org.apache.hadoop.yarn.exceptions.YarnException: Container
>> container_1377823589199_0002_01_000214 is not handled by this NodeManager
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> at
>> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
>> at
>> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
>> at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210)
>> at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Any thing wrong here?
>>
>> Thanks,
>>
>> John
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: "bad interpreter: Text file busy" and other errors in Hadoop 2.1.0-beta
Posted by Jian Fang <ji...@gmail.com>.
For the first error, looked like one thread/process was writing the
following shell file,
/var/lib/hadoop/tmp/nm-local-
dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh
but another thread/process was trying to execute it at the same time, which
was why the error "bad interpreter: Text file busy" was thrown.
Is there any possibility that 2.1.0-beta introduced some race condition
here?
I will look more into the second error.
Thank,
John
On Sat, Aug 31, 2013 at 12:42 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi John
>
> This exception should indicate error from the container process. If the
> container process exits with non-zero exit code, it will be logged.
> In case of such errors, you'd better look at the per-container log see
> what's happening there.
>
> Jian
>
>
> On Fri, Aug 30, 2013 at 10:03 AM, Jian Fang <jian.fang.subscribe@gmail.com
> > wrote:
>
>> Hi,
>>
>> I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
>> messages as follows.
>>
>> Exception from container-launch:
>> org.apache.hadoop.util.Shell$ExitCodeException: bash:
>> /var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh:
>> /bin/bash: bad interpreter: Text file busy
>>
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
>> at org.apache.hadoop.util.Shell.run(Shell.java:373)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>> cleanup failed for container container_1377823589199_0002_01_000214 :
>> org.apache.hadoop.yarn.exceptions.YarnException: Container
>> container_1377823589199_0002_01_000214 is not handled by this NodeManager
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> at
>> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
>> at
>> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
>> at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210)
>> at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Any thing wrong here?
>>
>> Thanks,
>>
>> John
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: "bad interpreter: Text file busy" and other errors in Hadoop 2.1.0-beta
Posted by Jian He <jh...@hortonworks.com>.
Hi John
This exception should indicate error from the container process. If the
container process exits with non-zero exit code, it will be logged.
In case of such errors, you'd better look at the per-container log see
what's happening there.
Jian
On Fri, Aug 30, 2013 at 10:03 AM, Jian Fang
<ji...@gmail.com>wrote:
> Hi,
>
> I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
> messages as follows.
>
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: bash:
> /var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh:
> /bin/bash: bad interpreter: Text file busy
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
> at org.apache.hadoop.util.Shell.run(Shell.java:373)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
>
>
> cleanup failed for container container_1377823589199_0002_01_000214 :
> org.apache.hadoop.yarn.exceptions.YarnException: Container
> container_1377823589199_0002_01_000214 is not handled by this NodeManager
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
> Any thing wrong here?
>
> Thanks,
>
> John
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: "bad interpreter: Text file busy" and other errors in Hadoop 2.1.0-beta
Posted by Jian He <jh...@hortonworks.com>.
Hi John
This exception should indicate error from the container process. If the
container process exits with non-zero exit code, it will be logged.
In case of such errors, you'd better look at the per-container log see
what's happening there.
Jian
On Fri, Aug 30, 2013 at 10:03 AM, Jian Fang
<ji...@gmail.com>wrote:
> Hi,
>
> I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
> messages as follows.
>
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: bash:
> /var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh:
> /bin/bash: bad interpreter: Text file busy
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
> at org.apache.hadoop.util.Shell.run(Shell.java:373)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
>
>
> cleanup failed for container container_1377823589199_0002_01_000214 :
> org.apache.hadoop.yarn.exceptions.YarnException: Container
> container_1377823589199_0002_01_000214 is not handled by this NodeManager
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
> Any thing wrong here?
>
> Thanks,
>
> John
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: "bad interpreter: Text file busy" and other errors in Hadoop 2.1.0-beta
Posted by Jian He <jh...@hortonworks.com>.
Hi John
This exception should indicate error from the container process. If the
container process exits with non-zero exit code, it will be logged.
In case of such errors, you'd better look at the per-container log see
what's happening there.
Jian
On Fri, Aug 30, 2013 at 10:03 AM, Jian Fang
<ji...@gmail.com>wrote:
> Hi,
>
> I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
> messages as follows.
>
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: bash:
> /var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh:
> /bin/bash: bad interpreter: Text file busy
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
> at org.apache.hadoop.util.Shell.run(Shell.java:373)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
>
>
> cleanup failed for container container_1377823589199_0002_01_000214 :
> org.apache.hadoop.yarn.exceptions.YarnException: Container
> container_1377823589199_0002_01_000214 is not handled by this NodeManager
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
> Any thing wrong here?
>
> Thanks,
>
> John
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: "bad interpreter: Text file busy" and other errors in Hadoop 2.1.0-beta
Posted by Jian He <jh...@hortonworks.com>.
Hi John
This exception should indicate error from the container process. If the
container process exits with non-zero exit code, it will be logged.
In case of such errors, you'd better look at the per-container log see
what's happening there.
Jian
On Fri, Aug 30, 2013 at 10:03 AM, Jian Fang
<ji...@gmail.com>wrote:
> Hi,
>
> I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
> messages as follows.
>
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: bash:
> /var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh:
> /bin/bash: bad interpreter: Text file busy
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
> at org.apache.hadoop.util.Shell.run(Shell.java:373)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
>
>
> cleanup failed for container container_1377823589199_0002_01_000214 :
> org.apache.hadoop.yarn.exceptions.YarnException: Container
> container_1377823589199_0002_01_000214 is not handled by this NodeManager
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
> Any thing wrong here?
>
> Thanks,
>
> John
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.