Re: "bad interpreter: Text file busy" and other errors in Hadoop 2.1.0-beta

[Jian Fang <ji...@gmail.com>]

 For the first error, looked like one thread/process was writing the
following shell file,
/var/lib/hadoop/tmp/nm-local-
dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh

but another thread/process was trying to execute it at the same time, which
was why the error "bad interpreter: Text file busy" was thrown.

Is there any possibility that 2.1.0-beta introduced some race condition
here?

I will look more into the second error.

Thank,

John


On Sat, Aug 31, 2013 at 12:42 PM, Jian He <jh...@hortonworks.com> wrote:

> Hi John
>
> This exception should indicate error from the container process. If the
> container process exits with non-zero exit code, it will be logged.
> In case of such errors, you'd better look at the per-container log see
> what's happening there.
>
> Jian
>
>
> On Fri, Aug 30, 2013 at 10:03 AM, Jian Fang <jian.fang.subscribe@gmail.com
> > wrote:
>
>> Hi,
>>
>> I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
>> messages as follows.
>>
>> Exception from container-launch:
>> org.apache.hadoop.util.Shell$ExitCodeException: bash:
>> /var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh:
>> /bin/bash: bad interpreter: Text file busy
>>
>>         at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
>>         at org.apache.hadoop.util.Shell.run(Shell.java:373)
>>         at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
>>         at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>         at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
>>         at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
>>         at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>> cleanup failed for container container_1377823589199_0002_01_000214 :
>> org.apache.hadoop.yarn.exceptions.YarnException: Container
>> container_1377823589199_0002_01_000214 is not handled by this NodeManager
>>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>>         at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>         at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>         at
>> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
>>         at
>> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
>>         at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210)
>>         at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>> Any thing wrong here?
>>
>> Thanks,
>>
>> John
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.