Task process exit with nonzero status of 255

[Mathias De Maré <ma...@gmail.com>]

I'm getting a rather cryptic error while running a Map job with
MultithreadedMapper (no idea if it has anything to do with the
MultithreadedMapper).
It only occurs sometimes, occurs at different times during the Map
(sometimes at the start, sometimes at a random location), and it doesn't
really give any information.

Task Id : attempt_200908031207_0009_m_000000_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 255.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

I'm a bit at a loss. The only results I'm finding from Google are a few
other people who have had the same problem, but nobody with a solution.

Mathias De Maré

Re: Task process exit with nonzero status of 255

[Mathias De Maré <ma...@gmail.com>]

Thanks! Because of your information, I managed to find out that my crashes
have to do with the somewhat standard 'Too many open files'.
I added some 'close' and 'disconnect' to my InputStreams and
HttpURLConnections, but that only works up to a certain point.
The strange thing is that I do execute 'ulimit -n 32768', but I'm thinking
it somehow 'won't' stick.
The other alternative is that Hadoop is using 32768 file descriptors, but
that seems a bit over the top, as I'm using at most 200 threads (each of
which only sets up one HttpURLConnection).

Suggestions are welcome :-)

On Mon, Aug 3, 2009 at 3:58 PM, Jason Venner <ja...@gmail.com> wrote:

> That generally means that the process that is running the task, crashed.
> The actual map/reduce task is run in a separate jvm by the task tracker,
> and that JVM is exiting abnormally.
> This used to happen to my jobs quite a bit when they were using a buggy
> native library via jni.
>
> If you are trying to use the colorspace transforms via the java imaging
> APIs, it is not thread safe (at least through 1.6.10 under linux).
>
> There may be additional information available in the per task logs.
>
> 2009/8/3 Mathias De Maré <ma...@gmail.com>
>
> I'm getting a rather cryptic error while running a Map job with
>> MultithreadedMapper (no idea if it has anything to do with the
>> MultithreadedMapper).
>> It only occurs sometimes, occurs at different times during the Map
>> (sometimes at the start, sometimes at a random location), and it doesn't
>> really give any information.
>>
>> Task Id : attempt_200908031207_0009_m_000000_0, Status : FAILED
>> java.io.IOException: Task process exit with nonzero status of 255.
>>    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
>>
>> I'm a bit at a loss. The only results I'm finding from Google are a few
>> other people who have had the same problem, but nobody with a solution.
>>
>> Mathias De Maré
>>
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>

Re: Task process exit with nonzero status of 255

[Jason Venner <ja...@gmail.com>]

That generally means that the process that is running the task, crashed.
The actual map/reduce task is run in a separate jvm by the task tracker, and
that JVM is exiting abnormally.
This used to happen to my jobs quite a bit when they were using a buggy
native library via jni.

If you are trying to use the colorspace transforms via the java imaging
APIs, it is not thread safe (at least through 1.6.10 under linux).

There may be additional information available in the per task logs.

2009/8/3 Mathias De Maré <ma...@gmail.com>

> I'm getting a rather cryptic error while running a Map job with
> MultithreadedMapper (no idea if it has anything to do with the
> MultithreadedMapper).
> It only occurs sometimes, occurs at different times during the Map
> (sometimes at the start, sometimes at a random location), and it doesn't
> really give any information.
>
> Task Id : attempt_200908031207_0009_m_000000_0, Status : FAILED
> java.io.IOException: Task process exit with nonzero status of 255.
>    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
>
> I'm a bit at a loss. The only results I'm finding from Google are a few
> other people who have had the same problem, but nobody with a solution.
>
> Mathias De Maré
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals