Weird 'java.io.IOException: Task process exit with nonzero status of 134' problem

[tony_l <hl...@aol.com>]

we are running some hadoop map only job where each mapper takes 1-2 G of
memory at configuration time. When running on EC2 large instance, we could
run two mappers per node in parallel. The problem is that each mapper can
only work well the first time it is configured/initialized. If a job
attempts to run a mapper for a second time due to multiple input files or
one large input file is split into multiple chucks, we run into this 134
error from time to time. This problem only persists on 64bit JVM. When we
move to 32bit platforms, this issue is gone. In both cases, we use most
recent SUN JAVA 1.6 on centos.

Anybody knows what's wrong?
-- 
View this message in context: http://www.nabble.com/Weird-%27java.io.IOException%3A-Task-process-exit-with-nonzero-status-of-134%27-problem-tp25107532p25107532.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.

Re: Weird 'java.io.IOException: Task process exit with nonzero status of 134' problem

[indoos <in...@gmail.com>]

Hi,
Todd, right on target!!

Tony, heap usage would be at least 30% extra in 64 bit as compared to 32
bit.
Increasing the SWAP size might help in bypassing the out of memory error,
but does impact processing speed.

Todd is referring using -XX:+UseCompressedOops option. You might have to
find if your JVM version suports it, else you might have to upgrade.
Here is some more information about it at
http://blog.juma.me.uk/2008/10/14/32-bit-or-64-bit-jvm-how-about-a-hybrid/     
Another useful option may be looking at the garbage collection tuning along
with compressed opt option.

-Sanjay


Todd Lipcon-4 wrote:
> 
> Hi Tony,
> Signal 134 usually means the JVM crashed hard. So, you're looking at
> either
> a JVM bug or simply an OutOfMemory situation. There are two possibilities
> that might explain why you see the issue on the 64-bit JVM and not the
> 32-bit:
> 
> 1) There could be a bug present in the 64-bit but not the 32-bit. Are you
> running the same exact Java release or is your 32-bit possibly newer?
> 
> 2) The 64-bit JVM will take more heap size for the same program than the
> 32-bit. This is due to the extra overhead of object references in a 64-bit
> heap. There's a "Compressed Object Pointers" option recently introduced
> that
> can reduce this overhead, but it's not enabled by default as of yet.
> 
> You should be able to look at the stderr output of the tasks that fail to
> deduce what's going on.
> 
> -Todd
> 
> On Sun, Aug 23, 2009 at 2:20 PM, tony_l <hl...@aol.com> wrote:
> 
>>
>> we are running some hadoop map only job where each mapper takes 1-2 G of
>> memory at configuration time. When running on EC2 large instance, we
>> could
>> run two mappers per node in parallel. The problem is that each mapper can
>> only work well the first time it is configured/initialized. If a job
>> attempts to run a mapper for a second time due to multiple input files or
>> one large input file is split into multiple chucks, we run into this 134
>> error from time to time. This problem only persists on 64bit JVM. When we
>> move to 32bit platforms, this issue is gone. In both cases, we use most
>> recent SUN JAVA 1.6 on centos.
>>
>> Anybody knows what's wrong?
>> --
>> View this message in context:
>> http://www.nabble.com/Weird-%27java.io.IOException%3A-Task-process-exit-with-nonzero-status-of-134%27-problem-tp25107532p25107532.html
>> Sent from the Hadoop core-dev mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/Weird-%27java.io.IOException%3A-Task-process-exit-with-nonzero-status-of-134%27-problem-tp25107532p25199621.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.

Re: Weird 'java.io.IOException: Task process exit with nonzero status of 134' problem

[Todd Lipcon <to...@cloudera.com>]

Hi Tony,
Signal 134 usually means the JVM crashed hard. So, you're looking at either
a JVM bug or simply an OutOfMemory situation. There are two possibilities
that might explain why you see the issue on the 64-bit JVM and not the
32-bit:

1) There could be a bug present in the 64-bit but not the 32-bit. Are you
running the same exact Java release or is your 32-bit possibly newer?

2) The 64-bit JVM will take more heap size for the same program than the
32-bit. This is due to the extra overhead of object references in a 64-bit
heap. There's a "Compressed Object Pointers" option recently introduced that
can reduce this overhead, but it's not enabled by default as of yet.

You should be able to look at the stderr output of the tasks that fail to
deduce what's going on.

-Todd

On Sun, Aug 23, 2009 at 2:20 PM, tony_l <hl...@aol.com> wrote:

>
> we are running some hadoop map only job where each mapper takes 1-2 G of
> memory at configuration time. When running on EC2 large instance, we could
> run two mappers per node in parallel. The problem is that each mapper can
> only work well the first time it is configured/initialized. If a job
> attempts to run a mapper for a second time due to multiple input files or
> one large input file is split into multiple chucks, we run into this 134
> error from time to time. This problem only persists on 64bit JVM. When we
> move to 32bit platforms, this issue is gone. In both cases, we use most
> recent SUN JAVA 1.6 on centos.
>
> Anybody knows what's wrong?
> --
> View this message in context:
> http://www.nabble.com/Weird-%27java.io.IOException%3A-Task-process-exit-with-nonzero-status-of-134%27-problem-tp25107532p25107532.html
> Sent from the Hadoop core-dev mailing list archive at Nabble.com.
>
>