Mapper OOMs disappear after disabling JVM reuse

[Steven Wong <sw...@netflix.com>]

When the following query was run with mapred.job.reuse.jvm.num.tasks=20, some of the map tasks failed with "Error: Java heap space", causing the job to fail. After changing to mapred.job.reuse.jvm.num.tasks=1, the job succeeded.

FROM (
    FROM intable1
    SELECT acct_id, esn) b
JOIN (
    FROM intable2
    SELECT acct_id, xid, devtype_id, esn, other_properties['cdndldist'] AS cdndldist
    WHERE (dateint>=20110201 AND dateint<=20110228)
    AND (other_properties['cdndldist'] is not null)
    AND (client_msg_type='endplay')
    AND (devtype_id=272 OR devtype_id=129 OR devtype_id=12 OR devtype_id=13 OR devtype_id=14)) c
ON (b.acct_id=c.acct_id) AND (b.esn=c.esn)
INSERT OVERWRITE TABLE outtable PARTITION (dateint='20110201-20110228')
SELECT b.acct_id, c.xid, c.devtype_id, c.esn, c.cdndldist;

I'm on a pre-release version of Hive 0.7, with Hadoop 0.20. Does anyone know about any Hive/Hadoop issue that may be related to this?

Thanks.
Steven

RE: Mapper OOMs disappear after disabling JVM reuse

[Steven Wong <sw...@netflix.com>]

A likely or possible cause is memory leaks. Another possibility may be heap fragmentation. The OOM happened when the mapper was trying to allocate the buffer whose size is controlled by io.sort.mb, and my setup has io.sort.mb=500 and -Xmx1000m.


From: Igor Tatarinov [mailto:igor@decide.com]
Sent: Friday, April 08, 2011 9:50 AM
To: user@hive.apache.org
Cc: Steven Wong
Subject: Re: Mapper OOMs disappear after disabling JVM reuse

I had a similar problem until I set this parameter to 1 (although 3 seems to work fine too).

There is an explanation somewhere on the web. Basically, if you run 20 tasks and the garbage collector cannot catch up with accumulated garbage, the java process grows too big so when it finally decides to fork a new process, the forked process and the original one take up too much memory. Or something like that.

I imagine the same kind thing would still happen even when setting reuse to 1 if a task happens to consume a lot of memory and CPU cycles, so something else must be happening here. Perhaps, there is a memory 'leak' and some 'garbage' never gets collected between task executions.


On Thu, Apr 7, 2011 at 6:53 PM, Steven Wong <sw...@netflix.com>> wrote:
When the following query was run with mapred.job.reuse.jvm.num.tasks=20, some of the map tasks failed with "Error: Java heap space", causing the job to fail. After changing to mapred.job.reuse.jvm.num.tasks=1, the job succeeded.

FROM (
    FROM intable1
    SELECT acct_id, esn) b
JOIN (
    FROM intable2
    SELECT acct_id, xid, devtype_id, esn, other_properties['cdndldist'] AS cdndldist
    WHERE (dateint>=20110201 AND dateint<=20110228)
    AND (other_properties['cdndldist'] is not null)
    AND (client_msg_type='endplay')
    AND (devtype_id=272 OR devtype_id=129 OR devtype_id=12 OR devtype_id=13 OR devtype_id=14)) c
ON (b.acct_id=c.acct_id) AND (b.esn=c.esn)
INSERT OVERWRITE TABLE outtable PARTITION (dateint='20110201-20110228')
SELECT b.acct_id, c.xid, c.devtype_id, c.esn, c.cdndldist;

I'm on a pre-release version of Hive 0.7, with Hadoop 0.20. Does anyone know about any Hive/Hadoop issue that may be related to this?

Thanks.
Steven

Re: Mapper OOMs disappear after disabling JVM reuse

[Igor Tatarinov <ig...@decide.com>]

I had a similar problem until I set this parameter to 1 (although 3 seems to
work fine too).

There is an explanation somewhere on the web. Basically, if you run 20 tasks
and the garbage collector cannot catch up with accumulated garbage, the java
process grows too big so when it finally decides to fork a new process, the
forked process and the original one take up too much memory. Or something
like that.

I imagine the same kind thing would still happen even when setting reuse to
1 if a task happens to consume a lot of memory and CPU cycles, so something
else must be happening here. Perhaps, there is a memory 'leak' and some
'garbage' never gets collected between task executions.


On Thu, Apr 7, 2011 at 6:53 PM, Steven Wong <sw...@netflix.com> wrote:

> When the following query was run with mapred.job.reuse.jvm.num.tasks=20,
> some of the map tasks failed with “Error: Java heap space”, causing the job
> to fail. After changing to mapred.job.reuse.jvm.num.tasks=1, the job
> succeeded.
>
>
>
> FROM (
>
>     FROM intable1
>
>     SELECT acct_id, esn) b
>
> JOIN (
>
>     FROM intable2
>
>     SELECT acct_id, xid, devtype_id, esn, other_properties['cdndldist'] AS
> cdndldist
>
>     WHERE (dateint>=20110201 AND dateint<=20110228)
>
>     AND (other_properties['cdndldist'] is not null)
>
>     AND (client_msg_type='endplay')
>
>     AND (devtype_id=272 OR devtype_id=129 OR devtype_id=12 OR devtype_id=13
> OR devtype_id=14)) c
>
> ON (b.acct_id=c.acct_id) AND (b.esn=c.esn)
>
> INSERT OVERWRITE TABLE outtable PARTITION (dateint='20110201-20110228')
>
> SELECT b.acct_id, c.xid, c.devtype_id, c.esn, c.cdndldist;
>
>
>
> I’m on a pre-release version of Hive 0.7, with Hadoop 0.20. Does anyone
> know about any Hive/Hadoop issue that may be related to this?
>
>
>
> Thanks.
>
> Steven
>