Out Of Memory on localmode.

[Bennie Schut <bs...@ebuddy.com>]

Hi,

Just in case anyone else ever runs into this.
Lately our cluster kept on killing itself with an OOM message in the kernel log. It took me a while to realize why this happened since no single process was causing it.
I traced it back to a few queries running concurrently on a really small datasets. This caused all of these queries to run localmode. Then I realized there isn't a limit to how many queries can run in localmode and since they use the normal hadoop memory settings it's pretty easy to hit OOM on a machine this way.
I'm not sure about the long term solution (some kind of limit on the number of localmode processes), but for now I'll probably disable localmode on these queries.

Bennie.

Re: Out Of Memory on localmode.

[Edward Capriolo <ed...@gmail.com>]

In that situation I might turn the setting off entirely. You would
guess that the common use case is not "really small datasets" better
the query to take longer then to oom a machine.

On Tue, Feb 5, 2013 at 3:12 AM, Bennie Schut <bs...@ebuddy.com> wrote:
> Hi,
>
>
>
> Just in case anyone else ever runs into this.
>
> Lately our cluster kept on killing itself with an OOM message in the kernel
> log. It took me a while to realize why this happened since no single process
> was causing it.
>
> I traced it back to a few queries running concurrently on a really small
> datasets. This caused all of these queries to run localmode. Then I realized
> there isn’t a limit to how many queries can run in localmode and since they
> use the normal hadoop memory settings it’s pretty easy to hit OOM on a
> machine this way.
>
> I’m not sure about the long term solution (some kind of limit on the number
> of localmode processes), but for now I’ll probably disable localmode on
> these queries.
>
>
>
> Bennie.