Jobs submitted by oozie are being in Pending state for long time

[Giridhar Addepalli <gi...@gmail.com>]

Hi all,

Recently many jobs submitted by oozie are being in pending state for every
long times.

We have shell actions in our workflows.

Before this shell action, there is one custom synchronous action to write
event into mysql db. This event is getting completed.
But the oozie-launcher's single mapper job to start shell action , is being
in pending state for very long time.
Currently, there are jobs which are in pending state for 19 hours!

This behavior is making all our pipelines to stall in production.

Other jobs that are getting submitted via hive/ direct map-reduce jobs are
getting completed properly.

Please let me know if you need further information.

What could be the reasons behind this behavior?

Please help.

Thanks,
Giridhar.

Re: Jobs submitted by oozie are being in Pending state for long time

[Mona Chitnis <ch...@yahoo-inc.com>]

Looks like resource contention is happening somewhere in your system. Are
you setting a specific queue name
Oozie.launcher.mapreduce.job.queuename
For the launcher for your shell action? If that queue is backed up or
already has max number of parallel applications, it will stay in pending.

If not, can you share some more details about the configuration
differences between this stalled shell action and the other successfully
proceeding hive/map-reduce actions?

Mona

On 3/21/14, 8:03 PM, "Giridhar Addepalli" <gi...@gmail.com> wrote:

>Hi all,
>
>Recently many jobs submitted by oozie are being in pending state for every
>long times.
>
>We have shell actions in our workflows.
>
>Before this shell action, there is one custom synchronous action to write
>event into mysql db. This event is getting completed.
>But the oozie-launcher's single mapper job to start shell action , is
>being
>in pending state for very long time.
>Currently, there are jobs which are in pending state for 19 hours!
>
>This behavior is making all our pipelines to stall in production.
>
>Other jobs that are getting submitted via hive/ direct map-reduce jobs are
>getting completed properly.
>
>Please let me know if you need further information.
>
>What could be the reasons behind this behavior?
>
>Please help.
>
>Thanks,
>Giridhar.