You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by lune voo <lu...@gmail.com> on 2018/10/19 15:34:39 UTC
Oozie actions staying in PREP status
Hello everyone.
Sending this mail because I have a problem with my Oozie server.
Currently using Oozie 4.2 in HDP 2.6.2 in a cluster of 10 worker nodes, 2
master with HA components and multiples edges with one containing the oozie
server.
I send this mail because I have a problem in the cluster.
All the actions of all the workflows are stuck in PREP state.
I cannot see any jobs running in Yarn, except some of them not using Oozie.
The problem begun this morning, before it was working well.
I checked oozie server logs and don't see anything special (no ERROR or
WARN) even in DEBUG log mode.
I checked the oozie admin web page to see if it was a JVM problem, but I
cannot see anything special :
###
Memory Information:
- Heap Memory:
INIT=197132288 USED=309877976 COMMITTED=832569344 MAX=3817865216
- NonHeap Memory:
INIT=2555904 USED=139238472 COMMITTED=142237696 MAX=-1
- ClassLoading:
Total Loaded Classes=10656 Loaded Classes=10564 Unloaded
Classes=92
- Threads:
Thread Count=62 Peak Thread Count=135 Total Started
Threads=2708300 Deamon Threads=40
RUNNABLE=6 BLOCKED=0 WAITING=39 TIMED_WAITING=17
NEW=0 TERMINATED=0
Time of Thread Dump=Fri Oct 19 17:32:43 CEST 2018
###
As an additionnal note, my environment is kerberized and everything except
Oozie works fine.
I was wondering if one of you had already seen this behaviour ?
Best regards.
Lune
Re: Oozie actions staying in PREP status
Posted by lune voo <lu...@gmail.com>.
No ERROR in the logs unfortunately :-(.
Best regards.
Lune
Le mer. 24 oct. 2018 à 12:08, Peter Cseh <ge...@cloudera.com.invalid> a
écrit :
> Have you seen any errors in the Oozie server logs?
> gp
>
> On Wed, Oct 24, 2018 at 12:05 PM lune voo <lu...@gmail.com> wrote:
>
> > Hello Andras.
> >
> > Nothing in the RM yet as I wrote in my initial mail.
> >
> > Best regards.
> >
> > Lune
> >
> >
> > Le mer. 24 oct. 2018 à 11:55, Andras Piros <an...@apache.org> a écrit :
> >
> > > Hello,
> > >
> > > in case there are some Oozie launcher / action applications inside YARN
> > RM
> > > WebUI but these haven't been run, chances are you need to set
> > > oozie.launcher.mapreduce.map.memory.mb to some high value in your
> > > workflow.xml file like this
> > > <
> > >
> >
> https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Oozie-Spark-Action-getting-out-java-lang-OutOfMemoryError/td-p/39732
> > > >
> > > .
> > >
> > > Regards,
> > >
> > > Andras
> > >
> > > On Fri, Oct 19, 2018 at 5:34 PM lune voo <lu...@gmail.com>
> wrote:
> > >
> > > > Hello everyone.
> > > >
> > > > Sending this mail because I have a problem with my Oozie server.
> > > > Currently using Oozie 4.2 in HDP 2.6.2 in a cluster of 10 worker
> > nodes, 2
> > > > master with HA components and multiples edges with one containing the
> > > oozie
> > > > server.
> > > >
> > > > I send this mail because I have a problem in the cluster.
> > > >
> > > > All the actions of all the workflows are stuck in PREP state.
> > > > I cannot see any jobs running in Yarn, except some of them not using
> > > Oozie.
> > > >
> > > > The problem begun this morning, before it was working well.
> > > >
> > > > I checked oozie server logs and don't see anything special (no ERROR
> or
> > > > WARN) even in DEBUG log mode.
> > > >
> > > > I checked the oozie admin web page to see if it was a JVM problem,
> but
> > I
> > > > cannot see anything special :
> > > > ###
> > > > Memory Information:
> > > >
> > > > - Heap Memory:
> > > > INIT=197132288 USED=309877976 COMMITTED=832569344
> > > MAX=3817865216
> > > > - NonHeap Memory:
> > > > INIT=2555904 USED=139238472 COMMITTED=142237696 MAX=-1
> > > > - ClassLoading:
> > > > Total Loaded Classes=10656 Loaded Classes=10564 Unloaded
> > > > Classes=92
> > > > - Threads:
> > > > Thread Count=62 Peak Thread Count=135 Total Started
> > > > Threads=2708300 Deamon Threads=40
> > > >
> > > > RUNNABLE=6 BLOCKED=0 WAITING=39 TIMED_WAITING=17
> > > > NEW=0 TERMINATED=0
> > > > Time of Thread Dump=Fri Oct 19 17:32:43 CEST 2018
> > > >
> > > > ###
> > > >
> > > > As an additionnal note, my environment is kerberized and everything
> > > except
> > > > Oozie works fine.
> > > >
> > > > I was wondering if one of you had already seen this behaviour ?
> > > >
> > > > Best regards.
> > > >
> > > > Lune
> > > >
> > >
> >
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com <https://www.cloudera.com>
>
> [image: Cloudera] <https://www.cloudera.com/>
>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
>
Re: Oozie actions staying in PREP status
Posted by Peter Cseh <ge...@cloudera.com.INVALID>.
Have you seen any errors in the Oozie server logs?
gp
On Wed, Oct 24, 2018 at 12:05 PM lune voo <lu...@gmail.com> wrote:
> Hello Andras.
>
> Nothing in the RM yet as I wrote in my initial mail.
>
> Best regards.
>
> Lune
>
>
> Le mer. 24 oct. 2018 à 11:55, Andras Piros <an...@apache.org> a écrit :
>
> > Hello,
> >
> > in case there are some Oozie launcher / action applications inside YARN
> RM
> > WebUI but these haven't been run, chances are you need to set
> > oozie.launcher.mapreduce.map.memory.mb to some high value in your
> > workflow.xml file like this
> > <
> >
> https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Oozie-Spark-Action-getting-out-java-lang-OutOfMemoryError/td-p/39732
> > >
> > .
> >
> > Regards,
> >
> > Andras
> >
> > On Fri, Oct 19, 2018 at 5:34 PM lune voo <lu...@gmail.com> wrote:
> >
> > > Hello everyone.
> > >
> > > Sending this mail because I have a problem with my Oozie server.
> > > Currently using Oozie 4.2 in HDP 2.6.2 in a cluster of 10 worker
> nodes, 2
> > > master with HA components and multiples edges with one containing the
> > oozie
> > > server.
> > >
> > > I send this mail because I have a problem in the cluster.
> > >
> > > All the actions of all the workflows are stuck in PREP state.
> > > I cannot see any jobs running in Yarn, except some of them not using
> > Oozie.
> > >
> > > The problem begun this morning, before it was working well.
> > >
> > > I checked oozie server logs and don't see anything special (no ERROR or
> > > WARN) even in DEBUG log mode.
> > >
> > > I checked the oozie admin web page to see if it was a JVM problem, but
> I
> > > cannot see anything special :
> > > ###
> > > Memory Information:
> > >
> > > - Heap Memory:
> > > INIT=197132288 USED=309877976 COMMITTED=832569344
> > MAX=3817865216
> > > - NonHeap Memory:
> > > INIT=2555904 USED=139238472 COMMITTED=142237696 MAX=-1
> > > - ClassLoading:
> > > Total Loaded Classes=10656 Loaded Classes=10564 Unloaded
> > > Classes=92
> > > - Threads:
> > > Thread Count=62 Peak Thread Count=135 Total Started
> > > Threads=2708300 Deamon Threads=40
> > >
> > > RUNNABLE=6 BLOCKED=0 WAITING=39 TIMED_WAITING=17
> > > NEW=0 TERMINATED=0
> > > Time of Thread Dump=Fri Oct 19 17:32:43 CEST 2018
> > >
> > > ###
> > >
> > > As an additionnal note, my environment is kerberized and everything
> > except
> > > Oozie works fine.
> > >
> > > I was wondering if one of you had already seen this behaviour ?
> > >
> > > Best regards.
> > >
> > > Lune
> > >
> >
>
--
*Peter Cseh *| Software Engineer
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
Re: Oozie actions staying in PREP status
Posted by lune voo <lu...@gmail.com>.
Hello Andras.
Nothing in the RM yet as I wrote in my initial mail.
Best regards.
Lune
Le mer. 24 oct. 2018 à 11:55, Andras Piros <an...@apache.org> a écrit :
> Hello,
>
> in case there are some Oozie launcher / action applications inside YARN RM
> WebUI but these haven't been run, chances are you need to set
> oozie.launcher.mapreduce.map.memory.mb to some high value in your
> workflow.xml file like this
> <
> https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Oozie-Spark-Action-getting-out-java-lang-OutOfMemoryError/td-p/39732
> >
> .
>
> Regards,
>
> Andras
>
> On Fri, Oct 19, 2018 at 5:34 PM lune voo <lu...@gmail.com> wrote:
>
> > Hello everyone.
> >
> > Sending this mail because I have a problem with my Oozie server.
> > Currently using Oozie 4.2 in HDP 2.6.2 in a cluster of 10 worker nodes, 2
> > master with HA components and multiples edges with one containing the
> oozie
> > server.
> >
> > I send this mail because I have a problem in the cluster.
> >
> > All the actions of all the workflows are stuck in PREP state.
> > I cannot see any jobs running in Yarn, except some of them not using
> Oozie.
> >
> > The problem begun this morning, before it was working well.
> >
> > I checked oozie server logs and don't see anything special (no ERROR or
> > WARN) even in DEBUG log mode.
> >
> > I checked the oozie admin web page to see if it was a JVM problem, but I
> > cannot see anything special :
> > ###
> > Memory Information:
> >
> > - Heap Memory:
> > INIT=197132288 USED=309877976 COMMITTED=832569344
> MAX=3817865216
> > - NonHeap Memory:
> > INIT=2555904 USED=139238472 COMMITTED=142237696 MAX=-1
> > - ClassLoading:
> > Total Loaded Classes=10656 Loaded Classes=10564 Unloaded
> > Classes=92
> > - Threads:
> > Thread Count=62 Peak Thread Count=135 Total Started
> > Threads=2708300 Deamon Threads=40
> >
> > RUNNABLE=6 BLOCKED=0 WAITING=39 TIMED_WAITING=17
> > NEW=0 TERMINATED=0
> > Time of Thread Dump=Fri Oct 19 17:32:43 CEST 2018
> >
> > ###
> >
> > As an additionnal note, my environment is kerberized and everything
> except
> > Oozie works fine.
> >
> > I was wondering if one of you had already seen this behaviour ?
> >
> > Best regards.
> >
> > Lune
> >
>
Re: Oozie actions staying in PREP status
Posted by Andras Piros <an...@apache.org>.
Hello,
in case there are some Oozie launcher / action applications inside YARN RM
WebUI but these haven't been run, chances are you need to set
oozie.launcher.mapreduce.map.memory.mb to some high value in your
workflow.xml file like this
<https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Oozie-Spark-Action-getting-out-java-lang-OutOfMemoryError/td-p/39732>
.
Regards,
Andras
On Fri, Oct 19, 2018 at 5:34 PM lune voo <lu...@gmail.com> wrote:
> Hello everyone.
>
> Sending this mail because I have a problem with my Oozie server.
> Currently using Oozie 4.2 in HDP 2.6.2 in a cluster of 10 worker nodes, 2
> master with HA components and multiples edges with one containing the oozie
> server.
>
> I send this mail because I have a problem in the cluster.
>
> All the actions of all the workflows are stuck in PREP state.
> I cannot see any jobs running in Yarn, except some of them not using Oozie.
>
> The problem begun this morning, before it was working well.
>
> I checked oozie server logs and don't see anything special (no ERROR or
> WARN) even in DEBUG log mode.
>
> I checked the oozie admin web page to see if it was a JVM problem, but I
> cannot see anything special :
> ###
> Memory Information:
>
> - Heap Memory:
> INIT=197132288 USED=309877976 COMMITTED=832569344 MAX=3817865216
> - NonHeap Memory:
> INIT=2555904 USED=139238472 COMMITTED=142237696 MAX=-1
> - ClassLoading:
> Total Loaded Classes=10656 Loaded Classes=10564 Unloaded
> Classes=92
> - Threads:
> Thread Count=62 Peak Thread Count=135 Total Started
> Threads=2708300 Deamon Threads=40
>
> RUNNABLE=6 BLOCKED=0 WAITING=39 TIMED_WAITING=17
> NEW=0 TERMINATED=0
> Time of Thread Dump=Fri Oct 19 17:32:43 CEST 2018
>
> ###
>
> As an additionnal note, my environment is kerberized and everything except
> Oozie works fine.
>
> I was wondering if one of you had already seen this behaviour ?
>
> Best regards.
>
> Lune
>