You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Gaetano Fabiano <fa...@gmail.com> on 2014/08/08 15:57:38 UTC
Problem with Oozie DeadLock
Hi All,
this is my first post on this mailing list I hope to stay here for long
time. I saw the project and I love it.
But recently I'm going crazy about issues.
Our problem is about Oozie Deadlock, we would like to execute more than one
Oozie job at the same time but when we try to execute its the entire
environment becames locked all.
We read a lot of post about this issue but without solution.
How is possible to set two different queue? We read a lot of post where
people suggest to use different queue for different execution
Where we could set these queue setting?
Is this the correct way to have different execution at the same time?
Our bewilderment is reading this issue
https://issues.apache.org/jira/browse/OOZIE-1673 and if is as described
the big doubt is about the usefulness of the enteire Oozie framework.
I hope someone can help us to resolve it and clear our mind about.
Any suggestion is welcome.
Regards
Gaetano
Dott. Gaetano Fabiano
via Timpone n° 79
87055 San Giovanni in Fiore (Cs) ITALY
mobile: +39 328 9469919
phone: +39 0984 991980
email: fabiano.gaetano@gmail.com
skype: deepyoudeep
skype:: gaetano.fab
msn: DeepYou@hotmail.it
twitter: @gaetanofabiano
gtalk/hangoout/Google+ fg.pappa@gmail.com
Re: Problem with Oozie DeadLock
Posted by Mona Chitnis <mo...@apache.org>.
Hi Fabiano,
You should definitely be able to execute multiple Oozie jobs simultaneously. The issue comes up when you have a small hadoop cluster setup, and thus very small number of queue slots for submitting jobs to the ResourceManager.
Can you look into adding an additional queue by configuring your Hadoop cluster through capacity-scheduler.xml? Then you can use the approach mentioned in OOZIE-1673, to specify in your workflow's properties
oozie.launcher.mapreduce.job.queuename=queue1
mapreduce.job.queuename=queue2
and avoid deadlock situation.
You can also avoid deadlocks by tuning the memory requirements of your oozie launcher and child jobs, to request for lower memory container slots, and increase the number of jobs you can submit and execute that way.http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html
Mona Chitnis
Yahoo!
On Friday, August 8, 2014 6:58 AM, Gaetano Fabiano <fa...@gmail.com> wrote:
Hi All,
this is my first post on this mailing list I hope to stay here for long
time. I saw the project and I love it.
But recently I'm going crazy about issues.
Our problem is about Oozie Deadlock, we would like to execute more than one
Oozie job at the same time but when we try to execute its the entire
environment becames locked all.
We read a lot of post about this issue but without solution.
How is possible to set two different queue? We read a lot of post where
people suggest to use different queue for different execution
Where we could set these queue setting?
Is this the correct way to have different execution at the same time?
Our bewilderment is reading this issue
https://issues.apache.org/jira/browse/OOZIE-1673 and if is as described
the big doubt is about the usefulness of the enteire Oozie framework.
I hope someone can help us to resolve it and clear our mind about.
Any suggestion is welcome.
Regards
Gaetano
Dott. Gaetano Fabiano
via Timpone n° 79
87055 San Giovanni in Fiore (Cs) ITALY
mobile: +39 328 9469919
phone: +39 0984 991980
email: fabiano.gaetano@gmail.com
skype: deepyoudeep
skype:: gaetano.fab
msn: DeepYou@hotmail.it
twitter: @gaetanofabiano
gtalk/hangoout/Google+ fg.pappa@gmail.com
Re: Problem with Oozie DeadLock
Posted by Mona Chitnis <mo...@apache.org>.
Hi Fabiano,
You should definitely be able to execute multiple Oozie jobs simultaneously. The issue comes up when you have a small hadoop cluster setup, and thus very small number of queue slots for submitting jobs to the ResourceManager.
Can you look into adding an additional queue by configuring your Hadoop cluster through capacity-scheduler.xml? Then you can use the approach mentioned in OOZIE-1673, to specify in your workflow's properties
oozie.launcher.mapreduce.job.queuename=queue1
mapreduce.job.queuename=queue2
and avoid deadlock situation.
You can also avoid deadlocks by tuning the memory requirements of your oozie launcher and child jobs, to request for lower memory container slots, and increase the number of jobs you can submit and execute that way.http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html
Mona Chitnis
Yahoo!
On Friday, August 8, 2014 6:58 AM, Gaetano Fabiano <fa...@gmail.com> wrote:
Hi All,
this is my first post on this mailing list I hope to stay here for long
time. I saw the project and I love it.
But recently I'm going crazy about issues.
Our problem is about Oozie Deadlock, we would like to execute more than one
Oozie job at the same time but when we try to execute its the entire
environment becames locked all.
We read a lot of post about this issue but without solution.
How is possible to set two different queue? We read a lot of post where
people suggest to use different queue for different execution
Where we could set these queue setting?
Is this the correct way to have different execution at the same time?
Our bewilderment is reading this issue
https://issues.apache.org/jira/browse/OOZIE-1673 and if is as described
the big doubt is about the usefulness of the enteire Oozie framework.
I hope someone can help us to resolve it and clear our mind about.
Any suggestion is welcome.
Regards
Gaetano
Dott. Gaetano Fabiano
via Timpone n° 79
87055 San Giovanni in Fiore (Cs) ITALY
mobile: +39 328 9469919
phone: +39 0984 991980
email: fabiano.gaetano@gmail.com
skype: deepyoudeep
skype:: gaetano.fab
msn: DeepYou@hotmail.it
twitter: @gaetanofabiano
gtalk/hangoout/Google+ fg.pappa@gmail.com