You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by tkg_cangkul <yu...@gmail.com> on 2016/02/11 04:33:44 UTC
oozie problem stuck on running state
i've been trying to run oozie from virtual machine. but the process
stuck at running state. i've set my memory of RM to 4gb and my Vcore is
2. is there any configuration that i've missed?
pls help
capture from my resource manager
Re: oozie problem stuck on running state
Posted by tkg_cangkul <yu...@gmail.com>.
yes i set the capacity scheduler and it's success..
thx for your help :)
On 11/02/16 13:14, Jaydeep Vishwakarma wrote:
> You have assigned only 4096MB memory to node manager and you have only
> one node.
> Increase it if you have enough memory by following parameters in cluster:
> *yarn-site.xml : yarn.nodemanager.resource.memory-mb *
> *
> *
> As I can see you are using capacity scheduler, resources in the
> cluster which can be
> used to run application masters is default to 10% , increase it to 50%
> or more.
> *capacity-scheduler.xml :
> yarn.scheduler.capacity.maximum-am-resource-percent 0.5*
>
> As you have very limited memory you should reduce other memory
> parameter as well.
> Reduce following memory parameters in hadoop conf of oozie :
> *mapred-site.xml : mapreduce.map.memory.mb 512*
> *mapred-site.xml : yarn.app.mapreduce.am.resource.mb 1024*
> *mapred-site.xml : yarn.app.mapreduce.am.command-opts -Xmx800m*
> *mapred-site.xml : mapreduce.map.java.opts -Xmx400m*
> *
> *
> If your oozie hadoop conf and cluster hadoop conf pointing to same
> path than
> both will get impacted. You may wish to reduce the memory size for
> cluster as well.
>
> Restart both cluster and oozie.
>
> Regards,
> Jaydeep
>
>
>
> On Thu, Feb 11, 2016 at 9:03 AM, tkg_cangkul <yuza.rasfar@gmail.com
> <ma...@gmail.com>> wrote:
>
> i've been trying to run oozie from virtual machine. but the
> process stuck at running state. i've set my memory of RM to 4gb
> and my Vcore is 2. is there any configuration that i've missed?
> pls help
>
> capture from my resource manager
>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for
> the use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally
> privileged information. If you are not the intended recipient you are
> hereby notified that any disclosure, copying, distribution or taking
> any action in reliance on the contents of this information is strictly
> prohibited and may be unlawful. If you have received this
> communication in error, please notify us immediately by responding to
> this email and then delete it from your system. The firm is neither
> liable for the proper and complete transmission of the information
> contained in this communication nor for any delay in its receipt.
Re: oozie problem stuck on running state
Posted by Jaydeep Vishwakarma <ja...@inmobi.com>.
You have assigned only 4096MB memory to node manager and you have only one
node.
Increase it if you have enough memory by following parameters in cluster:
*yarn-site.xml : yarn.nodemanager.resource.memory-mb *
As I can see you are using capacity scheduler, resources in the cluster
which can be
used to run application masters is default to 10% , increase it to 50% or
more.
*capacity-scheduler.xml :
yarn.scheduler.capacity.maximum-am-resource-percent 0.5*
As you have very limited memory you should reduce other memory parameter as
well.
Reduce following memory parameters in hadoop conf of oozie :
*mapred-site.xml : mapreduce.map.memory.mb 512*
*mapred-site.xml : yarn.app.mapreduce.am.resource.mb 1024*
*mapred-site.xml : yarn.app.mapreduce.am.command-opts -Xmx800m*
*mapred-site.xml : mapreduce.map.java.opts -Xmx400m*
If your oozie hadoop conf and cluster hadoop conf pointing to same path than
both will get impacted. You may wish to reduce the memory size for cluster
as well.
Restart both cluster and oozie.
Regards,
Jaydeep
On Thu, Feb 11, 2016 at 9:03 AM, tkg_cangkul <yu...@gmail.com> wrote:
> i've been trying to run oozie from virtual machine. but the process stuck
> at running state. i've set my memory of RM to 4gb and my Vcore is 2. is
> there any configuration that i've missed?
> pls help
>
> [image: capture from my resource manager]
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: oozie problem stuck on running state
Posted by tkg_cangkul <yu...@gmail.com>.
finnaly it's succeed.
thx for your help Nitin Kumar. Vary Helpful for me :)
On 11/02/16 12:01, Nitin Kumar wrote:
> I have a 5 node cluster with about 12GB of RAM at the disposal of the
> resource manager. When you are running your job take a look at the
> scheduler section of your resource manager UI. It will give you a hint
> whether you are starved of task slots or the queue is being overloaded or
> it is because your underlying map-reduce job is not getting enough memory
> for a yarn container.
>
> Limiting the maximum memory for a single container might allow multiple
> containers to get instantiated. This in turn will increase the number of
> jobs you can run simultaneously on yarn at the expense of smaller yarn
> containers.
>
> On Thu, Feb 11, 2016 at 10:17 AM, tkg_cangkul <yu...@gmail.com> wrote:
>
>> hi Nitin Kumar,
>>
>> thx for your reply. i've been try this way but still have the same issue.
>> can u tell me how much RM memory in your small cluster.
>> i've set my RM memory to 5Gb but still has the same issue
>>
>> On 11/02/16 10:52, Nitin Kumar wrote:
>>
>>> I faced a similar problem when running on a small test cluster. I'm not
>>> aware about the memory requirements of your underlying map-reduce job, but
>>> a prime cause of this 'deadlock' behaviour is because oozie map-reduce
>>> jobs
>>> are launched by an oozie launcher job (which is a map only job). This
>>> launcher job takes up task slots in the hadoop queue (the 'default' queue
>>> is used by default) and stays alive till the underlying map-reduce job
>>> terminates. If the the underlying map-reduce job gets no task slots it
>>> will
>>> wait for resources to be released and the launcher will wait for the job
>>> to
>>> terminate.
>>>
>>> A way to mitigate this issue is by splitting your default queue into 2 or
>>> more queues. Set the oozie.launcher.mapred.job.queue.name and
>>> mapred.job.queue.name properties to different queues and you should be
>>> fine.
>>>
>>> On Thu, Feb 11, 2016 at 9:06 AM, satish saley <sa...@gmail.com>
>>> wrote:
>>>
>>> Could you please share the Hadoop logs for your job?
>>>>
>>>> Sent from Yahoo Mail for iPhone <https://yho.com/footer0>
>>>>
>>>>
>>>> On Wednesday, February 10, 2016, 7:33 PM, tkg_cangkul <
>>>> yuza.rasfar@gmail.com> wrote:
>>>>
>>>> i've been trying to run oozie from virtual machine. but the process stuck
>>>> at running state. i've set my memory of RM to 4gb and my Vcore is 2. is
>>>> there any configuration that i've missed?
>>>> pls help
>>>>
>>>> [image: capture from my resource manager]
>>>>
>>>>
>>>>
Re: oozie problem stuck on running state
Posted by Nitin Kumar <nk...@gmail.com>.
I have a 5 node cluster with about 12GB of RAM at the disposal of the
resource manager. When you are running your job take a look at the
scheduler section of your resource manager UI. It will give you a hint
whether you are starved of task slots or the queue is being overloaded or
it is because your underlying map-reduce job is not getting enough memory
for a yarn container.
Limiting the maximum memory for a single container might allow multiple
containers to get instantiated. This in turn will increase the number of
jobs you can run simultaneously on yarn at the expense of smaller yarn
containers.
On Thu, Feb 11, 2016 at 10:17 AM, tkg_cangkul <yu...@gmail.com> wrote:
> hi Nitin Kumar,
>
> thx for your reply. i've been try this way but still have the same issue.
> can u tell me how much RM memory in your small cluster.
> i've set my RM memory to 5Gb but still has the same issue
>
> On 11/02/16 10:52, Nitin Kumar wrote:
>
>> I faced a similar problem when running on a small test cluster. I'm not
>> aware about the memory requirements of your underlying map-reduce job, but
>> a prime cause of this 'deadlock' behaviour is because oozie map-reduce
>> jobs
>> are launched by an oozie launcher job (which is a map only job). This
>> launcher job takes up task slots in the hadoop queue (the 'default' queue
>> is used by default) and stays alive till the underlying map-reduce job
>> terminates. If the the underlying map-reduce job gets no task slots it
>> will
>> wait for resources to be released and the launcher will wait for the job
>> to
>> terminate.
>>
>> A way to mitigate this issue is by splitting your default queue into 2 or
>> more queues. Set the oozie.launcher.mapred.job.queue.name and
>> mapred.job.queue.name properties to different queues and you should be
>> fine.
>>
>> On Thu, Feb 11, 2016 at 9:06 AM, satish saley <sa...@gmail.com>
>> wrote:
>>
>> Could you please share the Hadoop logs for your job?
>>>
>>>
>>> Sent from Yahoo Mail for iPhone <https://yho.com/footer0>
>>>
>>>
>>> On Wednesday, February 10, 2016, 7:33 PM, tkg_cangkul <
>>> yuza.rasfar@gmail.com> wrote:
>>>
>>> i've been trying to run oozie from virtual machine. but the process stuck
>>> at running state. i've set my memory of RM to 4gb and my Vcore is 2. is
>>> there any configuration that i've missed?
>>> pls help
>>>
>>> [image: capture from my resource manager]
>>>
>>>
>>>
>
Re: oozie problem stuck on running state
Posted by tkg_cangkul <yu...@gmail.com>.
hi Nitin Kumar,
thx for your reply. i've been try this way but still have the same issue.
can u tell me how much RM memory in your small cluster.
i've set my RM memory to 5Gb but still has the same issue
On 11/02/16 10:52, Nitin Kumar wrote:
> I faced a similar problem when running on a small test cluster. I'm not
> aware about the memory requirements of your underlying map-reduce job, but
> a prime cause of this 'deadlock' behaviour is because oozie map-reduce jobs
> are launched by an oozie launcher job (which is a map only job). This
> launcher job takes up task slots in the hadoop queue (the 'default' queue
> is used by default) and stays alive till the underlying map-reduce job
> terminates. If the the underlying map-reduce job gets no task slots it will
> wait for resources to be released and the launcher will wait for the job to
> terminate.
>
> A way to mitigate this issue is by splitting your default queue into 2 or
> more queues. Set the oozie.launcher.mapred.job.queue.name and
> mapred.job.queue.name properties to different queues and you should be fine.
>
> On Thu, Feb 11, 2016 at 9:06 AM, satish saley <sa...@gmail.com>
> wrote:
>
>> Could you please share the Hadoop logs for your job?
>>
>>
>> Sent from Yahoo Mail for iPhone <https://yho.com/footer0>
>>
>>
>> On Wednesday, February 10, 2016, 7:33 PM, tkg_cangkul <
>> yuza.rasfar@gmail.com> wrote:
>>
>> i've been trying to run oozie from virtual machine. but the process stuck
>> at running state. i've set my memory of RM to 4gb and my Vcore is 2. is
>> there any configuration that i've missed?
>> pls help
>>
>> [image: capture from my resource manager]
>>
>>
Re: oozie problem stuck on running state
Posted by Nitin Kumar <nk...@gmail.com>.
I faced a similar problem when running on a small test cluster. I'm not
aware about the memory requirements of your underlying map-reduce job, but
a prime cause of this 'deadlock' behaviour is because oozie map-reduce jobs
are launched by an oozie launcher job (which is a map only job). This
launcher job takes up task slots in the hadoop queue (the 'default' queue
is used by default) and stays alive till the underlying map-reduce job
terminates. If the the underlying map-reduce job gets no task slots it will
wait for resources to be released and the launcher will wait for the job to
terminate.
A way to mitigate this issue is by splitting your default queue into 2 or
more queues. Set the oozie.launcher.mapred.job.queue.name and
mapred.job.queue.name properties to different queues and you should be fine.
On Thu, Feb 11, 2016 at 9:06 AM, satish saley <sa...@gmail.com>
wrote:
> Could you please share the Hadoop logs for your job?
>
>
> Sent from Yahoo Mail for iPhone <https://yho.com/footer0>
>
>
> On Wednesday, February 10, 2016, 7:33 PM, tkg_cangkul <
> yuza.rasfar@gmail.com> wrote:
>
> i've been trying to run oozie from virtual machine. but the process stuck
> at running state. i've set my memory of RM to 4gb and my Vcore is 2. is
> there any configuration that i've missed?
> pls help
>
> [image: capture from my resource manager]
>
>
Re: oozie problem stuck on running state
Posted by satish saley <sa...@gmail.com>.
blockquote, div.yahoo_quoted { margin-left: 0 !important; border-left:1px #715FFA solid !important; padding-left:1ex !important; background-color:white !important; } Could you please share the Hadoop logs for your job?
Sent from Yahoo Mail for iPhone
On Wednesday, February 10, 2016, 7:33 PM, tkg_cangkul <yu...@gmail.com> wrote:
i've been trying to run oozie from virtual machine. but the process stuck at running state. i've set my memory of RM to 4gb and my Vcore is 2. is there any configuration that i've missed?
pls help