You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Dimple Patel <di...@gmail.com> on 2016/11/04 22:28:50 UTC
Operators stay in PENDING_DEPLOY state
Hello,
I am trying to run Top N Word application from DataTorrent's documentation
<http://docs.datatorrent.com/tutorials/topnwords/> .
After running the application on DataTorrent's console, when I monitor it in
dtDashboard, the app's status is running but all my operators are in
PENDING_DEPLOY state and are not switching to Running state. Attached
herewith is my AppMaster log. I think the problem is with memory allocation
in all operators as when I run a simple random number generation app, it's
operators run fine. But I am not entirely sure what exactly the problem is.
What is the problem here?
dt.log <http://apache-apex-users-list.78494.x6.nabble.com/file/n1110/dt.log>
Thanks,
Dimple
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Operators-stay-in-PENDING-DEPLOY-state-tp1110.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: Operators stay in PENDING_DEPLOY state
Posted by Dimple Patel <di...@gmail.com>.
I did that but it still didn't help.
With some help I tried to run a few operators in a way that I set two
operators in the same container where upstreams were not coming from more
than one operators but I still had too many containers for my machine to be
able to divide enough resources. I am able to run application with fewer
operators so I think I won't be able to run any application with many
containers in datatorrent vm.
Correct me if I am wrong.
Also, in general, to run n number of containers in a datatorrent virtual
machine what should be the system requirements?
Thanks.
Dimple
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Operators-stay-in-PENDING-DEPLOY-state-tp1110p1115.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: Operators stay in PENDING_DEPLOY state
Posted by Munagala Ramanath <ra...@datatorrent.com>.
>I have set the YARN minimum allocation property *in property.xml of the
>project*, which doesn't solve the problem.
Setting it in your project will not help; you need to set it in
yarn-site.xml
and restart YARN. If you're running on the sandbox, the location of that
file is in my earlier
message.
Ram
On Sat, Nov 5, 2016 at 12:21 AM, Dimple Patel <di...@gmail.com>
wrote:
> Ashwin, Ramanath,
>
> Thank you. I understood the problem.
>
> I have set the YARN minimum allocation property *in property.xml of the
> project*, which doesn't solve the problem.
>
> For the unavailable containers for operators, what are the ways I can make
> them available?
>
> One way I can think of doing that reducing the allocated memory of existing
> containers but they are using almost all the memory. In every container
> around 100s of MBs are free.
>
> How can I allocate resources to these operators?
>
>
>
> --
> View this message in context: http://apache-apex-users-list.
> 78494.x6.nabble.com/Operators-stay-in-PENDING-DEPLOY-state-
> tp1110p1113.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
>
Re: Operators stay in PENDING_DEPLOY state
Posted by Dimple Patel <di...@gmail.com>.
Ashwin, Ramanath,
Thank you. I understood the problem.
I have set the YARN minimum allocation property *in property.xml of the
project*, which doesn't solve the problem.
For the unavailable containers for operators, what are the ways I can make
them available?
One way I can think of doing that reducing the allocated memory of existing
containers but they are using almost all the memory. In every container
around 100s of MBs are free.
How can I allocate resources to these operators?
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Operators-stay-in-PENDING-DEPLOY-state-tp1110p1113.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: Operators stay in PENDING_DEPLOY state
Posted by Munagala Ramanath <ra...@datatorrent.com>.
To add to Ashwin's diagnosis, assuming you have *properties.xml* setup as
described in that tutorial,
you need to add to the YARN configuration file *yarn-site.xml* at:
*/sfw/hadoop/current/etc/hadoop*
the following stanza:
*<property><name>yarn.scheduler.minimum-allocation-mb</name><value>128</value></property>*
and restart the Hadoop daemons (the easiest way to do this is to simply
bounce the
sandbox or adapt the script shown in the "*Service Management*"
section at: *http://docs.datatorrent.com/sandbox
<http://docs.datatorrent.com/sandbox>*/)
Ram
On Fri, Nov 4, 2016 at 5:18 PM, Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:
> Dimple,
>
> Cluster does not have enough resources. Here is the initial container
> request set from the log.
>
> 2016-11-04 15:06:26,901 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1024, vCores:1>]Priority[0], Capability[<memory:1224, vCores:1>]Priority[1], Capability[<memory:1024, vCores:1>]Priority[2], Capability[<memory:1024, vCores:1>]Priority[3], Capability[<memory:1024, vCores:1>]Priority[4], Capability[<memory:1024, vCores:1>]Priority[5], Capability[<memory:1024, vCores:1>]Priority[6], Capability[<memory:1024, vCores:1>]Priority[7], Capability[<memory:1024, vCores:1>]Priority[8], Capability[<memory:2048, vCores:1>]Priority[9]]
>
> AM then keeps asking for the following 4 containers every 30 seconds but
> never gets.
>
> 2016-11-04 15:07:01,132 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1024, vCores:1>]Priority[7], Capability[<memory:1024, vCores:1>]Priority[8], Capability[<memory:2048, vCores:1>]Priority[9], Capability[<memory:1024, vCores:1>]Priority[6]]
>
> Regards,
> Ashwin.
>
> On Fri, Nov 4, 2016 at 3:28 PM, Dimple Patel <di...@gmail.com>
> wrote:
>
>> Hello,
>>
>> I am trying to run Top N Word application from DataTorrent's
>> documentation
>> <http://docs.datatorrent.com/tutorials/topnwords/> .
>> After running the application on DataTorrent's console, when I monitor it
>> in
>> dtDashboard, the app's status is running but all my operators are in
>> PENDING_DEPLOY state and are not switching to Running state. Attached
>> herewith is my AppMaster log. I think the problem is with memory
>> allocation
>> in all operators as when I run a simple random number generation app, it's
>> operators run fine. But I am not entirely sure what exactly the problem
>> is.
>>
>> What is the problem here?
>>
>> dt.log <http://apache-apex-users-list.78494.x6.nabble.com/file/n111
>> 0/dt.log>
>>
>> Thanks,
>> Dimple
>>
>>
>>
>> --
>> View this message in context: http://apache-apex-users-list.
>> 78494.x6.nabble.com/Operators-stay-in-PENDING-DEPLOY-state-tp1110.html
>> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
>>
>
>
>
> --
>
> Regards,
> Ashwin.
>
Re: Operators stay in PENDING_DEPLOY state
Posted by Ashwin Chandra Putta <as...@gmail.com>.
Dimple,
Cluster does not have enough resources. Here is the initial container
request set from the log.
2016-11-04 15:06:26,901 INFO
com.datatorrent.stram.StreamingAppMasterService: Asking RM for
containers: [Capability[<memory:1024, vCores:1>]Priority[0],
Capability[<memory:1224, vCores:1>]Priority[1],
Capability[<memory:1024, vCores:1>]Priority[2],
Capability[<memory:1024, vCores:1>]Priority[3],
Capability[<memory:1024, vCores:1>]Priority[4],
Capability[<memory:1024, vCores:1>]Priority[5],
Capability[<memory:1024, vCores:1>]Priority[6],
Capability[<memory:1024, vCores:1>]Priority[7],
Capability[<memory:1024, vCores:1>]Priority[8],
Capability[<memory:2048, vCores:1>]Priority[9]]
AM then keeps asking for the following 4 containers every 30 seconds but
never gets.
2016-11-04 15:07:01,132 INFO
com.datatorrent.stram.StreamingAppMasterService: Asking RM for
containers: [Capability[<memory:1024, vCores:1>]Priority[7],
Capability[<memory:1024, vCores:1>]Priority[8],
Capability[<memory:2048, vCores:1>]Priority[9],
Capability[<memory:1024, vCores:1>]Priority[6]]
Regards,
Ashwin.
On Fri, Nov 4, 2016 at 3:28 PM, Dimple Patel <di...@gmail.com>
wrote:
> Hello,
>
> I am trying to run Top N Word application from DataTorrent's documentation
> <http://docs.datatorrent.com/tutorials/topnwords/> .
> After running the application on DataTorrent's console, when I monitor it
> in
> dtDashboard, the app's status is running but all my operators are in
> PENDING_DEPLOY state and are not switching to Running state. Attached
> herewith is my AppMaster log. I think the problem is with memory allocation
> in all operators as when I run a simple random number generation app, it's
> operators run fine. But I am not entirely sure what exactly the problem is.
>
> What is the problem here?
>
> dt.log <http://apache-apex-users-list.78494.x6.nabble.com/file/
> n1110/dt.log>
>
> Thanks,
> Dimple
>
>
>
> --
> View this message in context: http://apache-apex-users-list.
> 78494.x6.nabble.com/Operators-stay-in-PENDING-DEPLOY-state-tp1110.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
>
--
Regards,
Ashwin.