You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Vijay Srinivasaraghavan <vi...@yahoo.com> on 2016/12/09 22:54:53 UTC
Quota
I have a standalone DCOS setup (Single node Vagrant VM running DCOS v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are running on same VM.
Resource: 4 CPU, 16GB Memory, 20G Disk
I have created a quota using new V1 API which creates a role "test" with resource constraints of 0.5 CPU and 1G Memory.
When I try to deploy Spark package, Marathon receives the request but the task is in "waiting" state since it did not receive any offers from Master though I don't see any resource constraints from the hardware perspective.
However, when I deleted the quota, Marathon is able to move forward with the deployment and Spark was deployed/up and running. I could see from the Mesos master logs that it had sent an offer to the Marathon framework.
To debug the issue, I was trying to create a quota but this time did not provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to deploy Spark from DCOS UI, I could see Marathon getting offer from Master and able to deploy Spark without the need to delete the quota this time.
Did anyone notice similar behavior?
RegardsVijay
Re: Quota
Posted by Vijay Srinivasaraghavan <vi...@yahoo.com>.
Hi Alex,
>>Granularity in the allocator is a single agent.
Does this mean if I have only one agent and the moment if I set any quota, the framework running on the agent will not be allocated with any resource?
From the logs, I don't get to see much details for the scenario when the quota is set following deploying a package through Marathon. However, when I remove the quota, I see the following message in master log "Allocating ports(*):[*****]; disk(*):....." "Sending 1 offers to framework XX (marathon) at scheduler-XXX"
RegardsVijay
On Sunday, December 11, 2016 9:05 AM, Alex Rukletsov <al...@mesosphere.io> wrote:
Granularity in the allocator is a single agent. Hence even though you set quota for 0.0001 CPU, at least one agent is "blocked". This is probably the reason why marathon is not getting offers. You can turn verbose master logs and check allocator messages to confirm. Alex.
On 10 Dec 2016 2:14 am, "Vijay" <vi...@yahoo.com.invalid> wrote:
The dispatcher needs 1cpu and 1G memory.
Regards,
Vijay
Sent from my iPhone
> On Dec 9, 2016, at 4:51 PM, Vinod Kone <vi...@apache.org> wrote:
>
> And how many resources does spark need?
>
>> On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan <vi...@yahoo.com> wrote:
>> Here is the slave state info. I see marathon is registered as "slave_public" role and is configured with "default_accepted_resource_ roles" as "*"
>>
>> "slaves":[
>> {
>> "id":"69356344-e2c4-453d-baaf- 22df4a4cc430-S0",
>> "pid":"slave(1)@xxx.xxx.xxx. 100:5051",
>> "hostname":"xxx.xxx.xxx.100",
>> "registered_time":1481267726. 19244,
>> "resources":{
>> "disk":12099.0,
>> "mem":14863.0,
>> "gpus":0.0,
>> "cpus":4.0,
>> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
>> },
>> "used_resources":{
>> "disk":0.0,
>> "mem":0.0,
>> "gpus":0.0,
>> "cpus":0.0
>> },
>> "offered_resources":{
>> "disk":0.0,
>> "mem":0.0,
>> "gpus":0.0,
>> "cpus":0.0
>> },
>> "reserved_resources":{
>>
>> },
>> "unreserved_resources":{
>> "disk":12099.0,
>> "mem":14863.0,
>> "gpus":0.0,
>> "cpus":4.0,
>> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
>> },
>> "attributes":{
>>
>> },
>> "active":true,
>> "version":"1.0.1"
>> }
>> ],
>>
>> Regards
>> Vijay
>> On Friday, December 9, 2016 3:48 PM, Vinod Kone <vi...@apache.org> wrote:
>>
>>
>> How many resources does the agent register with the master? How many resources does spark task need?
>>
>> I'm guessing marathon is not registered with "test" role so it is only getting un-reserved resources which are not enough for spark task?
>>
>> On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <vi...@yahoo.com> wrote:
>> I have a standalone DCOS setup (Single node Vagrant VM running DCOS v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are running on same VM.
>>
>> Resource: 4 CPU, 16GB Memory, 20G Disk
>>
>> I have created a quota using new V1 API which creates a role "test" with resource constraints of 0.5 CPU and 1G Memory.
>>
>> When I try to deploy Spark package, Marathon receives the request but the task is in "waiting" state since it did not receive any offers from Master though I don't see any resource constraints from the hardware perspective.
>>
>> However, when I deleted the quota, Marathon is able to move forward with the deployment and Spark was deployed/up and running. I could see from the Mesos master logs that it had sent an offer to the Marathon framework.
>>
>> To debug the issue, I was trying to create a quota but this time did not provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to deploy Spark from DCOS UI, I could see Marathon getting offer from Master and able to deploy Spark without the need to delete the quota this time.
>>
>> Did anyone notice similar behavior?
>>
>> Regards
>> Vijay
>>
>>
>>
>
Re: Quota
Posted by Vijay Srinivasaraghavan <vi...@yahoo.com.INVALID>.
Hi Alex,
>>Granularity in the allocator is a single agent.
Does this mean if I have only one agent and the moment if I set any quota, the framework running on the agent will not be allocated with any resource?
From the logs, I don't get to see much details for the scenario when the quota is set following deploying a package through Marathon. However, when I remove the quota, I see the following message in master log "Allocating ports(*):[*****]; disk(*):....." "Sending 1 offers to framework XX (marathon) at scheduler-XXX"
RegardsVijay
On Sunday, December 11, 2016 9:05 AM, Alex Rukletsov <al...@mesosphere.io> wrote:
Granularity in the allocator is a single agent. Hence even though you set quota for 0.0001 CPU, at least one agent is "blocked". This is probably the reason why marathon is not getting offers. You can turn verbose master logs and check allocator messages to confirm. Alex.
On 10 Dec 2016 2:14 am, "Vijay" <vi...@yahoo.com.invalid> wrote:
The dispatcher needs 1cpu and 1G memory.
Regards,
Vijay
Sent from my iPhone
> On Dec 9, 2016, at 4:51 PM, Vinod Kone <vi...@apache.org> wrote:
>
> And how many resources does spark need?
>
>> On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan <vi...@yahoo.com> wrote:
>> Here is the slave state info. I see marathon is registered as "slave_public" role and is configured with "default_accepted_resource_ roles" as "*"
>>
>> "slaves":[
>> {
>> "id":"69356344-e2c4-453d-baaf- 22df4a4cc430-S0",
>> "pid":"slave(1)@xxx.xxx.xxx. 100:5051",
>> "hostname":"xxx.xxx.xxx.100",
>> "registered_time":1481267726. 19244,
>> "resources":{
>> "disk":12099.0,
>> "mem":14863.0,
>> "gpus":0.0,
>> "cpus":4.0,
>> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
>> },
>> "used_resources":{
>> "disk":0.0,
>> "mem":0.0,
>> "gpus":0.0,
>> "cpus":0.0
>> },
>> "offered_resources":{
>> "disk":0.0,
>> "mem":0.0,
>> "gpus":0.0,
>> "cpus":0.0
>> },
>> "reserved_resources":{
>>
>> },
>> "unreserved_resources":{
>> "disk":12099.0,
>> "mem":14863.0,
>> "gpus":0.0,
>> "cpus":4.0,
>> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
>> },
>> "attributes":{
>>
>> },
>> "active":true,
>> "version":"1.0.1"
>> }
>> ],
>>
>> Regards
>> Vijay
>> On Friday, December 9, 2016 3:48 PM, Vinod Kone <vi...@apache.org> wrote:
>>
>>
>> How many resources does the agent register with the master? How many resources does spark task need?
>>
>> I'm guessing marathon is not registered with "test" role so it is only getting un-reserved resources which are not enough for spark task?
>>
>> On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <vi...@yahoo.com> wrote:
>> I have a standalone DCOS setup (Single node Vagrant VM running DCOS v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are running on same VM.
>>
>> Resource: 4 CPU, 16GB Memory, 20G Disk
>>
>> I have created a quota using new V1 API which creates a role "test" with resource constraints of 0.5 CPU and 1G Memory.
>>
>> When I try to deploy Spark package, Marathon receives the request but the task is in "waiting" state since it did not receive any offers from Master though I don't see any resource constraints from the hardware perspective.
>>
>> However, when I deleted the quota, Marathon is able to move forward with the deployment and Spark was deployed/up and running. I could see from the Mesos master logs that it had sent an offer to the Marathon framework.
>>
>> To debug the issue, I was trying to create a quota but this time did not provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to deploy Spark from DCOS UI, I could see Marathon getting offer from Master and able to deploy Spark without the need to delete the quota this time.
>>
>> Did anyone notice similar behavior?
>>
>> Regards
>> Vijay
>>
>>
>>
>
Re: Quota
Posted by Alex Rukletsov <al...@mesosphere.io>.
Granularity in the allocator is a single agent. Hence even though you set
quota for 0.0001 CPU, at least one agent is "blocked". This is probably the
reason why marathon is not getting offers. You can turn verbose master logs
and check allocator messages to confirm.
Alex.
On 10 Dec 2016 2:14 am, "Vijay" <vi...@yahoo.com.invalid> wrote:
> The dispatcher needs 1cpu and 1G memory.
>
> Regards,
> Vijay
>
> Sent from my iPhone
>
> > On Dec 9, 2016, at 4:51 PM, Vinod Kone <vi...@apache.org> wrote:
> >
> > And how many resources does spark need?
> >
> >> On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan <
> vijikarthi@yahoo.com> wrote:
> >> Here is the slave state info. I see marathon is registered as
> "slave_public" role and is configured with "default_accepted_resource_roles"
> as "*"
> >>
> >> "slaves":[
> >> {
> >> "id":"69356344-e2c4-453d-baaf-22df4a4cc430-S0",
> >> "pid":"slave(1)@xxx.xxx.xxx.100:5051",
> >> "hostname":"xxx.xxx.xxx.100",
> >> "registered_time":1481267726.19244,
> >> "resources":{
> >> "disk":12099.0,
> >> "mem":14863.0,
> >> "gpus":0.0,
> >> "cpus":4.0,
> >> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079,
> 8082-8180, 8182-32000]"
> >> },
> >> "used_resources":{
> >> "disk":0.0,
> >> "mem":0.0,
> >> "gpus":0.0,
> >> "cpus":0.0
> >> },
> >> "offered_resources":{
> >> "disk":0.0,
> >> "mem":0.0,
> >> "gpus":0.0,
> >> "cpus":0.0
> >> },
> >> "reserved_resources":{
> >>
> >> },
> >> "unreserved_resources":{
> >> "disk":12099.0,
> >> "mem":14863.0,
> >> "gpus":0.0,
> >> "cpus":4.0,
> >> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079,
> 8082-8180, 8182-32000]"
> >> },
> >> "attributes":{
> >>
> >> },
> >> "active":true,
> >> "version":"1.0.1"
> >> }
> >> ],
> >>
> >> Regards
> >> Vijay
> >> On Friday, December 9, 2016 3:48 PM, Vinod Kone <vi...@apache.org>
> wrote:
> >>
> >>
> >> How many resources does the agent register with the master? How many
> resources does spark task need?
> >>
> >> I'm guessing marathon is not registered with "test" role so it is only
> getting un-reserved resources which are not enough for spark task?
> >>
> >> On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <
> vijikarthi@yahoo.com> wrote:
> >> I have a standalone DCOS setup (Single node Vagrant VM running DCOS
> v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are
> running on same VM.
> >>
> >> Resource: 4 CPU, 16GB Memory, 20G Disk
> >>
> >> I have created a quota using new V1 API which creates a role "test"
> with resource constraints of 0.5 CPU and 1G Memory.
> >>
> >> When I try to deploy Spark package, Marathon receives the request but
> the task is in "waiting" state since it did not receive any offers from
> Master though I don't see any resource constraints from the hardware
> perspective.
> >>
> >> However, when I deleted the quota, Marathon is able to move forward
> with the deployment and Spark was deployed/up and running. I could see from
> the Mesos master logs that it had sent an offer to the Marathon framework.
> >>
> >> To debug the issue, I was trying to create a quota but this time did
> not provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to
> deploy Spark from DCOS UI, I could see Marathon getting offer from Master
> and able to deploy Spark without the need to delete the quota this time.
> >>
> >> Did anyone notice similar behavior?
> >>
> >> Regards
> >> Vijay
> >>
> >>
> >>
> >
>
Re: Quota
Posted by Alex Rukletsov <al...@mesosphere.io>.
Granularity in the allocator is a single agent. Hence even though you set
quota for 0.0001 CPU, at least one agent is "blocked". This is probably the
reason why marathon is not getting offers. You can turn verbose master logs
and check allocator messages to confirm.
Alex.
On 10 Dec 2016 2:14 am, "Vijay" <vi...@yahoo.com.invalid> wrote:
> The dispatcher needs 1cpu and 1G memory.
>
> Regards,
> Vijay
>
> Sent from my iPhone
>
> > On Dec 9, 2016, at 4:51 PM, Vinod Kone <vi...@apache.org> wrote:
> >
> > And how many resources does spark need?
> >
> >> On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan <
> vijikarthi@yahoo.com> wrote:
> >> Here is the slave state info. I see marathon is registered as
> "slave_public" role and is configured with "default_accepted_resource_roles"
> as "*"
> >>
> >> "slaves":[
> >> {
> >> "id":"69356344-e2c4-453d-baaf-22df4a4cc430-S0",
> >> "pid":"slave(1)@xxx.xxx.xxx.100:5051",
> >> "hostname":"xxx.xxx.xxx.100",
> >> "registered_time":1481267726.19244,
> >> "resources":{
> >> "disk":12099.0,
> >> "mem":14863.0,
> >> "gpus":0.0,
> >> "cpus":4.0,
> >> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079,
> 8082-8180, 8182-32000]"
> >> },
> >> "used_resources":{
> >> "disk":0.0,
> >> "mem":0.0,
> >> "gpus":0.0,
> >> "cpus":0.0
> >> },
> >> "offered_resources":{
> >> "disk":0.0,
> >> "mem":0.0,
> >> "gpus":0.0,
> >> "cpus":0.0
> >> },
> >> "reserved_resources":{
> >>
> >> },
> >> "unreserved_resources":{
> >> "disk":12099.0,
> >> "mem":14863.0,
> >> "gpus":0.0,
> >> "cpus":4.0,
> >> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079,
> 8082-8180, 8182-32000]"
> >> },
> >> "attributes":{
> >>
> >> },
> >> "active":true,
> >> "version":"1.0.1"
> >> }
> >> ],
> >>
> >> Regards
> >> Vijay
> >> On Friday, December 9, 2016 3:48 PM, Vinod Kone <vi...@apache.org>
> wrote:
> >>
> >>
> >> How many resources does the agent register with the master? How many
> resources does spark task need?
> >>
> >> I'm guessing marathon is not registered with "test" role so it is only
> getting un-reserved resources which are not enough for spark task?
> >>
> >> On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <
> vijikarthi@yahoo.com> wrote:
> >> I have a standalone DCOS setup (Single node Vagrant VM running DCOS
> v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are
> running on same VM.
> >>
> >> Resource: 4 CPU, 16GB Memory, 20G Disk
> >>
> >> I have created a quota using new V1 API which creates a role "test"
> with resource constraints of 0.5 CPU and 1G Memory.
> >>
> >> When I try to deploy Spark package, Marathon receives the request but
> the task is in "waiting" state since it did not receive any offers from
> Master though I don't see any resource constraints from the hardware
> perspective.
> >>
> >> However, when I deleted the quota, Marathon is able to move forward
> with the deployment and Spark was deployed/up and running. I could see from
> the Mesos master logs that it had sent an offer to the Marathon framework.
> >>
> >> To debug the issue, I was trying to create a quota but this time did
> not provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to
> deploy Spark from DCOS UI, I could see Marathon getting offer from Master
> and able to deploy Spark without the need to delete the quota this time.
> >>
> >> Did anyone notice similar behavior?
> >>
> >> Regards
> >> Vijay
> >>
> >>
> >>
> >
>
Re: Quota
Posted by Vijay <vi...@yahoo.com>.
The dispatcher needs 1cpu and 1G memory.
Regards,
Vijay
Sent from my iPhone
> On Dec 9, 2016, at 4:51 PM, Vinod Kone <vi...@apache.org> wrote:
>
> And how many resources does spark need?
>
>> On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan <vi...@yahoo.com> wrote:
>> Here is the slave state info. I see marathon is registered as "slave_public" role and is configured with "default_accepted_resource_roles" as "*"
>>
>> "slaves":[
>> {
>> "id":"69356344-e2c4-453d-baaf-22df4a4cc430-S0",
>> "pid":"slave(1)@xxx.xxx.xxx.100:5051",
>> "hostname":"xxx.xxx.xxx.100",
>> "registered_time":1481267726.19244,
>> "resources":{
>> "disk":12099.0,
>> "mem":14863.0,
>> "gpus":0.0,
>> "cpus":4.0,
>> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
>> },
>> "used_resources":{
>> "disk":0.0,
>> "mem":0.0,
>> "gpus":0.0,
>> "cpus":0.0
>> },
>> "offered_resources":{
>> "disk":0.0,
>> "mem":0.0,
>> "gpus":0.0,
>> "cpus":0.0
>> },
>> "reserved_resources":{
>>
>> },
>> "unreserved_resources":{
>> "disk":12099.0,
>> "mem":14863.0,
>> "gpus":0.0,
>> "cpus":4.0,
>> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
>> },
>> "attributes":{
>>
>> },
>> "active":true,
>> "version":"1.0.1"
>> }
>> ],
>>
>> Regards
>> Vijay
>> On Friday, December 9, 2016 3:48 PM, Vinod Kone <vi...@apache.org> wrote:
>>
>>
>> How many resources does the agent register with the master? How many resources does spark task need?
>>
>> I'm guessing marathon is not registered with "test" role so it is only getting un-reserved resources which are not enough for spark task?
>>
>> On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <vi...@yahoo.com> wrote:
>> I have a standalone DCOS setup (Single node Vagrant VM running DCOS v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are running on same VM.
>>
>> Resource: 4 CPU, 16GB Memory, 20G Disk
>>
>> I have created a quota using new V1 API which creates a role "test" with resource constraints of 0.5 CPU and 1G Memory.
>>
>> When I try to deploy Spark package, Marathon receives the request but the task is in "waiting" state since it did not receive any offers from Master though I don't see any resource constraints from the hardware perspective.
>>
>> However, when I deleted the quota, Marathon is able to move forward with the deployment and Spark was deployed/up and running. I could see from the Mesos master logs that it had sent an offer to the Marathon framework.
>>
>> To debug the issue, I was trying to create a quota but this time did not provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to deploy Spark from DCOS UI, I could see Marathon getting offer from Master and able to deploy Spark without the need to delete the quota this time.
>>
>> Did anyone notice similar behavior?
>>
>> Regards
>> Vijay
>>
>>
>>
>
Re: Quota
Posted by Vijay <vi...@yahoo.com.INVALID>.
The dispatcher needs 1cpu and 1G memory.
Regards,
Vijay
Sent from my iPhone
> On Dec 9, 2016, at 4:51 PM, Vinod Kone <vi...@apache.org> wrote:
>
> And how many resources does spark need?
>
>> On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan <vi...@yahoo.com> wrote:
>> Here is the slave state info. I see marathon is registered as "slave_public" role and is configured with "default_accepted_resource_roles" as "*"
>>
>> "slaves":[
>> {
>> "id":"69356344-e2c4-453d-baaf-22df4a4cc430-S0",
>> "pid":"slave(1)@xxx.xxx.xxx.100:5051",
>> "hostname":"xxx.xxx.xxx.100",
>> "registered_time":1481267726.19244,
>> "resources":{
>> "disk":12099.0,
>> "mem":14863.0,
>> "gpus":0.0,
>> "cpus":4.0,
>> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
>> },
>> "used_resources":{
>> "disk":0.0,
>> "mem":0.0,
>> "gpus":0.0,
>> "cpus":0.0
>> },
>> "offered_resources":{
>> "disk":0.0,
>> "mem":0.0,
>> "gpus":0.0,
>> "cpus":0.0
>> },
>> "reserved_resources":{
>>
>> },
>> "unreserved_resources":{
>> "disk":12099.0,
>> "mem":14863.0,
>> "gpus":0.0,
>> "cpus":4.0,
>> "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
>> },
>> "attributes":{
>>
>> },
>> "active":true,
>> "version":"1.0.1"
>> }
>> ],
>>
>> Regards
>> Vijay
>> On Friday, December 9, 2016 3:48 PM, Vinod Kone <vi...@apache.org> wrote:
>>
>>
>> How many resources does the agent register with the master? How many resources does spark task need?
>>
>> I'm guessing marathon is not registered with "test" role so it is only getting un-reserved resources which are not enough for spark task?
>>
>> On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <vi...@yahoo.com> wrote:
>> I have a standalone DCOS setup (Single node Vagrant VM running DCOS v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are running on same VM.
>>
>> Resource: 4 CPU, 16GB Memory, 20G Disk
>>
>> I have created a quota using new V1 API which creates a role "test" with resource constraints of 0.5 CPU and 1G Memory.
>>
>> When I try to deploy Spark package, Marathon receives the request but the task is in "waiting" state since it did not receive any offers from Master though I don't see any resource constraints from the hardware perspective.
>>
>> However, when I deleted the quota, Marathon is able to move forward with the deployment and Spark was deployed/up and running. I could see from the Mesos master logs that it had sent an offer to the Marathon framework.
>>
>> To debug the issue, I was trying to create a quota but this time did not provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to deploy Spark from DCOS UI, I could see Marathon getting offer from Master and able to deploy Spark without the need to delete the quota this time.
>>
>> Did anyone notice similar behavior?
>>
>> Regards
>> Vijay
>>
>>
>>
>
Re: Quota
Posted by Vinod Kone <vi...@apache.org>.
And how many resources does spark need?
On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan <
vijikarthi@yahoo.com> wrote:
> Here is the slave state info. I see marathon is registered as
> "slave_public" role and is configured with "default_accepted_resource_roles"
> as "*"
>
> "slaves":[
> {
> "id":"69356344-e2c4-453d-baaf-22df4a4cc430-S0",
> "pid":"slave(1)@xxx.xxx.xxx.100:5051",
> "hostname":"xxx.xxx.xxx.100",
> "registered_time":1481267726.19244,
> "resources":{
> "disk":12099.0,
> "mem":14863.0,
> "gpus":0.0,
> "cpus":4.0,
> "ports":"[1025-2180, 2182-3887, 3889-5049,
> 5052-8079, 8082-8180, 8182-32000]"
> },
> "used_resources":{
> "disk":0.0,
> "mem":0.0,
> "gpus":0.0,
> "cpus":0.0
> },
> "offered_resources":{
> "disk":0.0,
> "mem":0.0,
> "gpus":0.0,
> "cpus":0.0
> },
> "reserved_resources":{
>
> },
> "unreserved_resources":{
> "disk":12099.0,
> "mem":14863.0,
> "gpus":0.0,
> "cpus":4.0,
> "ports":"[1025-2180, 2182-3887, 3889-5049,
> 5052-8079, 8082-8180, 8182-32000]"
> },
> "attributes":{
>
> },
> "active":true,
> "version":"1.0.1"
> }
> ],
>
> Regards
> Vijay
> On Friday, December 9, 2016 3:48 PM, Vinod Kone <vi...@apache.org>
> wrote:
>
>
> How many resources does the agent register with the master? How many
> resources does spark task need?
>
> I'm guessing marathon is not registered with "test" role so it is only
> getting un-reserved resources which are not enough for spark task?
>
> On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <
> vijikarthi@yahoo.com> wrote:
>
> I have a standalone DCOS setup (Single node Vagrant VM running DCOS
> v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are
> running on same VM.
>
> Resource: 4 CPU, 16GB Memory, 20G Disk
>
> I have created a quota using new V1 API which creates a role "test" with
> resource constraints of 0.5 CPU and 1G Memory.
>
> When I try to deploy Spark package, Marathon receives the request but the
> task is in "waiting" state since it did not receive any offers from Master
> though I don't see any resource constraints from the hardware perspective.
>
> However, when I deleted the quota, Marathon is able to move forward with
> the deployment and Spark was deployed/up and running. I could see from the
> Mesos master logs that it had sent an offer to the Marathon framework.
>
> To debug the issue, I was trying to create a quota but this time did not
> provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to
> deploy Spark from DCOS UI, I could see Marathon getting offer from Master
> and able to deploy Spark without the need to delete the quota this time.
>
> Did anyone notice similar behavior?
>
> Regards
> Vijay
>
>
>
>
>
Re: Quota
Posted by Vinod Kone <vi...@apache.org>.
And how many resources does spark need?
On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan <
vijikarthi@yahoo.com> wrote:
> Here is the slave state info. I see marathon is registered as
> "slave_public" role and is configured with "default_accepted_resource_roles"
> as "*"
>
> "slaves":[
> {
> "id":"69356344-e2c4-453d-baaf-22df4a4cc430-S0",
> "pid":"slave(1)@xxx.xxx.xxx.100:5051",
> "hostname":"xxx.xxx.xxx.100",
> "registered_time":1481267726.19244,
> "resources":{
> "disk":12099.0,
> "mem":14863.0,
> "gpus":0.0,
> "cpus":4.0,
> "ports":"[1025-2180, 2182-3887, 3889-5049,
> 5052-8079, 8082-8180, 8182-32000]"
> },
> "used_resources":{
> "disk":0.0,
> "mem":0.0,
> "gpus":0.0,
> "cpus":0.0
> },
> "offered_resources":{
> "disk":0.0,
> "mem":0.0,
> "gpus":0.0,
> "cpus":0.0
> },
> "reserved_resources":{
>
> },
> "unreserved_resources":{
> "disk":12099.0,
> "mem":14863.0,
> "gpus":0.0,
> "cpus":4.0,
> "ports":"[1025-2180, 2182-3887, 3889-5049,
> 5052-8079, 8082-8180, 8182-32000]"
> },
> "attributes":{
>
> },
> "active":true,
> "version":"1.0.1"
> }
> ],
>
> Regards
> Vijay
> On Friday, December 9, 2016 3:48 PM, Vinod Kone <vi...@apache.org>
> wrote:
>
>
> How many resources does the agent register with the master? How many
> resources does spark task need?
>
> I'm guessing marathon is not registered with "test" role so it is only
> getting un-reserved resources which are not enough for spark task?
>
> On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <
> vijikarthi@yahoo.com> wrote:
>
> I have a standalone DCOS setup (Single node Vagrant VM running DCOS
> v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are
> running on same VM.
>
> Resource: 4 CPU, 16GB Memory, 20G Disk
>
> I have created a quota using new V1 API which creates a role "test" with
> resource constraints of 0.5 CPU and 1G Memory.
>
> When I try to deploy Spark package, Marathon receives the request but the
> task is in "waiting" state since it did not receive any offers from Master
> though I don't see any resource constraints from the hardware perspective.
>
> However, when I deleted the quota, Marathon is able to move forward with
> the deployment and Spark was deployed/up and running. I could see from the
> Mesos master logs that it had sent an offer to the Marathon framework.
>
> To debug the issue, I was trying to create a quota but this time did not
> provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to
> deploy Spark from DCOS UI, I could see Marathon getting offer from Master
> and able to deploy Spark without the need to delete the quota this time.
>
> Did anyone notice similar behavior?
>
> Regards
> Vijay
>
>
>
>
>
Re: Quota
Posted by Vijay Srinivasaraghavan <vi...@yahoo.com>.
Here is the slave state info. I see marathon is registered as "slave_public" role and is configured with "default_accepted_resource_roles" as "*"
"slaves":[
{
"id":"69356344-e2c4-453d-baaf-22df4a4cc430-S0",
"pid":"slave(1)@xxx.xxx.xxx.100:5051",
"hostname":"xxx.xxx.xxx.100",
"registered_time":1481267726.19244,
"resources":{
"disk":12099.0,
"mem":14863.0,
"gpus":0.0,
"cpus":4.0,
"ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
},
"used_resources":{
"disk":0.0,
"mem":0.0,
"gpus":0.0,
"cpus":0.0
},
"offered_resources":{
"disk":0.0,
"mem":0.0,
"gpus":0.0,
"cpus":0.0
},
"reserved_resources":{
},
"unreserved_resources":{
"disk":12099.0,
"mem":14863.0,
"gpus":0.0,
"cpus":4.0,
"ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
},
"attributes":{
},
"active":true,
"version":"1.0.1"
}
],
RegardsVijay On Friday, December 9, 2016 3:48 PM, Vinod Kone <vi...@apache.org> wrote:
How many resources does the agent register with the master? How many resources does spark task need?
I'm guessing marathon is not registered with "test" role so it is only getting un-reserved resources which are not enough for spark task?
On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <vi...@yahoo.com> wrote:
I have a standalone DCOS setup (Single node Vagrant VM running DCOS v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are running on same VM.
Resource: 4 CPU, 16GB Memory, 20G Disk
I have created a quota using new V1 API which creates a role "test" with resource constraints of 0.5 CPU and 1G Memory.
When I try to deploy Spark package, Marathon receives the request but the task is in "waiting" state since it did not receive any offers from Master though I don't see any resource constraints from the hardware perspective.
However, when I deleted the quota, Marathon is able to move forward with the deployment and Spark was deployed/up and running. I could see from the Mesos master logs that it had sent an offer to the Marathon framework.
To debug the issue, I was trying to create a quota but this time did not provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to deploy Spark from DCOS UI, I could see Marathon getting offer from Master and able to deploy Spark without the need to delete the quota this time.
Did anyone notice similar behavior?
RegardsVijay
Re: Quota
Posted by Vijay Srinivasaraghavan <vi...@yahoo.com.INVALID>.
Here is the slave state info. I see marathon is registered as "slave_public" role and is configured with "default_accepted_resource_roles" as "*"
"slaves":[
{
"id":"69356344-e2c4-453d-baaf-22df4a4cc430-S0",
"pid":"slave(1)@xxx.xxx.xxx.100:5051",
"hostname":"xxx.xxx.xxx.100",
"registered_time":1481267726.19244,
"resources":{
"disk":12099.0,
"mem":14863.0,
"gpus":0.0,
"cpus":4.0,
"ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
},
"used_resources":{
"disk":0.0,
"mem":0.0,
"gpus":0.0,
"cpus":0.0
},
"offered_resources":{
"disk":0.0,
"mem":0.0,
"gpus":0.0,
"cpus":0.0
},
"reserved_resources":{
},
"unreserved_resources":{
"disk":12099.0,
"mem":14863.0,
"gpus":0.0,
"cpus":4.0,
"ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]"
},
"attributes":{
},
"active":true,
"version":"1.0.1"
}
],
RegardsVijay On Friday, December 9, 2016 3:48 PM, Vinod Kone <vi...@apache.org> wrote:
How many resources does the agent register with the master? How many resources does spark task need?
I'm guessing marathon is not registered with "test" role so it is only getting un-reserved resources which are not enough for spark task?
On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <vi...@yahoo.com> wrote:
I have a standalone DCOS setup (Single node Vagrant VM running DCOS v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are running on same VM.
Resource: 4 CPU, 16GB Memory, 20G Disk
I have created a quota using new V1 API which creates a role "test" with resource constraints of 0.5 CPU and 1G Memory.
When I try to deploy Spark package, Marathon receives the request but the task is in "waiting" state since it did not receive any offers from Master though I don't see any resource constraints from the hardware perspective.
However, when I deleted the quota, Marathon is able to move forward with the deployment and Spark was deployed/up and running. I could see from the Mesos master logs that it had sent an offer to the Marathon framework.
To debug the issue, I was trying to create a quota but this time did not provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to deploy Spark from DCOS UI, I could see Marathon getting offer from Master and able to deploy Spark without the need to delete the quota this time.
Did anyone notice similar behavior?
RegardsVijay
Re: Quota
Posted by Vinod Kone <vi...@apache.org>.
How many resources does the agent register with the master? How many
resources does spark task need?
I'm guessing marathon is not registered with "test" role so it is only
getting un-reserved resources which are not enough for spark task?
On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <
vijikarthi@yahoo.com> wrote:
> I have a standalone DCOS setup (Single node Vagrant VM running DCOS
> v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are
> running on same VM.
>
> Resource: 4 CPU, 16GB Memory, 20G Disk
>
> I have created a quota using new V1 API which creates a role "test" with
> resource constraints of 0.5 CPU and 1G Memory.
>
> When I try to deploy Spark package, Marathon receives the request but the
> task is in "waiting" state since it did not receive any offers from Master
> though I don't see any resource constraints from the hardware perspective.
>
> However, when I deleted the quota, Marathon is able to move forward with
> the deployment and Spark was deployed/up and running. I could see from the
> Mesos master logs that it had sent an offer to the Marathon framework.
>
> To debug the issue, I was trying to create a quota but this time did not
> provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to
> deploy Spark from DCOS UI, I could see Marathon getting offer from Master
> and able to deploy Spark without the need to delete the quota this time.
>
> Did anyone notice similar behavior?
>
> Regards
> Vijay
>
Re: Quota
Posted by Vinod Kone <vi...@apache.org>.
How many resources does the agent register with the master? How many
resources does spark task need?
I'm guessing marathon is not registered with "test" role so it is only
getting un-reserved resources which are not enough for spark task?
On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <
vijikarthi@yahoo.com> wrote:
> I have a standalone DCOS setup (Single node Vagrant VM running DCOS
> v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are
> running on same VM.
>
> Resource: 4 CPU, 16GB Memory, 20G Disk
>
> I have created a quota using new V1 API which creates a role "test" with
> resource constraints of 0.5 CPU and 1G Memory.
>
> When I try to deploy Spark package, Marathon receives the request but the
> task is in "waiting" state since it did not receive any offers from Master
> though I don't see any resource constraints from the hardware perspective.
>
> However, when I deleted the quota, Marathon is able to move forward with
> the deployment and Spark was deployed/up and running. I could see from the
> Mesos master logs that it had sent an offer to the Marathon framework.
>
> To debug the issue, I was trying to create a quota but this time did not
> provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to
> deploy Spark from DCOS UI, I could see Marathon getting offer from Master
> and able to deploy Spark without the need to delete the quota this time.
>
> Did anyone notice similar behavior?
>
> Regards
> Vijay
>