resource reservation through a long running service

[ricky l <ri...@gmail.com>]

hi,

I have a question about mesos behavior when a long running service job
(such as web-server or traditional RDBMS) accepts offers and run for a long
period of time with sporadic resource usage. Assuming that a long running
job gets assigned 30% of total available resources, I think Mesos will
assume that the 30% of resources are being used and do not give offers to
others for the 30% even if the framework is running very long without
actually using the resources. Is this correct understanding?  If this is
true, the overall system utilization can get lower even if there are many
other frameworks waiting for offers. How does Mesos handle this type of
situation? thx.

Re: resource reservation through a long running service

[ricky l <ri...@gmail.com>]

The resource over-subscription in a platform like Mesos is the thing that I
am also interested in. Looking forward to seeing progresses for the feature
- I will also update from my-side if I can observe something interesting.
thx.


On Mon, Nov 25, 2013 at 5:00 PM, Benjamin Mahler
<be...@gmail.com>wrote:

> In the longer term, utilization will be improved through
> "oversubscription" of the slave's resources. This means, giving out
> resources inside existing resource allocations if they remain unused. These
> "over-subscribed" resources would be revokable at any time.
>
>
> On Mon, Nov 25, 2013 at 12:29 PM, ricky l <ri...@gmail.com> wrote:
>
>> Thanks, Vinod. From the system utilization perspective, it might not be a
>> good idea to run a long-running service job (with sporadic workloads) that
>> might reserve a large portion of resources.
>> For the new feature, It seems like to deal with a case when a framework
>> is taking long to make accept/decline decision. Looking forward to that
>> feature.
>>
>> Thanks.
>>
>>
>> On Mon, Nov 25, 2013 at 1:12 PM, Vinod Kone <vi...@gmail.com> wrote:
>>
>>> Your understanding is correct. Currently, if a framework holds on to an
>>> offer then it might impact other frameworks. If you want to guarantee that
>>> each of your frameworks can receive certain amount of resources, you could
>>> "reserve" resources to frameworks (roles). See the "--resources" flag on
>>> the slave and "roles" flag on the master.
>>>
>>> In the near future, we also plan to implement a feature where the same
>>> offer is sent to multiple frameworks; if one of the frameworks uses up the
>>> offer to launch that offer is rescinded on another frameworks.
>>>
>>> HTH,
>>>
>>>
>>> On Mon, Nov 25, 2013 at 8:08 AM, ricky l <ri...@gmail.com> wrote:
>>>
>>>> hi,
>>>>
>>>> I have a question about mesos behavior when a long running service job
>>>> (such as web-server or traditional RDBMS) accepts offers and run for a long
>>>> period of time with sporadic resource usage. Assuming that a long running
>>>> job gets assigned 30% of total available resources, I think Mesos will
>>>> assume that the 30% of resources are being used and do not give offers to
>>>> others for the 30% even if the framework is running very long without
>>>> actually using the resources. Is this correct understanding?  If this is
>>>> true, the overall system utilization can get lower even if there are many
>>>> other frameworks waiting for offers. How does Mesos handle this type of
>>>> situation? thx.
>>>>
>>>
>>>
>>
>

Re: resource reservation through a long running service

[Benjamin Mahler <be...@gmail.com>]

In the longer term, utilization will be improved through "oversubscription"
of the slave's resources. This means, giving out resources inside existing
resource allocations if they remain unused. These "over-subscribed"
resources would be revokable at any time.


On Mon, Nov 25, 2013 at 12:29 PM, ricky l <ri...@gmail.com> wrote:

> Thanks, Vinod. From the system utilization perspective, it might not be a
> good idea to run a long-running service job (with sporadic workloads) that
> might reserve a large portion of resources.
> For the new feature, It seems like to deal with a case when a framework is
> taking long to make accept/decline decision. Looking forward to that
> feature.
>
> Thanks.
>
>
> On Mon, Nov 25, 2013 at 1:12 PM, Vinod Kone <vi...@gmail.com> wrote:
>
>> Your understanding is correct. Currently, if a framework holds on to an
>> offer then it might impact other frameworks. If you want to guarantee that
>> each of your frameworks can receive certain amount of resources, you could
>> "reserve" resources to frameworks (roles). See the "--resources" flag on
>> the slave and "roles" flag on the master.
>>
>> In the near future, we also plan to implement a feature where the same
>> offer is sent to multiple frameworks; if one of the frameworks uses up the
>> offer to launch that offer is rescinded on another frameworks.
>>
>> HTH,
>>
>>
>> On Mon, Nov 25, 2013 at 8:08 AM, ricky l <ri...@gmail.com> wrote:
>>
>>> hi,
>>>
>>> I have a question about mesos behavior when a long running service job
>>> (such as web-server or traditional RDBMS) accepts offers and run for a long
>>> period of time with sporadic resource usage. Assuming that a long running
>>> job gets assigned 30% of total available resources, I think Mesos will
>>> assume that the 30% of resources are being used and do not give offers to
>>> others for the 30% even if the framework is running very long without
>>> actually using the resources. Is this correct understanding?  If this is
>>> true, the overall system utilization can get lower even if there are many
>>> other frameworks waiting for offers. How does Mesos handle this type of
>>> situation? thx.
>>>
>>
>>
>

Re: resource reservation through a long running service

[ricky l <ri...@gmail.com>]

Thanks, Vinod. From the system utilization perspective, it might not be a
good idea to run a long-running service job (with sporadic workloads) that
might reserve a large portion of resources.
For the new feature, It seems like to deal with a case when a framework is
taking long to make accept/decline decision. Looking forward to that
feature.

Thanks.


On Mon, Nov 25, 2013 at 1:12 PM, Vinod Kone <vi...@gmail.com> wrote:

> Your understanding is correct. Currently, if a framework holds on to an
> offer then it might impact other frameworks. If you want to guarantee that
> each of your frameworks can receive certain amount of resources, you could
> "reserve" resources to frameworks (roles). See the "--resources" flag on
> the slave and "roles" flag on the master.
>
> In the near future, we also plan to implement a feature where the same
> offer is sent to multiple frameworks; if one of the frameworks uses up the
> offer to launch that offer is rescinded on another frameworks.
>
> HTH,
>
>
> On Mon, Nov 25, 2013 at 8:08 AM, ricky l <ri...@gmail.com> wrote:
>
>> hi,
>>
>> I have a question about mesos behavior when a long running service job
>> (such as web-server or traditional RDBMS) accepts offers and run for a long
>> period of time with sporadic resource usage. Assuming that a long running
>> job gets assigned 30% of total available resources, I think Mesos will
>> assume that the 30% of resources are being used and do not give offers to
>> others for the 30% even if the framework is running very long without
>> actually using the resources. Is this correct understanding?  If this is
>> true, the overall system utilization can get lower even if there are many
>> other frameworks waiting for offers. How does Mesos handle this type of
>> situation? thx.
>>
>
>

Re: resource reservation through a long running service

[Vinod Kone <vi...@gmail.com>]

Your understanding is correct. Currently, if a framework holds on to an
offer then it might impact other frameworks. If you want to guarantee that
each of your frameworks can receive certain amount of resources, you could
"reserve" resources to frameworks (roles). See the "--resources" flag on
the slave and "roles" flag on the master.

In the near future, we also plan to implement a feature where the same
offer is sent to multiple frameworks; if one of the frameworks uses up the
offer to launch that offer is rescinded on another frameworks.

HTH,


On Mon, Nov 25, 2013 at 8:08 AM, ricky l <ri...@gmail.com> wrote:

> hi,
>
> I have a question about mesos behavior when a long running service job
> (such as web-server or traditional RDBMS) accepts offers and run for a long
> period of time with sporadic resource usage. Assuming that a long running
> job gets assigned 30% of total available resources, I think Mesos will
> assume that the 30% of resources are being used and do not give offers to
> others for the 30% even if the framework is running very long without
> actually using the resources. Is this correct understanding?  If this is
> true, the overall system utilization can get lower even if there are many
> other frameworks waiting for offers. How does Mesos handle this type of
> situation? thx.
>