You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Tobias Pfeiffer <tg...@preferred.jp> on 2016/08/31 11:16:36 UTC
Low-latency task submission
Hi,
as written in a previous thread, I am trying to build a system where users
submit code and this code is run in a Docker container. I want to keep
latency low, so the time between "user submits request" and "task is
submitted" should be short.
First I tried to (a) buffer arriving tasks in an internal queue and
whenever `resourceOffers()` was called, process that queue. All not-used
offers were declined. That worked, but the time between resource offerings
was rather long, so that I tried to find a different way.
The next try (b) was to instead buffer the offers I received and process
new requests immediately from that buffer. However, the drawbacks were that
1) no other frameworks received those resource offers while they are in my
buffer, and 2) when other frameworks finished a task, a lot of resource
offers for the same slave piled up (like "0.5 CPUs and 500 MB memory", then
"0.8 CPUs and 0 MB memory" etc.).
I guess once I have understood these mechanisms I can work around them in
my scheduler and do *something* (like "buffer two resource offers and
decline the rest" or so), but is there any best practice approach for that?
Thanks,
Tobias
Re: Low-latency task submission
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi Sharma,
thank you for your feedback. I had also considered caching "just a couple
of offers" while declining the rest, so I guess that's one possible
approach. Also thanks for pointing me to Fenzo, I will give it a good look
and consider using it in my scheduler.
Thanks,
Tobias
On Wed, Aug 31, 2016 at 10:48 PM, Sharma Podila <sp...@netflix.com> wrote:
> Tobias, we have taken the approach of holding on to "some" offers to
> support fast launch of tasks while being a "good citizen" of a cluster by
> not holding on to all unused offers for too long. We routinely have tasks
> launched within a low number of milliSeconds after submission. This is
> supported in our open source Fenzo scheduling library. While I don't want
> to assume you can use Fenzo for your scheduler (although, you should be
> able to if your framework runs on the JVM), you may look at Fenzo
> documentation to understand the approach we have taken. Specifically, look
> at the JavaDocs for OfferExpirySecs
> <http://netflix.github.io/Fenzo/fenzo-core/com/netflix/fenzo/TaskScheduler.Builder.html#withLeaseOfferExpirySecs-long->,
> maxOffersToReject
> <http://netflix.github.io/Fenzo/fenzo-core/com/netflix/fenzo/TaskScheduler.Builder.html#withMaxOffersToReject-int->,
> and rejectAllExpiredOffers
> <http://netflix.github.io/Fenzo/fenzo-core/com/netflix/fenzo/TaskScheduler.Builder.html#withRejectAllExpiredOffers-->.
> Fenzo is being used in multi-framework clusters using these controls.
>
> On Wed, Aug 31, 2016 at 2:25 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
>
>> Hi Alexander,
>>
>> thanks for your suggestion.
>>
>> On Wed, Aug 31, 2016 at 9:11 PM, Alexander Rojas <alexander@mesosphere.io
>> > wrote:
>>>
>>> There are some things you can try, one would be to change the default
>>> value of the master’s `allocation_interval` flag, which is 1. This will
>>> increase the rate at which offers are being generated.
>>>
>>
>> I see. Do you think that in general this is the right approach,
>> processing all tasks that have piled up in the last second within the
>> `resourceOffers()` method?
>> Also, I read somewhere that on accepting an offer, all other offers will
>> be declined; but is this only true for offers from the same slave?
>>
>> Thanks,
>> Tobias
>>
>>
>>
>
Re: Low-latency task submission
Posted by Sharma Podila <sp...@netflix.com>.
Tobias, we have taken the approach of holding on to "some" offers to
support fast launch of tasks while being a "good citizen" of a cluster by
not holding on to all unused offers for too long. We routinely have tasks
launched within a low number of milliSeconds after submission. This is
supported in our open source Fenzo scheduling library. While I don't want
to assume you can use Fenzo for your scheduler (although, you should be
able to if your framework runs on the JVM), you may look at Fenzo
documentation to understand the approach we have taken. Specifically, look
at the JavaDocs for OfferExpirySecs
<http://netflix.github.io/Fenzo/fenzo-core/com/netflix/fenzo/TaskScheduler.Builder.html#withLeaseOfferExpirySecs-long->,
maxOffersToReject
<http://netflix.github.io/Fenzo/fenzo-core/com/netflix/fenzo/TaskScheduler.Builder.html#withMaxOffersToReject-int->,
and rejectAllExpiredOffers
<http://netflix.github.io/Fenzo/fenzo-core/com/netflix/fenzo/TaskScheduler.Builder.html#withRejectAllExpiredOffers-->.
Fenzo is being used in multi-framework clusters using these controls.
On Wed, Aug 31, 2016 at 2:25 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi Alexander,
>
> thanks for your suggestion.
>
> On Wed, Aug 31, 2016 at 9:11 PM, Alexander Rojas <al...@mesosphere.io>
> wrote:
>>
>> There are some things you can try, one would be to change the default
>> value of the master’s `allocation_interval` flag, which is 1. This will
>> increase the rate at which offers are being generated.
>>
>
> I see. Do you think that in general this is the right approach, processing
> all tasks that have piled up in the last second within the
> `resourceOffers()` method?
> Also, I read somewhere that on accepting an offer, all other offers will
> be declined; but is this only true for offers from the same slave?
>
> Thanks,
> Tobias
>
>
>
Re: Low-latency task submission
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi Alexander,
thanks for your suggestion.
On Wed, Aug 31, 2016 at 9:11 PM, Alexander Rojas <al...@mesosphere.io>
wrote:
>
> There are some things you can try, one would be to change the default
> value of the master’s `allocation_interval` flag, which is 1. This will
> increase the rate at which offers are being generated.
>
I see. Do you think that in general this is the right approach, processing
all tasks that have piled up in the last second within the
`resourceOffers()` method?
Also, I read somewhere that on accepting an offer, all other offers will be
declined; but is this only true for offers from the same slave?
Thanks,
Tobias
Re: Low-latency task submission
Posted by Alexander Rojas <al...@mesosphere.io>.
Hi Tobias,
There are some things you can try, one would be to change the default value of the master’s `allocation_interval` flag, which is 1. This will increase the rate at which offers are being generated.
> On 31 Aug 2016, at 13:16, Tobias Pfeiffer <tg...@preferred.jp> wrote:
>
> Hi,
>
> as written in a previous thread, I am trying to build a system where users submit code and this code is run in a Docker container. I want to keep latency low, so the time between "user submits request" and "task is submitted" should be short.
> First I tried to (a) buffer arriving tasks in an internal queue and whenever `resourceOffers()` was called, process that queue. All not-used offers were declined. That worked, but the time between resource offerings was rather long, so that I tried to find a different way.
> The next try (b) was to instead buffer the offers I received and process new requests immediately from that buffer. However, the drawbacks were that 1) no other frameworks received those resource offers while they are in my buffer, and 2) when other frameworks finished a task, a lot of resource offers for the same slave piled up (like "0.5 CPUs and 500 MB memory", then "0.8 CPUs and 0 MB memory" etc.).
> I guess once I have understood these mechanisms I can work around them in my scheduler and do *something* (like "buffer two resource offers and decline the rest" or so), but is there any best practice approach for that?
>
> Thanks,
> Tobias
>