You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Andrii Biletskyi <an...@stealth.ly> on 2016/01/22 15:49:15 UTC
Get Task's labels on reconciliation
Hi,
I'm using Mesos 0.25. Is there any way to get some more information about
the task (like label, taskData)
when requesting task reconciliation? Right now i see that driver returns
only taskId, slaveId, status, timestamp.
Let me describe my use case so you have a better understanding.
I'm trying to implement HA framework that will manage a cluster of database
nodes.
The scheduler will be run on marathon, and since I'm managing database
nodes I want to store mesos state
there to tolerate scheduler failures. So the workflow is the following:
User starts Scheduler on marathon providing some generated frameworkId.
Scheduler registers itself with the given
frameworkId and immediately starts reconciliation. If it's a fresh start
nothing has to be done - we are ready to accept
"add database node" commands. If it's a failover the idea was to inspect
each task state returned by the reconciliation
procedure, extract labels to distinguish whether it's a node, responsible
for storing mesos state, then extract task data
where we store node's host:port. With that you can initialize your storage
and may operate normally.
Thanks in advance,
Andrii
Re: Get Task's labels on reconciliation
Posted by Benjamin Mahler <bm...@apache.org>.
Ah I'm sorry, not sure if the concept of "latest labels" applies in the
same way as "latest health" information for a task, since health is an
active property of the task, whereas labels can be specific to updates.
On Fri, Jan 29, 2016 at 12:09 PM, Benjamin Mahler <bm...@apache.org>
wrote:
> I see that the following was filed:
> https://issues.apache.org/jira/browse/MESOS-4477
>
> But this sounds like a bug: if the master knows about the task during
> reconciliation, labels should be sent. For example we had this same bug for
> health information:
> https://issues.apache.org/jira/browse/MESOS-2206
>
> On Fri, Jan 22, 2016 at 2:10 PM, Andrii Biletskyi <
> andrii.biletskyi@stealth.ly> wrote:
>
>> Okay, thanks for your answer. I will rework my algorithm so it doesn't
>> rely on task labels/data.
>>
>> Thanks,
>> Andrii
>>
>> On Fri, Jan 22, 2016 at 11:49 PM, Vinod Kone <vi...@apache.org>
>> wrote:
>>
>>>
>>> On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi <
>>> andrii.biletskyi@stealth.ly> wrote:
>>>
>>>> 8. Scheduler.statusUpdate callback is called for
>>>> taskId=t1;state=Running. But here I don't receive as part of TaskStatus
>>>> task labels and taskData, they are just empty.
>>>>
>>>
>>> Yea, this is expected behavior. Currently master *generates* new status
>>> updates (with only a few fields set) when it gets a reconciliation request.
>>>
>>
>>
>
Re: Get Task's labels on reconciliation
Posted by Benjamin Mahler <bm...@apache.org>.
I see that the following was filed:
https://issues.apache.org/jira/browse/MESOS-4477
But this sounds like a bug: if the master knows about the task during
reconciliation, labels should be sent. For example we had this same bug for
health information:
https://issues.apache.org/jira/browse/MESOS-2206
On Fri, Jan 22, 2016 at 2:10 PM, Andrii Biletskyi <
andrii.biletskyi@stealth.ly> wrote:
> Okay, thanks for your answer. I will rework my algorithm so it doesn't
> rely on task labels/data.
>
> Thanks,
> Andrii
>
> On Fri, Jan 22, 2016 at 11:49 PM, Vinod Kone <vi...@apache.org> wrote:
>
>>
>> On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi <
>> andrii.biletskyi@stealth.ly> wrote:
>>
>>> 8. Scheduler.statusUpdate callback is called for
>>> taskId=t1;state=Running. But here I don't receive as part of TaskStatus
>>> task labels and taskData, they are just empty.
>>>
>>
>> Yea, this is expected behavior. Currently master *generates* new status
>> updates (with only a few fields set) when it gets a reconciliation request.
>>
>
>
Re: Get Task's labels on reconciliation
Posted by Andrii Biletskyi <an...@stealth.ly>.
Okay, thanks for your answer. I will rework my algorithm so it doesn't rely
on task labels/data.
Thanks,
Andrii
On Fri, Jan 22, 2016 at 11:49 PM, Vinod Kone <vi...@apache.org> wrote:
>
> On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi <
> andrii.biletskyi@stealth.ly> wrote:
>
>> 8. Scheduler.statusUpdate callback is called for taskId=t1;state=Running.
>> But here I don't receive as part of TaskStatus task labels and taskData,
>> they are just empty.
>>
>
> Yea, this is expected behavior. Currently master *generates* new status
> updates (with only a few fields set) when it gets a reconciliation request.
>
Re: Get Task's labels on reconciliation
Posted by Vinod Kone <vi...@apache.org>.
On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi <
andrii.biletskyi@stealth.ly> wrote:
> 8. Scheduler.statusUpdate callback is called for taskId=t1;state=Running.
> But here I don't receive as part of TaskStatus task labels and taskData,
> they are just empty.
>
Yea, this is expected behavior. Currently master *generates* new status
updates (with only a few fields set) when it gets a reconciliation request.
Re: Get Task's labels on reconciliation
Posted by Andrii Biletskyi <an...@stealth.ly>.
Neil, thanks for the link!
Regarding task labels - maybe I'm doing something wrong because I can't see
task labels/
taskData received upon task reconciliation. Here's what I'm doing:
1. Start scheduler
2. Scheduler asks to launch some task (taskId=t1)
3. Executor successfully launches task, through mesosDriver returns state
Running and sets some labels, taskData
4. Scheduler receives status update, persists it somewhere. Important to
note that here I *do receive* as part of TaskStatus object valid task
labels and taskData
5. After some time I restart the scheduler
6. Scheduler is registered with defined frameworkId
7. Scheduler requests task reconciliation through
driver.reconcileTasks(Collections.empty)
8. Scheduler.statusUpdate callback is called for taskId=t1;state=Running.
But here I don't receive as part of TaskStatus task labels and taskData,
they are just empty.
So is it expected behavior?
Thanks,
Andrii
On Fri, Jan 22, 2016 at 9:02 PM, Neil Conway <ne...@gmail.com> wrote:
> Hi Andrii,
>
> TaskStatus includes the task's labels [1], so what you're trying to do
> should work.
>
> BTW, we recently wrote up some suggestions on how to write highly
> available frameworks. The docs will be on the website the next time it
> is refreshed; in the mean time, you can find them here:
>
> https://reviews.apache.org/r/41896/diff/8#1
>
> Neil
>
> [1]
> https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1162
>
> On Fri, Jan 22, 2016 at 6:49 AM, Andrii Biletskyi
> <an...@stealth.ly> wrote:
> > Hi,
> >
> > I'm using Mesos 0.25. Is there any way to get some more information about
> > the task (like label, taskData)
> > when requesting task reconciliation? Right now i see that driver returns
> > only taskId, slaveId, status, timestamp.
> >
> > Let me describe my use case so you have a better understanding.
> >
> > I'm trying to implement HA framework that will manage a cluster of
> database
> > nodes.
> > The scheduler will be run on marathon, and since I'm managing database
> nodes
> > I want to store mesos state
> > there to tolerate scheduler failures. So the workflow is the following:
> > User starts Scheduler on marathon providing some generated frameworkId.
> > Scheduler registers itself with the given
> > frameworkId and immediately starts reconciliation. If it's a fresh start
> > nothing has to be done - we are ready to accept
> > "add database node" commands. If it's a failover the idea was to inspect
> > each task state returned by the reconciliation
> > procedure, extract labels to distinguish whether it's a node, responsible
> > for storing mesos state, then extract task data
> > where we store node's host:port. With that you can initialize your
> storage
> > and may operate normally.
> >
> > Thanks in advance,
> > Andrii
> >
>
Re: Get Task's labels on reconciliation
Posted by Neil Conway <ne...@gmail.com>.
Hi Andrii,
TaskStatus includes the task's labels [1], so what you're trying to do
should work.
BTW, we recently wrote up some suggestions on how to write highly
available frameworks. The docs will be on the website the next time it
is refreshed; in the mean time, you can find them here:
https://reviews.apache.org/r/41896/diff/8#1
Neil
[1] https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1162
On Fri, Jan 22, 2016 at 6:49 AM, Andrii Biletskyi
<an...@stealth.ly> wrote:
> Hi,
>
> I'm using Mesos 0.25. Is there any way to get some more information about
> the task (like label, taskData)
> when requesting task reconciliation? Right now i see that driver returns
> only taskId, slaveId, status, timestamp.
>
> Let me describe my use case so you have a better understanding.
>
> I'm trying to implement HA framework that will manage a cluster of database
> nodes.
> The scheduler will be run on marathon, and since I'm managing database nodes
> I want to store mesos state
> there to tolerate scheduler failures. So the workflow is the following:
> User starts Scheduler on marathon providing some generated frameworkId.
> Scheduler registers itself with the given
> frameworkId and immediately starts reconciliation. If it's a fresh start
> nothing has to be done - we are ready to accept
> "add database node" commands. If it's a failover the idea was to inspect
> each task state returned by the reconciliation
> procedure, extract labels to distinguish whether it's a node, responsible
> for storing mesos state, then extract task data
> where we store node's host:port. With that you can initialize your storage
> and may operate normally.
>
> Thanks in advance,
> Andrii
>