You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by 王国栋 <wa...@gmail.com> on 2013/04/09 11:43:18 UTC
long lived framework can not launch task
hi
I am trying to run long-lived-framework in the trunk.
But I find the following error after task 1 finished.
W0409 17:18:03.841472 15305 master.cpp:1566] Error validating task 1 : Task
has invalid ExecutorInfo (existing ExecutorInfo with same ExecutorID is not
compatible)
Then all the tasks will be lost.
The log of the framework is as follow:
Registered!
.Starting task 0 on guodong-Vostro-3400
Task 0 is in state 1
Task 0 is in state 2
.Starting task 1 on guodong-Vostro-3400
Task 1 is in state 5
.Starting task 2 on guodong-Vostro-3400
Task 2 is in state 5
.Starting task 3 on guodong-Vostro-3400
Task 3 is in state 5
I also go through the code of LongLivedFramework. And I can not understand
this error, since the ExecutorInfo is passed as the constructor arguments
of Scheduler.
Best regards.
Guodong
Re: long lived framework can not launch task
Posted by 王国栋 <wa...@gmail.com>.
Thanks for your detailed explanation. It is quite clear to me now.
Guodong
On Thu, Apr 11, 2013 at 12:58 AM, Benjamin Mahler <benjamin.mahler@gmail.com
> wrote:
> Yes, each executor ID can have only one 'source'. That is the only
> constraint, which means you can share the same 'source' across different
> executors. To see the benefit of the 'source' field, take this example
> below.
>
> Say we have a framework and we want to spin up three executors to run a
> job. We can have three executors:
> 1: ExecutorID: "e_1", source: "page_rank_job"
> 1: ExecutorID: "e_2", source: "page_rank_job"
> 1: ExecutorID: "e_3", source: "page_rank_job"
>
> This allows us to identify the source of the executor. This is also
> exported as part of the monitoring stats on each slave.
>
> $ curl <slave>:5051/monitor/usage.json
> [
> {"executor_id":"e1",
> "executor_name": "task_tracker",
> "source": "page_rank_job",
> "framework_id":"201301240028-2449582272-5050-61291-0000",
> "resource_usage":
> {"cpu_time":1.0, // CPU time (user + system) in seconds.
> "cpu_usage":0.5, // CPU usage [0.5 ~ 50% of 1 core].
> "memory_rss":1048576, // RSS in bytes.
> // ... Additional stats in the future.
> },
> },
> {"executor_id":"e2",
> "executor_name": "task_tracker",
> "source": "page_rank_job",
> "framework_id":"201301240028-2449582272-5050-61291-0000",
> "resource_usage":
> {"cpu_time":1.0, // CPU time (user + system) in seconds.
> "cpu_usage":16.0, // CPU usage [16.0 ~ 1600% usage ~ 100% of 16
> cores]
> "memory_rss":1048576, // RSS in bytes.
> // ... Additional stats in the future.
> },
> },
>
>
> Here we can aggregate all the resource usage for the page_rank_job by
> asking all of the slaves for the usage information. But if you're looking
> to write a framework, you can do without setting the source field for now,
> and set it later if the need arises.
>
> I'll eventually have some more documentation around this:
> https://issues.apache.org/jira/browse/MESOS-373
>
> On Wed, Apr 10, 2013 at 2:28 AM, 王国栋 <wa...@gmail.com> wrote:
>
> > Hi Ben,
> >
> > I find some comment in mesos.proto file,
> > // Source is an identifier style string used by frameworks to track
> > // the source of an executor.
> >
> > Does this mean each executorId can only have the identical source ?
> > In the old long-lived-framework, we are trying to assign different
> sources
> > to the same executor. So the error happens.
> >
> > Am I right?
> >
> > Thanks.
> >
> > Guodong
> >
> >
> > On Wed, Apr 10, 2013 at 3:02 PM, 王国栋 <wa...@gmail.com> wrote:
> >
> > > Hi Ben,
> > >
> > > It works now. Thank you for your reply.
> > >
> > > I am trying to learn to write a framework on mesos. But I can not find
> > the
> > > exact meaning of each param in the API. eg. I do not know the meaning
> of
> > > "source". Where can I find the docs about the programming guide?
> > >
> > > Thanks.
> > >
> > >
> > > Guodong
> > >
> > >
> > > On Wed, Apr 10, 2013 at 2:09 AM, Benjamin Mahler <
> > > benjamin.mahler@gmail.com> wrote:
> > >
> > >> Thanks for the report!
> > >>
> > >> The bug here is that we set the source for each task:
> > >>
> > >> TaskInfo task;
> > >> task.set_name("Task " + lexical_cast<string>(taskId));
> > >>
> task.mutable_task_id()->set_value(lexical_cast<string>(taskId));
> > >> task.mutable_slave_id()->MergeFrom(offer.slave_id());
> > >> task.mutable_executor()->MergeFrom(executor);
> > >> *task.mutable_executor()->set_source("task_" +
> > >> stringify(taskId));*
> > >>
> > >> I'll have a review out shortly to fix this.
> > >>
> > >>
> > >> On Tue, Apr 9, 2013 at 2:43 AM, 王国栋 <wa...@gmail.com> wrote:
> > >>
> > >> > hi
> > >> >
> > >> > I am trying to run long-lived-framework in the trunk.
> > >> > But I find the following error after task 1 finished.
> > >> >
> > >> > W0409 17:18:03.841472 15305 master.cpp:1566] Error validating task
> 1 :
> > >> Task
> > >> > has invalid ExecutorInfo (existing ExecutorInfo with same ExecutorID
> > is
> > >> not
> > >> > compatible)
> > >> >
> > >> > Then all the tasks will be lost.
> > >> >
> > >> > The log of the framework is as follow:
> > >> > Registered!
> > >> > .Starting task 0 on guodong-Vostro-3400
> > >> > Task 0 is in state 1
> > >> > Task 0 is in state 2
> > >> > .Starting task 1 on guodong-Vostro-3400
> > >> > Task 1 is in state 5
> > >> > .Starting task 2 on guodong-Vostro-3400
> > >> > Task 2 is in state 5
> > >> > .Starting task 3 on guodong-Vostro-3400
> > >> > Task 3 is in state 5
> > >> >
> > >> >
> > >> > I also go through the code of LongLivedFramework. And I can not
> > >> understand
> > >> > this error, since the ExecutorInfo is passed as the constructor
> > >> arguments
> > >> > of Scheduler.
> > >> >
> > >> > Best regards.
> > >> >
> > >> > Guodong
> > >> >
> > >>
> > >
> > >
> >
>
Re: long lived framework can not launch task
Posted by Benjamin Mahler <be...@gmail.com>.
Yes, each executor ID can have only one 'source'. That is the only
constraint, which means you can share the same 'source' across different
executors. To see the benefit of the 'source' field, take this example
below.
Say we have a framework and we want to spin up three executors to run a
job. We can have three executors:
1: ExecutorID: "e_1", source: "page_rank_job"
1: ExecutorID: "e_2", source: "page_rank_job"
1: ExecutorID: "e_3", source: "page_rank_job"
This allows us to identify the source of the executor. This is also
exported as part of the monitoring stats on each slave.
$ curl <slave>:5051/monitor/usage.json
[
{"executor_id":"e1",
"executor_name": "task_tracker",
"source": "page_rank_job",
"framework_id":"201301240028-2449582272-5050-61291-0000",
"resource_usage":
{"cpu_time":1.0, // CPU time (user + system) in seconds.
"cpu_usage":0.5, // CPU usage [0.5 ~ 50% of 1 core].
"memory_rss":1048576, // RSS in bytes.
// ... Additional stats in the future.
},
},
{"executor_id":"e2",
"executor_name": "task_tracker",
"source": "page_rank_job",
"framework_id":"201301240028-2449582272-5050-61291-0000",
"resource_usage":
{"cpu_time":1.0, // CPU time (user + system) in seconds.
"cpu_usage":16.0, // CPU usage [16.0 ~ 1600% usage ~ 100% of 16 cores]
"memory_rss":1048576, // RSS in bytes.
// ... Additional stats in the future.
},
},
Here we can aggregate all the resource usage for the page_rank_job by
asking all of the slaves for the usage information. But if you're looking
to write a framework, you can do without setting the source field for now,
and set it later if the need arises.
I'll eventually have some more documentation around this:
https://issues.apache.org/jira/browse/MESOS-373
On Wed, Apr 10, 2013 at 2:28 AM, 王国栋 <wa...@gmail.com> wrote:
> Hi Ben,
>
> I find some comment in mesos.proto file,
> // Source is an identifier style string used by frameworks to track
> // the source of an executor.
>
> Does this mean each executorId can only have the identical source ?
> In the old long-lived-framework, we are trying to assign different sources
> to the same executor. So the error happens.
>
> Am I right?
>
> Thanks.
>
> Guodong
>
>
> On Wed, Apr 10, 2013 at 3:02 PM, 王国栋 <wa...@gmail.com> wrote:
>
> > Hi Ben,
> >
> > It works now. Thank you for your reply.
> >
> > I am trying to learn to write a framework on mesos. But I can not find
> the
> > exact meaning of each param in the API. eg. I do not know the meaning of
> > "source". Where can I find the docs about the programming guide?
> >
> > Thanks.
> >
> >
> > Guodong
> >
> >
> > On Wed, Apr 10, 2013 at 2:09 AM, Benjamin Mahler <
> > benjamin.mahler@gmail.com> wrote:
> >
> >> Thanks for the report!
> >>
> >> The bug here is that we set the source for each task:
> >>
> >> TaskInfo task;
> >> task.set_name("Task " + lexical_cast<string>(taskId));
> >> task.mutable_task_id()->set_value(lexical_cast<string>(taskId));
> >> task.mutable_slave_id()->MergeFrom(offer.slave_id());
> >> task.mutable_executor()->MergeFrom(executor);
> >> *task.mutable_executor()->set_source("task_" +
> >> stringify(taskId));*
> >>
> >> I'll have a review out shortly to fix this.
> >>
> >>
> >> On Tue, Apr 9, 2013 at 2:43 AM, 王国栋 <wa...@gmail.com> wrote:
> >>
> >> > hi
> >> >
> >> > I am trying to run long-lived-framework in the trunk.
> >> > But I find the following error after task 1 finished.
> >> >
> >> > W0409 17:18:03.841472 15305 master.cpp:1566] Error validating task 1 :
> >> Task
> >> > has invalid ExecutorInfo (existing ExecutorInfo with same ExecutorID
> is
> >> not
> >> > compatible)
> >> >
> >> > Then all the tasks will be lost.
> >> >
> >> > The log of the framework is as follow:
> >> > Registered!
> >> > .Starting task 0 on guodong-Vostro-3400
> >> > Task 0 is in state 1
> >> > Task 0 is in state 2
> >> > .Starting task 1 on guodong-Vostro-3400
> >> > Task 1 is in state 5
> >> > .Starting task 2 on guodong-Vostro-3400
> >> > Task 2 is in state 5
> >> > .Starting task 3 on guodong-Vostro-3400
> >> > Task 3 is in state 5
> >> >
> >> >
> >> > I also go through the code of LongLivedFramework. And I can not
> >> understand
> >> > this error, since the ExecutorInfo is passed as the constructor
> >> arguments
> >> > of Scheduler.
> >> >
> >> > Best regards.
> >> >
> >> > Guodong
> >> >
> >>
> >
> >
>
Re: long lived framework can not launch task
Posted by 王国栋 <wa...@gmail.com>.
Hi Ben,
I find some comment in mesos.proto file,
// Source is an identifier style string used by frameworks to track
// the source of an executor.
Does this mean each executorId can only have the identical source ?
In the old long-lived-framework, we are trying to assign different sources
to the same executor. So the error happens.
Am I right?
Thanks.
Guodong
On Wed, Apr 10, 2013 at 3:02 PM, 王国栋 <wa...@gmail.com> wrote:
> Hi Ben,
>
> It works now. Thank you for your reply.
>
> I am trying to learn to write a framework on mesos. But I can not find the
> exact meaning of each param in the API. eg. I do not know the meaning of
> "source". Where can I find the docs about the programming guide?
>
> Thanks.
>
>
> Guodong
>
>
> On Wed, Apr 10, 2013 at 2:09 AM, Benjamin Mahler <
> benjamin.mahler@gmail.com> wrote:
>
>> Thanks for the report!
>>
>> The bug here is that we set the source for each task:
>>
>> TaskInfo task;
>> task.set_name("Task " + lexical_cast<string>(taskId));
>> task.mutable_task_id()->set_value(lexical_cast<string>(taskId));
>> task.mutable_slave_id()->MergeFrom(offer.slave_id());
>> task.mutable_executor()->MergeFrom(executor);
>> *task.mutable_executor()->set_source("task_" +
>> stringify(taskId));*
>>
>> I'll have a review out shortly to fix this.
>>
>>
>> On Tue, Apr 9, 2013 at 2:43 AM, 王国栋 <wa...@gmail.com> wrote:
>>
>> > hi
>> >
>> > I am trying to run long-lived-framework in the trunk.
>> > But I find the following error after task 1 finished.
>> >
>> > W0409 17:18:03.841472 15305 master.cpp:1566] Error validating task 1 :
>> Task
>> > has invalid ExecutorInfo (existing ExecutorInfo with same ExecutorID is
>> not
>> > compatible)
>> >
>> > Then all the tasks will be lost.
>> >
>> > The log of the framework is as follow:
>> > Registered!
>> > .Starting task 0 on guodong-Vostro-3400
>> > Task 0 is in state 1
>> > Task 0 is in state 2
>> > .Starting task 1 on guodong-Vostro-3400
>> > Task 1 is in state 5
>> > .Starting task 2 on guodong-Vostro-3400
>> > Task 2 is in state 5
>> > .Starting task 3 on guodong-Vostro-3400
>> > Task 3 is in state 5
>> >
>> >
>> > I also go through the code of LongLivedFramework. And I can not
>> understand
>> > this error, since the ExecutorInfo is passed as the constructor
>> arguments
>> > of Scheduler.
>> >
>> > Best regards.
>> >
>> > Guodong
>> >
>>
>
>
Re: long lived framework can not launch task
Posted by 王国栋 <wa...@gmail.com>.
Hi Ben,
It works now. Thank you for your reply.
I am trying to learn to write a framework on mesos. But I can not find the
exact meaning of each param in the API. eg. I do not know the meaning of
"source". Where can I find the docs about the programming guide?
Thanks.
Guodong
On Wed, Apr 10, 2013 at 2:09 AM, Benjamin Mahler
<be...@gmail.com>wrote:
> Thanks for the report!
>
> The bug here is that we set the source for each task:
>
> TaskInfo task;
> task.set_name("Task " + lexical_cast<string>(taskId));
> task.mutable_task_id()->set_value(lexical_cast<string>(taskId));
> task.mutable_slave_id()->MergeFrom(offer.slave_id());
> task.mutable_executor()->MergeFrom(executor);
> *task.mutable_executor()->set_source("task_" + stringify(taskId));*
>
> I'll have a review out shortly to fix this.
>
>
> On Tue, Apr 9, 2013 at 2:43 AM, 王国栋 <wa...@gmail.com> wrote:
>
> > hi
> >
> > I am trying to run long-lived-framework in the trunk.
> > But I find the following error after task 1 finished.
> >
> > W0409 17:18:03.841472 15305 master.cpp:1566] Error validating task 1 :
> Task
> > has invalid ExecutorInfo (existing ExecutorInfo with same ExecutorID is
> not
> > compatible)
> >
> > Then all the tasks will be lost.
> >
> > The log of the framework is as follow:
> > Registered!
> > .Starting task 0 on guodong-Vostro-3400
> > Task 0 is in state 1
> > Task 0 is in state 2
> > .Starting task 1 on guodong-Vostro-3400
> > Task 1 is in state 5
> > .Starting task 2 on guodong-Vostro-3400
> > Task 2 is in state 5
> > .Starting task 3 on guodong-Vostro-3400
> > Task 3 is in state 5
> >
> >
> > I also go through the code of LongLivedFramework. And I can not
> understand
> > this error, since the ExecutorInfo is passed as the constructor arguments
> > of Scheduler.
> >
> > Best regards.
> >
> > Guodong
> >
>
Re: long lived framework can not launch task
Posted by Benjamin Mahler <be...@gmail.com>.
I just committed the fix: https://reviews.apache.org/r/10381/
Let me know if you're still seeing issues.
On Tue, Apr 9, 2013 at 11:09 AM, Benjamin Mahler
<be...@gmail.com>wrote:
> Thanks for the report!
>
> The bug here is that we set the source for each task:
>
> TaskInfo task;
> task.set_name("Task " + lexical_cast<string>(taskId));
> task.mutable_task_id()->set_value(lexical_cast<string>(taskId));
> task.mutable_slave_id()->MergeFrom(offer.slave_id());
> task.mutable_executor()->MergeFrom(executor);
> *task.mutable_executor()->set_source("task_" + stringify(taskId));
> *
>
> I'll have a review out shortly to fix this.
>
>
> On Tue, Apr 9, 2013 at 2:43 AM, 王国栋 <wa...@gmail.com> wrote:
>
>> hi
>>
>> I am trying to run long-lived-framework in the trunk.
>> But I find the following error after task 1 finished.
>>
>> W0409 17:18:03.841472 15305 master.cpp:1566] Error validating task 1 :
>> Task
>> has invalid ExecutorInfo (existing ExecutorInfo with same ExecutorID is
>> not
>> compatible)
>>
>> Then all the tasks will be lost.
>>
>> The log of the framework is as follow:
>> Registered!
>> .Starting task 0 on guodong-Vostro-3400
>> Task 0 is in state 1
>> Task 0 is in state 2
>> .Starting task 1 on guodong-Vostro-3400
>> Task 1 is in state 5
>> .Starting task 2 on guodong-Vostro-3400
>> Task 2 is in state 5
>> .Starting task 3 on guodong-Vostro-3400
>> Task 3 is in state 5
>>
>>
>> I also go through the code of LongLivedFramework. And I can not understand
>> this error, since the ExecutorInfo is passed as the constructor arguments
>> of Scheduler.
>>
>> Best regards.
>>
>> Guodong
>>
>
>
Re: long lived framework can not launch task
Posted by Benjamin Mahler <be...@gmail.com>.
Thanks for the report!
The bug here is that we set the source for each task:
TaskInfo task;
task.set_name("Task " + lexical_cast<string>(taskId));
task.mutable_task_id()->set_value(lexical_cast<string>(taskId));
task.mutable_slave_id()->MergeFrom(offer.slave_id());
task.mutable_executor()->MergeFrom(executor);
*task.mutable_executor()->set_source("task_" + stringify(taskId));*
I'll have a review out shortly to fix this.
On Tue, Apr 9, 2013 at 2:43 AM, 王国栋 <wa...@gmail.com> wrote:
> hi
>
> I am trying to run long-lived-framework in the trunk.
> But I find the following error after task 1 finished.
>
> W0409 17:18:03.841472 15305 master.cpp:1566] Error validating task 1 : Task
> has invalid ExecutorInfo (existing ExecutorInfo with same ExecutorID is not
> compatible)
>
> Then all the tasks will be lost.
>
> The log of the framework is as follow:
> Registered!
> .Starting task 0 on guodong-Vostro-3400
> Task 0 is in state 1
> Task 0 is in state 2
> .Starting task 1 on guodong-Vostro-3400
> Task 1 is in state 5
> .Starting task 2 on guodong-Vostro-3400
> Task 2 is in state 5
> .Starting task 3 on guodong-Vostro-3400
> Task 3 is in state 5
>
>
> I also go through the code of LongLivedFramework. And I can not understand
> this error, since the ExecutorInfo is passed as the constructor arguments
> of Scheduler.
>
> Best regards.
>
> Guodong
>