You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Vladimir Vivien <vl...@gmail.com> on 2013/10/06 20:35:09 UTC
Ability to pass params to Task without TASK_LOST
All,
I ran into a situation where I need to pass params/values to tasks as they
are launched from the scheduler. The only mechanism that I see is
Command.Environment variables.
However, when I attempt to launch tasks with same environment variable
names, the master rejects subsequent tasks with the taskinfo as invalid due
task already exists (paraphrase) and the task is marked as TASK_LOST.
Is there another way to reliably pass params/values to a task? In my
situation, I want to launch a task (upon resource offer) and pass a new
value to the task upon each Executor.launch callback.
Thanks.
--
Vladimir Vivien
Re: Ability to pass params to Task without TASK_LOST
Posted by Vinod Kone <vi...@twitter.com>.
Cool. LGTM.
@vinodkone
Sent from my mobile
> On Oct 7, 2013, at 12:53 AM, Vladimir Vivien <vl...@gmail.com> wrote:
>
> @vinodkone
> I was getting task rejection for the reasons you mentioned i think because I was same ExecutorID chaning ExecutorInfo (Environment Vars were changing). I changed the code to the snippet below (hope formatting holds) . I am using setData() as you suggested. This works much better.
>
> Protos.TaskID taskId = Protos.TaskID.newBuilder().setValue("task-" + UUID.randomUUID().toString()).build();
> Protos.TaskInfo task = Protos.TaskInfo.newBuilder()
> .setName(taskId.getValue())
> .setTaskId(taskId)
> .setSlaveId(offer.getSlaveId())
> .addResources(
> Protos.Resource.newBuilder()
> .setName("cpus")
> .setType(Protos.Value.Type.SCALAR)
> .setScalar(Protos.Value.Scalar.newBuilder().setValue(1).build())
> .build()
> )
> .addResources(
> Protos.Resource.newBuilder()
> .setName("mem")
> .setType(Protos.Value.Type.SCALAR)
> .setScalar(Protos.Value.Scalar.newBuilder().setValue(256).build())
> .build()
> )
> .setExecutor(
> Protos.ExecutorInfo.newBuilder()
> .setExecutorId(Protos.ExecutorID.newBuilder().setValue("my-executor"))
> .setCommand(Protos.CommandInfo.newBuilder()
> .setValue(new File(TASK_CLI_COMMAND).getAbsolutePath())
> .build()
> )
> .setName("My Task Executor")
> .build() // ExecutorInfo
> )
> .setData(ByteString.copyFromUtf8(Integer.toString(taskSize)))
> .build();
>
>
>> On Sun, Oct 6, 2013 at 9:49 PM, Vinod Kone <vi...@twitter.com> wrote:
>> I see. So were you trying to set ExecutorInfo.Command() to pass environment variables to the tasks? If yes, I would not recommend it. For one, ExecutorInfo.Command() should be used for setting up properties specific to Executor but not tasks. This is especially true if the Executor can run multiple tasks. Second, Mesos would reject a task if it reuses Task.ExecutorInfo.ExecutorID but uses a different ExecutorInfo. In your case I imagine ExecutorInfo changed between tasks because ExecutorInfo.Command changed?
>>
>>
>>
>>
>> @vinodkone
>>
>>
>>> On Sun, Oct 6, 2013 at 1:45 PM, Vladimir Vivien <vl...@gmail.com> wrote:
>>> @Vinodkone
>>> Thanks for the reply. It clarifies things.
>>> To be clear I am using an ExecutorInfo.
>>> I am also generating new taskId for each launch request.
>>>
>>> But, I will look into TaskInfo.data to pass data into my task.
>>>
>>> Thanks
>>>
>>>
>>>> On Sun, Oct 6, 2013 at 2:55 PM, Vinod Kone <vi...@twitter.com> wrote:
>>>> Hi Vladimir,
>>>>
>>>> You cannot launch multiple tasks with same taskID for a given framework. TaskInfo.Command should only be used if you want to use Mesos's Command Executor. If you have your own executor, TaskInfo.ExecutorInfo should be set instead.
>>>>
>>>> Also, I would recommend using TaskInfo.data to send any data that your task might need.
>>>>
>>>>
>>>>
>>>> @vinodkone
>>>>
>>>>
>>>>> On Sun, Oct 6, 2013 at 11:35 AM, Vladimir Vivien <vl...@gmail.com> wrote:
>>>>> All,
>>>>> I ran into a situation where I need to pass params/values to tasks as they are launched from the scheduler. The only mechanism that I see is Command.Environment variables.
>>>>>
>>>>> However, when I attempt to launch tasks with same environment variable names, the master rejects subsequent tasks with the taskinfo as invalid due task already exists (paraphrase) and the task is marked as TASK_LOST.
>>>>>
>>>>> Is there another way to reliably pass params/values to a task? In my situation, I want to launch a task (upon resource offer) and pass a new value to the task upon each Executor.launch callback.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> --
>>>>> Vladimir Vivien
>>>
>>>
>>>
>>> --
>>> Vladimir Vivien
>
>
>
> --
> Vladimir Vivien
Re: Ability to pass params to Task without TASK_LOST
Posted by Vladimir Vivien <vl...@gmail.com>.
@vinodkone
I was getting task rejection for the reasons you mentioned i think because
I was same ExecutorID chaning ExecutorInfo (Environment Vars were
changing). I changed the code to the snippet below (hope formatting holds)
. I am using setData() as you suggested. This works much better.
Protos.TaskID taskId = Protos.TaskID.newBuilder().setValue("task-" +
UUID.randomUUID().toString()).build();
Protos.TaskInfo task = Protos.TaskInfo.newBuilder()
.setName(taskId.getValue())
.setTaskId(taskId)
.setSlaveId(offer.getSlaveId())
.addResources(
Protos.Resource.newBuilder()
.setName("cpus")
.setType(Protos.Value.Type.SCALAR)
.setScalar(Protos.Value.Scalar.newBuilder().setValue(1).build())
.build()
)
.addResources(
Protos.Resource.newBuilder()
.setName("mem")
.setType(Protos.Value.Type.SCALAR)
.setScalar(Protos.Value.Scalar.newBuilder().setValue(256).build())
.build()
)
.setExecutor(
Protos.ExecutorInfo.newBuilder()
.setExecutorId(Protos.ExecutorID.newBuilder().setValue("my-executor"))
.setCommand(Protos.CommandInfo.newBuilder()
.setValue(new File(TASK_CLI_COMMAND).getAbsolutePath())
.build()
)
.setName("My Task Executor")
.build() // ExecutorInfo
)
.setData(ByteString.copyFromUtf8(Integer.toString(taskSize)))
.build();
On Sun, Oct 6, 2013 at 9:49 PM, Vinod Kone <vi...@twitter.com> wrote:
> I see. So were you trying to set ExecutorInfo.Command() to pass
> environment variables to the tasks? If yes, I would not recommend it. For
> one, ExecutorInfo.Command() should be used for setting up properties
> specific to Executor but not tasks. This is especially true if the Executor
> can run multiple tasks. Second, Mesos would reject a task if it reuses
> Task.ExecutorInfo.ExecutorID but uses a different ExecutorInfo. In your
> case I imagine ExecutorInfo changed between tasks because
> ExecutorInfo.Command changed?
>
>
>
>
> @vinodkone
>
>
> On Sun, Oct 6, 2013 at 1:45 PM, Vladimir Vivien <vladimir.vivien@gmail.com
> > wrote:
>
>> @Vinodkone
>> Thanks for the reply. It clarifies things.
>> To be clear I am using an ExecutorInfo.
>> I am also generating new taskId for each launch request.
>>
>> But, I will look into TaskInfo.data to pass data into my task.
>>
>> Thanks
>>
>>
>> On Sun, Oct 6, 2013 at 2:55 PM, Vinod Kone <vi...@twitter.com> wrote:
>>
>>> Hi Vladimir,
>>>
>>> You cannot launch multiple tasks with same taskID for a given framework.
>>> TaskInfo.Command should only be used if you want to use Mesos's Command
>>> Executor. If you have your own executor, TaskInfo.ExecutorInfo should be
>>> set instead.
>>>
>>> Also, I would recommend using TaskInfo.data to send any data that your
>>> task might need.
>>>
>>>
>>>
>>> @vinodkone
>>>
>>>
>>> On Sun, Oct 6, 2013 at 11:35 AM, Vladimir Vivien <
>>> vladimir.vivien@gmail.com> wrote:
>>>
>>>> All,
>>>> I ran into a situation where I need to pass params/values to tasks as
>>>> they are launched from the scheduler. The only mechanism that I see is
>>>> Command.Environment variables.
>>>>
>>>> However, when I attempt to launch tasks with same environment variable
>>>> names, the master rejects subsequent tasks with the taskinfo as invalid due
>>>> task already exists (paraphrase) and the task is marked as TASK_LOST.
>>>>
>>>> Is there another way to reliably pass params/values to a task? In my
>>>> situation, I want to launch a task (upon resource offer) and pass a new
>>>> value to the task upon each Executor.launch callback.
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Vladimir Vivien
>>>>
>>>
>>>
>>
>>
>> --
>> Vladimir Vivien
>>
>
>
--
Vladimir Vivien
Re: Ability to pass params to Task without TASK_LOST
Posted by Vinod Kone <vi...@twitter.com>.
I see. So were you trying to set ExecutorInfo.Command() to pass environment
variables to the tasks? If yes, I would not recommend it. For one,
ExecutorInfo.Command() should be used for setting up properties specific to
Executor but not tasks. This is especially true if the Executor can run
multiple tasks. Second, Mesos would reject a task if it reuses
Task.ExecutorInfo.ExecutorID but uses a different ExecutorInfo. In your
case I imagine ExecutorInfo changed between tasks because
ExecutorInfo.Command changed?
@vinodkone
On Sun, Oct 6, 2013 at 1:45 PM, Vladimir Vivien
<vl...@gmail.com>wrote:
> @Vinodkone
> Thanks for the reply. It clarifies things.
> To be clear I am using an ExecutorInfo.
> I am also generating new taskId for each launch request.
>
> But, I will look into TaskInfo.data to pass data into my task.
>
> Thanks
>
>
> On Sun, Oct 6, 2013 at 2:55 PM, Vinod Kone <vi...@twitter.com> wrote:
>
>> Hi Vladimir,
>>
>> You cannot launch multiple tasks with same taskID for a given framework.
>> TaskInfo.Command should only be used if you want to use Mesos's Command
>> Executor. If you have your own executor, TaskInfo.ExecutorInfo should be
>> set instead.
>>
>> Also, I would recommend using TaskInfo.data to send any data that your
>> task might need.
>>
>>
>>
>> @vinodkone
>>
>>
>> On Sun, Oct 6, 2013 at 11:35 AM, Vladimir Vivien <
>> vladimir.vivien@gmail.com> wrote:
>>
>>> All,
>>> I ran into a situation where I need to pass params/values to tasks as
>>> they are launched from the scheduler. The only mechanism that I see is
>>> Command.Environment variables.
>>>
>>> However, when I attempt to launch tasks with same environment variable
>>> names, the master rejects subsequent tasks with the taskinfo as invalid due
>>> task already exists (paraphrase) and the task is marked as TASK_LOST.
>>>
>>> Is there another way to reliably pass params/values to a task? In my
>>> situation, I want to launch a task (upon resource offer) and pass a new
>>> value to the task upon each Executor.launch callback.
>>>
>>> Thanks.
>>>
>>> --
>>> Vladimir Vivien
>>>
>>
>>
>
>
> --
> Vladimir Vivien
>
Re: Ability to pass params to Task without TASK_LOST
Posted by Vladimir Vivien <vl...@gmail.com>.
@Vinodkone
Thanks for the reply. It clarifies things.
To be clear I am using an ExecutorInfo.
I am also generating new taskId for each launch request.
But, I will look into TaskInfo.data to pass data into my task.
Thanks
On Sun, Oct 6, 2013 at 2:55 PM, Vinod Kone <vi...@twitter.com> wrote:
> Hi Vladimir,
>
> You cannot launch multiple tasks with same taskID for a given framework.
> TaskInfo.Command should only be used if you want to use Mesos's Command
> Executor. If you have your own executor, TaskInfo.ExecutorInfo should be
> set instead.
>
> Also, I would recommend using TaskInfo.data to send any data that your
> task might need.
>
>
>
> @vinodkone
>
>
> On Sun, Oct 6, 2013 at 11:35 AM, Vladimir Vivien <
> vladimir.vivien@gmail.com> wrote:
>
>> All,
>> I ran into a situation where I need to pass params/values to tasks as
>> they are launched from the scheduler. The only mechanism that I see is
>> Command.Environment variables.
>>
>> However, when I attempt to launch tasks with same environment variable
>> names, the master rejects subsequent tasks with the taskinfo as invalid due
>> task already exists (paraphrase) and the task is marked as TASK_LOST.
>>
>> Is there another way to reliably pass params/values to a task? In my
>> situation, I want to launch a task (upon resource offer) and pass a new
>> value to the task upon each Executor.launch callback.
>>
>> Thanks.
>>
>> --
>> Vladimir Vivien
>>
>
>
--
Vladimir Vivien
Re: Ability to pass params to Task without TASK_LOST
Posted by Vinod Kone <vi...@gmail.com>.
>
> Hi Vladimir,
>
> You cannot launch multiple tasks with same taskID for a given framework.
> TaskInfo.Command should only be used if you want to use Mesos's Command
> Executor. If you have your own executor, TaskInfo.ExecutorInfo should be
> set instead.
>
> Also, I would recommend using TaskInfo.data to send any data that your
> task might need.
>
>
>
> @vinodkone
>
>
> On Sun, Oct 6, 2013 at 11:35 AM, Vladimir Vivien <
> vladimir.vivien@gmail.com> wrote:
>
>> All,
>> I ran into a situation where I need to pass params/values to tasks as
>> they are launched from the scheduler. The only mechanism that I see is
>> Command.Environment variables.
>>
>> However, when I attempt to launch tasks with same environment variable
>> names, the master rejects subsequent tasks with the taskinfo as invalid due
>> task already exists (paraphrase) and the task is marked as TASK_LOST.
>>
>> Is there another way to reliably pass params/values to a task? In my
>> situation, I want to launch a task (upon resource offer) and pass a new
>> value to the task upon each Executor.launch callback.
>>
>> Thanks.
>>
>> --
>> Vladimir Vivien
>>
>
>
Re: Ability to pass params to Task without TASK_LOST
Posted by Vinod Kone <vi...@twitter.com>.
Hi Vladimir,
You cannot launch multiple tasks with same taskID for a given framework.
TaskInfo.Command should only be used if you want to use Mesos's Command
Executor. If you have your own executor, TaskInfo.ExecutorInfo should be
set instead.
Also, I would recommend using TaskInfo.data to send any data that your task
might need.
@vinodkone
On Sun, Oct 6, 2013 at 11:35 AM, Vladimir Vivien
<vl...@gmail.com>wrote:
> All,
> I ran into a situation where I need to pass params/values to tasks as they
> are launched from the scheduler. The only mechanism that I see is
> Command.Environment variables.
>
> However, when I attempt to launch tasks with same environment variable
> names, the master rejects subsequent tasks with the taskinfo as invalid due
> task already exists (paraphrase) and the task is marked as TASK_LOST.
>
> Is there another way to reliably pass params/values to a task? In my
> situation, I want to launch a task (upon resource offer) and pass a new
> value to the task upon each Executor.launch callback.
>
> Thanks.
>
> --
> Vladimir Vivien
>