Distinct docker jobs in .aurora

[Krish <kr...@gmail.com>]

Hi,
Is there a way to specify distinct IDs to various shards of jobs? Like
specifying {{mesos.instance}} in the Job configuration in the .aurora file?

I am trying to use a custom mountpoint for instances of the same job across
a cluster and could like to name the mountpoint distinctly, so that if the
job restarts on the same machine on the cluster, it can recover state from
the appropriate folder.

I have tried using  {{mesos.instance}} and {{thermos.task_id}} in the
.aurora file.
However, they only work in the Process() and not in Job().

Thanks.

Re: Distinct docker jobs in .aurora

[Krish <kr...@gmail.com>]

Hi Maxim,
Thanks for the heads up.
Is it possible to use {{mesos.instance}} in a Job config, say, as a Docker
parameter name=volume and value=<my job>{{mesos.instance}}?

I was thinking of using an EFS mapping at a later stage to map the
containers to different storage backends. A unique name for every process
that can be consistent across process restarts will be helpful.



--
κρισhναν

On Thu, Aug 11, 2016 at 10:44 AM, Maxim Khutornenko <ma...@apache.org>
wrote:

> No, this is not supported. Moreover, relying on local disk state is
> extremely unreliable and becomes impossible with the file system isolation
> rollout. If you need to persist some state your best bet at the moment is
> the external storage (HDFS, Blobstore, SQL and etc.
>
> That said, we have plans to *eventually* make your use case possible via
> Mesos persistent volumes. There is no ETA yet but there is a dynamic
> resource reservation design
> <https://docs.google.com/document/d/19gV8Po6DIHO14tOC7Qouk8RnboY8UCfRTninwn_5-7c>
> effort in progress that is the stepping stone towards that goal.
>
> On Wed, Aug 10, 2016 at 9:55 PM, Krish <kr...@gmail.com> wrote:
>
>> Hi,
>> Is there a way to specify distinct IDs to various shards of jobs? Like
>> specifying {{mesos.instance}} in the Job configuration in the .aurora file?
>>
>> I am trying to use a custom mountpoint for instances of the same job
>> across a cluster and could like to name the mountpoint distinctly, so that
>> if the job restarts on the same machine on the cluster, it can recover
>> state from the appropriate folder.
>>
>> I have tried using  {{mesos.instance}} and {{thermos.task_id}} in the
>> .aurora file.
>> However, they only work in the Process() and not in Job().
>>
>> Thanks.
>>
>
>

Re: Distinct docker jobs in .aurora

[Maxim Khutornenko <ma...@apache.org>]

No, this is not supported. Moreover, relying on local disk state is
extremely unreliable and becomes impossible with the file system isolation
rollout. If you need to persist some state your best bet at the moment is
the external storage (HDFS, Blobstore, SQL and etc.

That said, we have plans to *eventually* make your use case possible via
Mesos persistent volumes. There is no ETA yet but there is a dynamic
resource reservation design
<https://docs.google.com/document/d/19gV8Po6DIHO14tOC7Qouk8RnboY8UCfRTninwn_5-7c>
effort in progress that is the stepping stone towards that goal.

On Wed, Aug 10, 2016 at 9:55 PM, Krish <kr...@gmail.com> wrote:

> Hi,
> Is there a way to specify distinct IDs to various shards of jobs? Like
> specifying {{mesos.instance}} in the Job configuration in the .aurora file?
>
> I am trying to use a custom mountpoint for instances of the same job
> across a cluster and could like to name the mountpoint distinctly, so that
> if the job restarts on the same machine on the cluster, it can recover
> state from the appropriate folder.
>
> I have tried using  {{mesos.instance}} and {{thermos.task_id}} in the
> .aurora file.
> However, they only work in the Process() and not in Job().
>
> Thanks.
>