Can Marathon ensure single instance of a service at any give time?

[Petr Novak <os...@gmail.com>]

Hello,
if I need to run single stateless instance or only a single leader doing a
work at any given time. Something I would typically implement using Zoo
Curator LeaderSelector. Can I use Marathon to ensure this without having to
implement mutual exclusion myself? Let's assume that other parts of the
architecture aren't designed well to support more running workers at a time.

Currently we have a service which updates cache, it runs on one node and
when it fails it is restarted and PID file is used to ensure single
instance. Pretty naive implementation, possibly doesn;t work in all edge
cases.

If I would like to allow it to restart on any node in a cluster can I use
Marathon to simplify the implementation or it warrants more involved
implementation using Zoo. Does Mesos provide any other helpers to simplify
this use case?

Or is Marathon designed only to run stateless services which can possibly
run in multiple instances?

Many thanks,
Petr

Re: Can Marathon ensure single instance of a service at any give time?

[Guangya Liu <gy...@gmail.com>]

Marathon is designed for long running services no matter the service is
stateless or stateful with HA by default.

The Marathon will help restart your services if it is down.

Thanks,

Guangya

On Tue, Feb 23, 2016 at 6:06 PM, Petr Novak <os...@gmail.com> wrote:

> Hello,
> if I need to run single stateless instance or only a single leader doing a
> work at any given time. Something I would typically implement using Zoo
> Curator LeaderSelector. Can I use Marathon to ensure this without having to
> implement mutual exclusion myself? Let's assume that other parts of the
> architecture aren't designed well to support more running workers at a time.
>
> Currently we have a service which updates cache, it runs on one node and
> when it fails it is restarted and PID file is used to ensure single
> instance. Pretty naive implementation, possibly doesn;t work in all edge
> cases.
>
> If I would like to allow it to restart on any node in a cluster can I use
> Marathon to simplify the implementation or it warrants more involved
> implementation using Zoo. Does Mesos provide any other helpers to simplify
> this use case?
>
> Or is Marathon designed only to run stateless services which can possibly
> run in multiple instances?
>
> Many thanks,
> Petr
>
>
>

Re: Can Marathon ensure single instance of a service at any give time?

[Klaus Ma <kl...@gmail.com>]

Marathon can meet your requirement by set 1 instance of that service; if
you want to start it on special host, constraint will help. Refer to
Marathon's doc for the detail (https://mesosphere.github.io/marathon/docs/).



----
Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me

On Tue, Feb 23, 2016 at 6:06 PM, Petr Novak <os...@gmail.com> wrote:

> Hello,
> if I need to run single stateless instance or only a single leader doing a
> work at any given time. Something I would typically implement using Zoo
> Curator LeaderSelector. Can I use Marathon to ensure this without having to
> implement mutual exclusion myself? Let's assume that other parts of the
> architecture aren't designed well to support more running workers at a time.
>
> Currently we have a service which updates cache, it runs on one node and
> when it fails it is restarted and PID file is used to ensure single
> instance. Pretty naive implementation, possibly doesn;t work in all edge
> cases.
>
> If I would like to allow it to restart on any node in a cluster can I use
> Marathon to simplify the implementation or it warrants more involved
> implementation using Zoo. Does Mesos provide any other helpers to simplify
> this use case?
>
> Or is Marathon designed only to run stateless services which can possibly
> run in multiple instances?
>
> Many thanks,
> Petr
>
>
>

Re: Can Marathon ensure single instance of a service at any give time?

[Mauricio Garavaglia <ma...@medallia.com>]

Right, Marathon can't provide uniqueness guarantees. As you said, network
partitions are really common in distributed systems and shouldn't be
considered edge cases.

On Wed, Feb 24, 2016 at 8:49 AM, Petr Novak <os...@gmail.com> wrote:

> Thanks everybody for the great input. If I understand it correctly it
> doesn't help in this case, it just blindly restart service somewhere else
> once it looses heartbeat. Partition doesn't happen only because network
> failure it can be as simple as JVM "stop the world" with large heap or
> pretty much whatever. In cases when 2 and more potentially running services
> can throw havoc on my cluster I have to implement advanced coordination
> myself. It probably make sense because production level implementation for
> Zoo has to use LeaderSelector and equivalent and the actual logic is
> probably quite connected to business logic in service, e.g. when to ensure
> that service is still leader at this exact moment right before performing
> an action. Unsure if there is enough general use cases for a simple "racing
> for a lock on service startup" generic implementation.
>
> Petr
>
>

Re: Can Marathon ensure single instance of a service at any give time?

[Petr Novak <os...@gmail.com>]

Thanks everybody for the great input. If I understand it correctly it
doesn't help in this case, it just blindly restart service somewhere else
once it looses heartbeat. Partition doesn't happen only because network
failure it can be as simple as JVM "stop the world" with large heap or
pretty much whatever. In cases when 2 and more potentially running services
can throw havoc on my cluster I have to implement advanced coordination
myself. It probably make sense because production level implementation for
Zoo has to use LeaderSelector and equivalent and the actual logic is
probably quite connected to business logic in service, e.g. when to ensure
that service is still leader at this exact moment right before performing
an action. Unsure if there is enough general use cases for a simple "racing
for a lock on service startup" generic implementation.

Petr

Re: Can Marathon ensure single instance of a service at any give time?

[Shuai Lin <li...@gmail.com>]

>
> If I would like to allow it to restart on any node in a cluster can I use
> Marathon to simplify the implementation or it warrants more involved
> implementation using Zoo. Does Mesos provide any other helpers to simplify
> this use case?


Marathon can do that, but be aware that there is possibility that in some
edge cases like network partitions, there could be two instances running at
the same time.

For example, if the mesos slave that runs your cache-updating service can't
connect with mesos master due to network connection problems, after a while
mesos master would think the slave is dead, and tells marathon that the
task is lost. In this case marathon would launch another instance of your
cache-updating service, the result - two instances are running at the same
time.

To avoid this, you can pin the service to a specific slave by using the
constraints provided by marathon, as @klaus suggested above, but this would
lost the flexibility of running it inside a mesos cluster. Otherwise you
have to use a distributed consensus solution like zookeeper.

On Tue, Feb 23, 2016 at 6:06 PM, Petr Novak <os...@gmail.com> wrote:

> Hello,
> if I need to run single stateless instance or only a single leader doing a
> work at any given time. Something I would typically implement using Zoo
> Curator LeaderSelector. Can I use Marathon to ensure this without having to
> implement mutual exclusion myself? Let's assume that other parts of the
> architecture aren't designed well to support more running workers at a time.
>
> Currently we have a service which updates cache, it runs on one node and
> when it fails it is restarted and PID file is used to ensure single
> instance. Pretty naive implementation, possibly doesn;t work in all edge
> cases.
>
> If I would like to allow it to restart on any node in a cluster can I use
> Marathon to simplify the implementation or it warrants more involved
> implementation using Zoo. Does Mesos provide any other helpers to simplify
> this use case?
>
> Or is Marathon designed only to run stateless services which can possibly
> run in multiple instances?
>
> Many thanks,
> Petr
>
>
>