hot swap of topology

[Aaron Zimmerman <az...@sproutsocial.com>]

Has anyone done any work with redeploying a topology with minimal downtime?
I'm imagining a new storm command, or maybe a new function of the
StormSubmitter class that:

uploads the new code to the cluster,
initializes bolts and spouts,
turns off the old spouts,
turns on the new spouts
waits for old cluster to finish anything in flight
kills the old topology.

I have a deploy process in place that does this manually but this prevents
the use of the Isolation Scheduler, since I have to rename the topology
something different to have multiple running at once.

I couldn't find a related JIRA,  but I seem to recall some discussion in
the past and I dont want to duplicate work.

Thanks,

Aaron Zimmerman

Re: hot swap of topology

[Michael Rose <mi...@fullcontact.com>]

Hi Aaron,

We do rolling deploys of our topologies by appending the build number to
each topology.

storm-topology-1 is active
Jenkins submits storm-topology-2
Allow storm-topology-2 to become active & check health (or else halt)
Deactivate storm-topology-1 & wait a few minutes (opportunity to halt and
reactivate)
Kill storm-topology-1

This works for us as we don't have any critical in-memory state that isn't
checkpointed to a persistent store, and the vast majority of our work can
be replayed safely.

Michael

Michael Rose (@Xorlev <https://twitter.com/xorlev>)
Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
michael@fullcontact.com


On Mon, Jun 16, 2014 at 7:32 AM, Aaron Zimmerman <
azimmerman@sproutsocial.com> wrote:

> Has anyone done any work with redeploying a topology with minimal
> downtime? I'm imagining a new storm command, or maybe a new function of the
> StormSubmitter class that:
>
> uploads the new code to the cluster,
> initializes bolts and spouts,
> turns off the old spouts,
> turns on the new spouts
> waits for old cluster to finish anything in flight
> kills the old topology.
>
> I have a deploy process in place that does this manually but this prevents
> the use of the Isolation Scheduler, since I have to rename the topology
> something different to have multiple running at once.
>
> I couldn't find a related JIRA,  but I seem to recall some discussion in
> the past and I dont want to duplicate work.
>
> Thanks,
>
> Aaron Zimmerman
>