Strategies for continuous deployment of services to avoid interruption?

[Phillip B Oldham <ph...@gmail.com>]

We have a number of Python & Java thrift services which we are
manually deploying on a regular basis; usually early in the AM while
it's "quiet" since deployment causes service interruption.

We'd like to move to continuous deployment, so that when our commits
successfully pass all the tests on our Hudson/Jenkins CI server
something (Hudson/Jenkins, Puppet, custom scripts) will deploy the
services without human intervention. The problem is that, in this
scenario, the services may be deployed multiple times a day. Since
each deployment causes service interruption we've held back.

So, my question is: how would one avoid service interruption during
deployment? Is there a common tool/strategy for such tasks?

-- 
Phillip B Oldham
phillip.oldham@gmail.com
+44 (0) 7525 01 09 01

Re: Strategies for continuous deployment of services to avoid interruption?

[Ted Dunning <te...@gmail.com>]

Yes.  But not like Erlang does.

Seriously, the problem is state, not code.  Typical server apps have a lot
of stuff cached which makes them run faster when warmed up.  A hot swap
doesn't provide that, really, even in Erlang.  Thus you need some pretty
fancy mechanisms including state replication and soft load reinstatement to
really handle hot swapping.

Take a look at Zookeeper for a system that manages to deal with hot swaps
and state.

On Tue, Feb 1, 2011 at 12:05 AM, Phillip B Oldham
<ph...@gmail.com>wrote:

> Would there be any way to do hot-code reloading, a bit like what Erlang
> does?
>
> On 25 January 2011 20:30, Ilya Maykov <iv...@gmail.com> wrote:
> > It can be very tricky to detect failures in a distributed system. In
> fact, it's not always possible. Suppose your thrift RPC is just taking a
> really long time - at some point it will time out (based on some config
> parameter that you set). However the timeout is client-side and the server
> may have completed the request and was just about to respond when the client
> gave up. Or, maybe the request failed. Or maybe the server is still working
> on it. But, that discussion goes well beyond the scope of the original
> question in this thread :)
>

Re: Strategies for continuous deployment of services to avoid interruption?

[Phillip B Oldham <ph...@gmail.com>]

Would there be any way to do hot-code reloading, a bit like what Erlang does?

On 25 January 2011 20:30, Ilya Maykov <iv...@gmail.com> wrote:
> It can be very tricky to detect failures in a distributed system. In fact, it's not always possible. Suppose your thrift RPC is just taking a really long time - at some point it will time out (based on some config parameter that you set). However the timeout is client-side and the server may have completed the request and was just about to respond when the client gave up. Or, maybe the request failed. Or maybe the server is still working on it. But, that discussion goes well beyond the scope of the original question in this thread :)
>
> -- Ilya
>
> Sent from my iPhone
>
> On Jan 25, 2011, at 7:09, Ted Dunning <te...@gmail.com> wrote:
>
>> On Tue, Jan 25, 2011 at 12:01 AM, Phillip B Oldham <phillip.oldham@gmail.com
>>> wrote:
>>
>>> I suppose it would be left up to the client then to test whether a
>>> failed response actually completed... adding a fair amount of work to
>>> the client.
>>>
>>
>> Yes. It does.  And if you don't design the system well, then you may not
>> even be able to tell if it has completed.
>>
>>>
>>> Would zookeeper be able to "buffer" requests? For instance, if there
>>> were two nodes behind it and they were both momentarily unresponsive,
>>> could zookeeper (& the client) keep the connection active and wait for
>>> a node to signal itself available and complete the request?
>>>
>>
>> ZK isn't really involved in your request.  It is merely helping you
>> coordinate things.  It is completely reasonable to keep "last completed
>> transaction id" in ZK, but that doesn't really solve the problem.  You don't
>> want to wait forever, after all.
>



-- 
Phillip B Oldham
phillip.oldham@gmail.com
+44 (0) 7525 01 09 01

Re: Strategies for continuous deployment of services to avoid interruption?

[Ilya Maykov <iv...@gmail.com>]

It can be very tricky to detect failures in a distributed system. In fact, it's not always possible. Suppose your thrift RPC is just taking a really long time - at some point it will time out (based on some config parameter that you set). However the timeout is client-side and the server may have completed the request and was just about to respond when the client gave up. Or, maybe the request failed. Or maybe the server is still working on it. But, that discussion goes well beyond the scope of the original question in this thread :)

-- Ilya

Sent from my iPhone

On Jan 25, 2011, at 7:09, Ted Dunning <te...@gmail.com> wrote:

> On Tue, Jan 25, 2011 at 12:01 AM, Phillip B Oldham <phillip.oldham@gmail.com
>> wrote:
> 
>> I suppose it would be left up to the client then to test whether a
>> failed response actually completed... adding a fair amount of work to
>> the client.
>> 
> 
> Yes. It does.  And if you don't design the system well, then you may not
> even be able to tell if it has completed.
> 
>> 
>> Would zookeeper be able to "buffer" requests? For instance, if there
>> were two nodes behind it and they were both momentarily unresponsive,
>> could zookeeper (& the client) keep the connection active and wait for
>> a node to signal itself available and complete the request?
>> 
> 
> ZK isn't really involved in your request.  It is merely helping you
> coordinate things.  It is completely reasonable to keep "last completed
> transaction id" in ZK, but that doesn't really solve the problem.  You don't
> want to wait forever, after all.

Re: Strategies for continuous deployment of services to avoid interruption?

[Ted Dunning <te...@gmail.com>]

On Tue, Jan 25, 2011 at 12:01 AM, Phillip B Oldham <phillip.oldham@gmail.com
> wrote:

> I suppose it would be left up to the client then to test whether a
> failed response actually completed... adding a fair amount of work to
> the client.
>

Yes. It does.  And if you don't design the system well, then you may not
even be able to tell if it has completed.

>
> Would zookeeper be able to "buffer" requests? For instance, if there
> were two nodes behind it and they were both momentarily unresponsive,
> could zookeeper (& the client) keep the connection active and wait for
> a node to signal itself available and complete the request?
>

ZK isn't really involved in your request.  It is merely helping you
coordinate things.  It is completely reasonable to keep "last completed
transaction id" in ZK, but that doesn't really solve the problem.  You don't
want to wait forever, after all.

Re: Strategies for continuous deployment of services to avoid interruption?

[Phillip B Oldham <ph...@gmail.com>]

On Tue, Jan 25, 2011 at 12:46 AM, Ted Dunning <te...@gmail.com> wrote:
> On Mon, Jan 24, 2011 at 4:43 PM, Ilya Maykov <iv...@gmail.com> wrote:
>
>> Right, but I assume you could still see a race condition where the server
>> just failed and zookeeper has not detected the failure yet. So zookeeper
>> still thinks it's up, thus the client still thinks it's up, thus the client
>> makes an RPC request that is doomed to fail :)
>>
>
> Yes.  That exists.
>
> The case where you select a server and it doesn't connect is a safe special
> case.  The client can just try another.
>
> The nastier issue, as you point out, is when the transaction starts but
> doesn't complete.
>

I suppose it would be left up to the client then to test whether a
failed response actually completed... adding a fair amount of work to
the client.

Would zookeeper be able to "buffer" requests? For instance, if there
were two nodes behind it and they were both momentarily unresponsive,
could zookeeper (& the client) keep the connection active and wait for
a node to signal itself available and complete the request?

-- 
Phillip B Oldham
phillip.oldham@gmail.com
+44 (0) 7525 01 09 01

Re: Strategies for continuous deployment of services to avoid interruption?

[Ted Dunning <te...@gmail.com>]

On Mon, Jan 24, 2011 at 4:43 PM, Ilya Maykov <iv...@gmail.com> wrote:

> Right, but I assume you could still see a race condition where the server
> just failed and zookeeper has not detected the failure yet. So zookeeper
> still thinks it's up, thus the client still thinks it's up, thus the client
> makes an RPC request that is doomed to fail :)
>

Yes.  That exists.

The case where you select a server and it doesn't connect is a safe special
case.  The client can just try another.

The nastier issue, as you point out, is when the transaction starts but
doesn't complete.

Re: Strategies for continuous deployment of services to avoid interruption?

[Ilya Maykov <iv...@gmail.com>]

Right, but I assume you could still see a race condition where the server just failed and zookeeper has not detected the failure yet. So zookeeper still thinks it's up, thus the client still thinks it's up, thus the client makes an RPC request that is doomed to fail :)

Using zookeeper is probably still a good idea, as it gives clients an almost-always-correct view of which servers are up and reduces the need for RPC retries. But the need is still there - taking the argument to its logical conclusion, a server can always crash in the middle of processing your (retry-able) request.

Of course if your requests have side effects and are not idempotent then you face the fun task of determining if the side effects happened or not before the server died, but that's a whole other set of problems :)

-- Ilya

Sent from my iPhone

On Jan 24, 2011, at 13:56, Bryan Duxbury <br...@rapleaf.com> wrote:

> If you use zookeeper, you can have the servers keep ephemeral nodes and have
> clients watch those nodes. Then you can detect server failure.
> 
> On Mon, Jan 24, 2011 at 1:38 PM, Ilya Maykov <iv...@gmail.com> wrote:
> 
>> As others have mentioned, multiple servers + rolling upgrade + clients that
>> retry to a different server when an RPC call fails. You could use zookeeper
>> to keep clients informed about the set of servers that are *supposed* to be
>> up so you don't try connecting to one that's in the process of upgrading,
>> but you still have to handle the randomly-crashed-server case so zookeeper
>> alone is likely not sufficient.
>> 
>> -- Ilya
>> 
>> Sent from my iPhone
>> 
>> On Jan 24, 2011, at 8:08, Ted Dunning <te...@gmail.com> wrote:
>> 
>>> Zookeeper uses a similar strategy but allows for more forceful movement
>> of
>>> connections.
>>> 
>>> I have used a similar strategy with other services with good results.
>>> 
>>> On Mon, Jan 24, 2011 at 7:36 AM, Bryan Duxbury <br...@rapleaf.com>
>> wrote:
>>> 
>>>> The strategy Rapleaf uses for purposes like this is to run multiple
>>>> servers.
>>>> The client is aware of all the possible servers, but usually only
>> connects
>>>> to one. When a connection becomes stale, you reconnect to another
>> server.
>>>> Then, to make your deploys less painful, you just deploy one server at a
>>>> time.
>>>> 
>>>> On Mon, Jan 24, 2011 at 1:33 AM, Phillip B Oldham
>>>> <ph...@gmail.com>wrote:
>>>> 
>>>>> We have a number of Python & Java thrift services which we are
>>>>> manually deploying on a regular basis; usually early in the AM while
>>>>> it's "quiet" since deployment causes service interruption.
>>>>> 
>>>>> We'd like to move to continuous deployment, so that when our commits
>>>>> successfully pass all the tests on our Hudson/Jenkins CI server
>>>>> something (Hudson/Jenkins, Puppet, custom scripts) will deploy the
>>>>> services without human intervention. The problem is that, in this
>>>>> scenario, the services may be deployed multiple times a day. Since
>>>>> each deployment causes service interruption we've held back.
>>>>> 
>>>>> So, my question is: how would one avoid service interruption during
>>>>> deployment? Is there a common tool/strategy for such tasks?
>>>>> 
>>>>> --
>>>>> Phillip B Oldham
>>>>> phillip.oldham@gmail.com
>>>>> +44 (0) 7525 01 09 01
>>>>> 
>>>> 
>> 

Re: Strategies for continuous deployment of services to avoid interruption?

[Bryan Duxbury <br...@rapleaf.com>]

If you use zookeeper, you can have the servers keep ephemeral nodes and have
clients watch those nodes. Then you can detect server failure.

On Mon, Jan 24, 2011 at 1:38 PM, Ilya Maykov <iv...@gmail.com> wrote:

> As others have mentioned, multiple servers + rolling upgrade + clients that
> retry to a different server when an RPC call fails. You could use zookeeper
> to keep clients informed about the set of servers that are *supposed* to be
> up so you don't try connecting to one that's in the process of upgrading,
> but you still have to handle the randomly-crashed-server case so zookeeper
> alone is likely not sufficient.
>
> -- Ilya
>
> Sent from my iPhone
>
> On Jan 24, 2011, at 8:08, Ted Dunning <te...@gmail.com> wrote:
>
> > Zookeeper uses a similar strategy but allows for more forceful movement
> of
> > connections.
> >
> > I have used a similar strategy with other services with good results.
> >
> > On Mon, Jan 24, 2011 at 7:36 AM, Bryan Duxbury <br...@rapleaf.com>
> wrote:
> >
> >> The strategy Rapleaf uses for purposes like this is to run multiple
> >> servers.
> >> The client is aware of all the possible servers, but usually only
> connects
> >> to one. When a connection becomes stale, you reconnect to another
> server.
> >> Then, to make your deploys less painful, you just deploy one server at a
> >> time.
> >>
> >> On Mon, Jan 24, 2011 at 1:33 AM, Phillip B Oldham
> >> <ph...@gmail.com>wrote:
> >>
> >>> We have a number of Python & Java thrift services which we are
> >>> manually deploying on a regular basis; usually early in the AM while
> >>> it's "quiet" since deployment causes service interruption.
> >>>
> >>> We'd like to move to continuous deployment, so that when our commits
> >>> successfully pass all the tests on our Hudson/Jenkins CI server
> >>> something (Hudson/Jenkins, Puppet, custom scripts) will deploy the
> >>> services without human intervention. The problem is that, in this
> >>> scenario, the services may be deployed multiple times a day. Since
> >>> each deployment causes service interruption we've held back.
> >>>
> >>> So, my question is: how would one avoid service interruption during
> >>> deployment? Is there a common tool/strategy for such tasks?
> >>>
> >>> --
> >>> Phillip B Oldham
> >>> phillip.oldham@gmail.com
> >>> +44 (0) 7525 01 09 01
> >>>
> >>
>

Re: Strategies for continuous deployment of services to avoid interruption?

[Ilya Maykov <iv...@gmail.com>]

As others have mentioned, multiple servers + rolling upgrade + clients that retry to a different server when an RPC call fails. You could use zookeeper to keep clients informed about the set of servers that are *supposed* to be up so you don't try connecting to one that's in the process of upgrading, but you still have to handle the randomly-crashed-server case so zookeeper alone is likely not sufficient.

-- Ilya

Sent from my iPhone

On Jan 24, 2011, at 8:08, Ted Dunning <te...@gmail.com> wrote:

> Zookeeper uses a similar strategy but allows for more forceful movement of
> connections.
> 
> I have used a similar strategy with other services with good results.
> 
> On Mon, Jan 24, 2011 at 7:36 AM, Bryan Duxbury <br...@rapleaf.com> wrote:
> 
>> The strategy Rapleaf uses for purposes like this is to run multiple
>> servers.
>> The client is aware of all the possible servers, but usually only connects
>> to one. When a connection becomes stale, you reconnect to another server.
>> Then, to make your deploys less painful, you just deploy one server at a
>> time.
>> 
>> On Mon, Jan 24, 2011 at 1:33 AM, Phillip B Oldham
>> <ph...@gmail.com>wrote:
>> 
>>> We have a number of Python & Java thrift services which we are
>>> manually deploying on a regular basis; usually early in the AM while
>>> it's "quiet" since deployment causes service interruption.
>>> 
>>> We'd like to move to continuous deployment, so that when our commits
>>> successfully pass all the tests on our Hudson/Jenkins CI server
>>> something (Hudson/Jenkins, Puppet, custom scripts) will deploy the
>>> services without human intervention. The problem is that, in this
>>> scenario, the services may be deployed multiple times a day. Since
>>> each deployment causes service interruption we've held back.
>>> 
>>> So, my question is: how would one avoid service interruption during
>>> deployment? Is there a common tool/strategy for such tasks?
>>> 
>>> --
>>> Phillip B Oldham
>>> phillip.oldham@gmail.com
>>> +44 (0) 7525 01 09 01
>>> 
>> 

Re: Strategies for continuous deployment of services to avoid interruption?

[Ted Dunning <te...@gmail.com>]

Zookeeper uses a similar strategy but allows for more forceful movement of
connections.

I have used a similar strategy with other services with good results.

On Mon, Jan 24, 2011 at 7:36 AM, Bryan Duxbury <br...@rapleaf.com> wrote:

> The strategy Rapleaf uses for purposes like this is to run multiple
> servers.
> The client is aware of all the possible servers, but usually only connects
> to one. When a connection becomes stale, you reconnect to another server.
> Then, to make your deploys less painful, you just deploy one server at a
> time.
>
> On Mon, Jan 24, 2011 at 1:33 AM, Phillip B Oldham
> <ph...@gmail.com>wrote:
>
> > We have a number of Python & Java thrift services which we are
> > manually deploying on a regular basis; usually early in the AM while
> > it's "quiet" since deployment causes service interruption.
> >
> > We'd like to move to continuous deployment, so that when our commits
> > successfully pass all the tests on our Hudson/Jenkins CI server
> > something (Hudson/Jenkins, Puppet, custom scripts) will deploy the
> > services without human intervention. The problem is that, in this
> > scenario, the services may be deployed multiple times a day. Since
> > each deployment causes service interruption we've held back.
> >
> > So, my question is: how would one avoid service interruption during
> > deployment? Is there a common tool/strategy for such tasks?
> >
> > --
> > Phillip B Oldham
> > phillip.oldham@gmail.com
> > +44 (0) 7525 01 09 01
> >
>

Re: Strategies for continuous deployment of services to avoid interruption?

[Bryan Duxbury <br...@rapleaf.com>]

The strategy Rapleaf uses for purposes like this is to run multiple servers.
The client is aware of all the possible servers, but usually only connects
to one. When a connection becomes stale, you reconnect to another server.
Then, to make your deploys less painful, you just deploy one server at a
time.

On Mon, Jan 24, 2011 at 1:33 AM, Phillip B Oldham
<ph...@gmail.com>wrote:

> We have a number of Python & Java thrift services which we are
> manually deploying on a regular basis; usually early in the AM while
> it's "quiet" since deployment causes service interruption.
>
> We'd like to move to continuous deployment, so that when our commits
> successfully pass all the tests on our Hudson/Jenkins CI server
> something (Hudson/Jenkins, Puppet, custom scripts) will deploy the
> services without human intervention. The problem is that, in this
> scenario, the services may be deployed multiple times a day. Since
> each deployment causes service interruption we've held back.
>
> So, my question is: how would one avoid service interruption during
> deployment? Is there a common tool/strategy for such tasks?
>
> --
> Phillip B Oldham
> phillip.oldham@gmail.com
> +44 (0) 7525 01 09 01
>