Is it possible to achieve "sticky" request routing?

[Mukil Kesavan <we...@gmail.com>]

Hello,

We currently have 3 Cassandra servers running in a single datacenter with a
replication factor of 3 for our keyspace. We also use the SimpleSnitch
wiith DynamicSnitching enabled by default. Our load balancing policy is
TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child. This
overall configuration results in our client requests spreading equally
across our 3 servers.

However, we have a new requirement where we need to restrict a client's
requests to a single server and only go to the other servers on failure of
the previous server. This particular use case does not have high request
traffic.

Looking at the documentation the options we have seem to be:

1. Play with the snitching (e.g. place each server into its own DC or Rack)
to ensure that requests always go to one server and failover to the others
if required. I understand that this may also affect replica placement and
we may need to run nodetool repair. So this is not our most preferred
option.

2. Write a new load balancing policy that also uses the HostStateListener
for tracking host up and down messages, that essentially accomplishes
"sticky" request routing with failover to other nodes.

Is option 2 the only clean way of accomplishing our requirement?

Thanks,
Micky

Re: Is it possible to achieve "sticky" request routing?

[David King <dk...@ketralnis.com>]

Many C* versions back I did this by writing a custom snitch. This was to maximise use of the row cache but had the effect of pinning requests for a key to a given server. It sounds like you want to do the same thing but ignoring the key. In more modern reversions I believe this may be done as a load balancing policy.

I don't have the code handy but it was only a few dozen lines of Java, mostly boilerplate. IIRC the interface method just asks you to sort a list of IP addresses in your order of preference.

Note that this only really does anything for the ONE consistency level. In any other level you'll end up hitting all 3 servers anyway.


> On 02 Apr 2016, at 20:06, Mukil Kesavan <we...@gmail.com> wrote:
> 
> Hello,
> 
> We currently have 3 Cassandra servers running in a single datacenter with a replication factor of 3 for our keyspace. We also use the SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child. This overall configuration results in our client requests spreading equally across our 3 servers.
> 
> However, we have a new requirement where we need to restrict a client's requests to a single server and only go to the other servers on failure of the previous server. This particular use case does not have high request traffic.
> 
> Looking at the documentation the options we have seem to be:
> 
> 1. Play with the snitching (e.g. place each server into its own DC or Rack) to ensure that requests always go to one server and failover to the others if required. I understand that this may also affect replica placement and we may need to run nodetool repair. So this is not our most preferred option.
> 
> 2. Write a new load balancing policy that also uses the HostStateListener for tracking host up and down messages, that essentially accomplishes "sticky" request routing with failover to other nodes.
> 
> Is option 2 the only clean way of accomplishing our requirement?
> 
> Thanks,
> Micky

Re: Is it possible to achieve "sticky" request routing?

[Jonathan Haddad <jo...@jonhaddad.com>]

Jim, that's not what he asked. He asked for the equivalent of a load
balancer with sticky sessions.


On Tue, Apr 5, 2016 at 2:24 PM Jim Ancona <ji...@anconafamily.com> wrote:

> Jon and Steve:
>
> I don't understand your point. The TokenAwareLoadBalancer identifies the
> nodes in the cluster that own the data for a particular token and route
> requests to one of them. As I understand it, the OP wants to send requests
> for a particular token to the same node every time (assuming it's
> available). How does that fail in a large cluster?
>
> Jim
>
> On Tue, Apr 5, 2016 at 4:31 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
>> Yep - Steve hit the nail on the head.  The odds of hitting the right
>> server with "sticky routing" goes down as your cluster size increases.  You
>> end up adding extra network hops instead of using token aware routing.
>>
>> Unless you're trying to do a coordinator tier (and you're not, according
>> to your original post), this is a pretty bad idea and I'd advise you to
>> push back on that requirement.
>>
>> On Tue, Apr 5, 2016 at 12:47 PM Steve Robenalt <sr...@highwire.org>
>> wrote:
>>
>>> Aside from Jon's "why" question, I would point out that this only really
>>> works because you are running a 3 node cluster with RF=3. If your cluster
>>> is going to grow, you can't guarantee that any one server would have all
>>> records. I'd be pretty hesitant to put an invisible constraint like that on
>>> a cluster unless you're pretty sure it'll only ever be 3 nodes.
>>>
>>> On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad <jo...@jonhaddad.com>
>>> wrote:
>>>
>>>> Why is this a requirement?  Honestly I don't know why you would do this.
>>>>
>>>>
>>>> On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan <we...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> We currently have 3 Cassandra servers running in a single datacenter
>>>>> with a replication factor of 3 for our keyspace. We also use the
>>>>> SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing
>>>>> policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child.
>>>>> This overall configuration results in our client requests spreading equally
>>>>> across our 3 servers.
>>>>>
>>>>> However, we have a new requirement where we need to restrict a
>>>>> client's requests to a single server and only go to the other servers on
>>>>> failure of the previous server. This particular use case does not have high
>>>>> request traffic.
>>>>>
>>>>> Looking at the documentation the options we have seem to be:
>>>>>
>>>>> 1. Play with the snitching (e.g. place each server into its own DC or
>>>>> Rack) to ensure that requests always go to one server and failover to the
>>>>> others if required. I understand that this may also affect replica
>>>>> placement and we may need to run nodetool repair. So this is not our most
>>>>> preferred option.
>>>>>
>>>>> 2. Write a new load balancing policy that also uses the
>>>>> HostStateListener for tracking host up and down messages, that essentially
>>>>> accomplishes "sticky" request routing with failover to other nodes.
>>>>>
>>>>> Is option 2 the only clean way of accomplishing our requirement?
>>>>>
>>>>> Thanks,
>>>>> Micky
>>>>>
>>>>
>>>
>>>
>>> --
>>> Steve Robenalt
>>> Software Architect
>>> srobenalt@highwire.org <bz...@highwire.org>
>>> (office/cell): 916-505-1785
>>>
>>> HighWire Press, Inc.
>>> 425 Broadway St, Redwood City, CA 94063
>>> www.highwire.org
>>>
>>> Technology for Scholarly Communication
>>>
>>
>

Re: Is it possible to achieve "sticky" request routing?

[Jim Ancona <ji...@anconafamily.com>]

Jon and Steve:

I don't understand your point. The TokenAwareLoadBalancer identifies the
nodes in the cluster that own the data for a particular token and route
requests to one of them. As I understand it, the OP wants to send requests
for a particular token to the same node every time (assuming it's
available). How does that fail in a large cluster?

Jim

On Tue, Apr 5, 2016 at 4:31 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:

> Yep - Steve hit the nail on the head.  The odds of hitting the right
> server with "sticky routing" goes down as your cluster size increases.  You
> end up adding extra network hops instead of using token aware routing.
>
> Unless you're trying to do a coordinator tier (and you're not, according
> to your original post), this is a pretty bad idea and I'd advise you to
> push back on that requirement.
>
> On Tue, Apr 5, 2016 at 12:47 PM Steve Robenalt <sr...@highwire.org>
> wrote:
>
>> Aside from Jon's "why" question, I would point out that this only really
>> works because you are running a 3 node cluster with RF=3. If your cluster
>> is going to grow, you can't guarantee that any one server would have all
>> records. I'd be pretty hesitant to put an invisible constraint like that on
>> a cluster unless you're pretty sure it'll only ever be 3 nodes.
>>
>> On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad <jo...@jonhaddad.com>
>> wrote:
>>
>>> Why is this a requirement?  Honestly I don't know why you would do this.
>>>
>>>
>>> On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan <we...@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> We currently have 3 Cassandra servers running in a single datacenter
>>>> with a replication factor of 3 for our keyspace. We also use the
>>>> SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing
>>>> policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child.
>>>> This overall configuration results in our client requests spreading equally
>>>> across our 3 servers.
>>>>
>>>> However, we have a new requirement where we need to restrict a client's
>>>> requests to a single server and only go to the other servers on failure of
>>>> the previous server. This particular use case does not have high request
>>>> traffic.
>>>>
>>>> Looking at the documentation the options we have seem to be:
>>>>
>>>> 1. Play with the snitching (e.g. place each server into its own DC or
>>>> Rack) to ensure that requests always go to one server and failover to the
>>>> others if required. I understand that this may also affect replica
>>>> placement and we may need to run nodetool repair. So this is not our most
>>>> preferred option.
>>>>
>>>> 2. Write a new load balancing policy that also uses the
>>>> HostStateListener for tracking host up and down messages, that essentially
>>>> accomplishes "sticky" request routing with failover to other nodes.
>>>>
>>>> Is option 2 the only clean way of accomplishing our requirement?
>>>>
>>>> Thanks,
>>>> Micky
>>>>
>>>
>>
>>
>> --
>> Steve Robenalt
>> Software Architect
>> srobenalt@highwire.org <bz...@highwire.org>
>> (office/cell): 916-505-1785
>>
>> HighWire Press, Inc.
>> 425 Broadway St, Redwood City, CA 94063
>> www.highwire.org
>>
>> Technology for Scholarly Communication
>>
>

Re: Is it possible to achieve "sticky" request routing?

[Steve Robenalt <sr...@highwire.org>]

Hi Micky,

The only issues I've seen personally with dropped updates due to even small
amounts of time skew were ones I was able to fix by the judicious use of
BatchStatements. This was particularly true with counter fields in the 2.0
release (i.e. before counters were fixed), but would also happen with
different columns updated in separate statements. I'm not sure what your
circumstances are for the lost updates, so I'm not sure if these will help.
I'm only pointing it out because it was effective for my cases.

Steve


On Tue, Apr 5, 2016 at 3:21 PM, Mukil Kesavan <we...@gmail.com>
wrote:

> John and Steve,
>
> We use QUORUM consistency for both READS and WRITES. So we won't have the
> problem of having to pick the right server. The real reason we have this
> requirement is because we run in a sometimes overloaded virtualized
> environment that causes the servers to have non-trivial time drifts (tens
> of milliseconds or a few seconds, even with NTP, which catches up slowly).
> This particular client was seeing some updates being dropped silently by
> Cassandra when it hit a server with an older local timestamp. They were
> relying on server side timestamp generation.
>
> So they were exploring "sticky" routing as an option since the likelihood
> of having monotonically increasing timestamps is higher if the client's
> requests always go to a single server. They are aware of the problem of
> disconnecting and reconnecting to a new server and have an application
> level solution for this. They are also testing using client side timestamps
> as well. We're just trying to explore all our options and their pros and
> cons.
>
> Thanks!
>
> On Tue, Apr 5, 2016 at 1:31 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
>> Yep - Steve hit the nail on the head.  The odds of hitting the right
>> server with "sticky routing" goes down as your cluster size increases.  You
>> end up adding extra network hops instead of using token aware routing.
>>
>> Unless you're trying to do a coordinator tier (and you're not, according
>> to your original post), this is a pretty bad idea and I'd advise you to
>> push back on that requirement.
>>
>> On Tue, Apr 5, 2016 at 12:47 PM Steve Robenalt <sr...@highwire.org>
>> wrote:
>>
>>> Aside from Jon's "why" question, I would point out that this only really
>>> works because you are running a 3 node cluster with RF=3. If your cluster
>>> is going to grow, you can't guarantee that any one server would have all
>>> records. I'd be pretty hesitant to put an invisible constraint like that on
>>> a cluster unless you're pretty sure it'll only ever be 3 nodes.
>>>
>>> On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad <jo...@jonhaddad.com>
>>> wrote:
>>>
>>>> Why is this a requirement?  Honestly I don't know why you would do this.
>>>>
>>>>
>>>> On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan <we...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> We currently have 3 Cassandra servers running in a single datacenter
>>>>> with a replication factor of 3 for our keyspace. We also use the
>>>>> SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing
>>>>> policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child.
>>>>> This overall configuration results in our client requests spreading equally
>>>>> across our 3 servers.
>>>>>
>>>>> However, we have a new requirement where we need to restrict a
>>>>> client's requests to a single server and only go to the other servers on
>>>>> failure of the previous server. This particular use case does not have high
>>>>> request traffic.
>>>>>
>>>>> Looking at the documentation the options we have seem to be:
>>>>>
>>>>> 1. Play with the snitching (e.g. place each server into its own DC or
>>>>> Rack) to ensure that requests always go to one server and failover to the
>>>>> others if required. I understand that this may also affect replica
>>>>> placement and we may need to run nodetool repair. So this is not our most
>>>>> preferred option.
>>>>>
>>>>> 2. Write a new load balancing policy that also uses the
>>>>> HostStateListener for tracking host up and down messages, that essentially
>>>>> accomplishes "sticky" request routing with failover to other nodes.
>>>>>
>>>>> Is option 2 the only clean way of accomplishing our requirement?
>>>>>
>>>>> Thanks,
>>>>> Micky
>>>>>
>>>>
>>>
>>>
>>> --
>>> Steve Robenalt
>>> Software Architect
>>> srobenalt@highwire.org <bz...@highwire.org>
>>> (office/cell): 916-505-1785
>>>
>>> HighWire Press, Inc.
>>> 425 Broadway St, Redwood City, CA 94063
>>> www.highwire.org
>>>
>>> Technology for Scholarly Communication
>>>
>>
>


-- 
Steve Robenalt
Software Architect
srobenalt@highwire.org <bz...@highwire.org>
(office/cell): 916-505-1785

HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org

Technology for Scholarly Communication

Re: Is it possible to achieve "sticky" request routing?

[Mukil Kesavan <we...@gmail.com>]

John and Steve,

We use QUORUM consistency for both READS and WRITES. So we won't have the
problem of having to pick the right server. The real reason we have this
requirement is because we run in a sometimes overloaded virtualized
environment that causes the servers to have non-trivial time drifts (tens
of milliseconds or a few seconds, even with NTP, which catches up slowly).
This particular client was seeing some updates being dropped silently by
Cassandra when it hit a server with an older local timestamp. They were
relying on server side timestamp generation.

So they were exploring "sticky" routing as an option since the likelihood
of having monotonically increasing timestamps is higher if the client's
requests always go to a single server. They are aware of the problem of
disconnecting and reconnecting to a new server and have an application
level solution for this. They are also testing using client side timestamps
as well. We're just trying to explore all our options and their pros and
cons.

Thanks!

On Tue, Apr 5, 2016 at 1:31 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:

> Yep - Steve hit the nail on the head.  The odds of hitting the right
> server with "sticky routing" goes down as your cluster size increases.  You
> end up adding extra network hops instead of using token aware routing.
>
> Unless you're trying to do a coordinator tier (and you're not, according
> to your original post), this is a pretty bad idea and I'd advise you to
> push back on that requirement.
>
> On Tue, Apr 5, 2016 at 12:47 PM Steve Robenalt <sr...@highwire.org>
> wrote:
>
>> Aside from Jon's "why" question, I would point out that this only really
>> works because you are running a 3 node cluster with RF=3. If your cluster
>> is going to grow, you can't guarantee that any one server would have all
>> records. I'd be pretty hesitant to put an invisible constraint like that on
>> a cluster unless you're pretty sure it'll only ever be 3 nodes.
>>
>> On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad <jo...@jonhaddad.com>
>> wrote:
>>
>>> Why is this a requirement?  Honestly I don't know why you would do this.
>>>
>>>
>>> On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan <we...@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> We currently have 3 Cassandra servers running in a single datacenter
>>>> with a replication factor of 3 for our keyspace. We also use the
>>>> SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing
>>>> policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child.
>>>> This overall configuration results in our client requests spreading equally
>>>> across our 3 servers.
>>>>
>>>> However, we have a new requirement where we need to restrict a client's
>>>> requests to a single server and only go to the other servers on failure of
>>>> the previous server. This particular use case does not have high request
>>>> traffic.
>>>>
>>>> Looking at the documentation the options we have seem to be:
>>>>
>>>> 1. Play with the snitching (e.g. place each server into its own DC or
>>>> Rack) to ensure that requests always go to one server and failover to the
>>>> others if required. I understand that this may also affect replica
>>>> placement and we may need to run nodetool repair. So this is not our most
>>>> preferred option.
>>>>
>>>> 2. Write a new load balancing policy that also uses the
>>>> HostStateListener for tracking host up and down messages, that essentially
>>>> accomplishes "sticky" request routing with failover to other nodes.
>>>>
>>>> Is option 2 the only clean way of accomplishing our requirement?
>>>>
>>>> Thanks,
>>>> Micky
>>>>
>>>
>>
>>
>> --
>> Steve Robenalt
>> Software Architect
>> srobenalt@highwire.org <bz...@highwire.org>
>> (office/cell): 916-505-1785
>>
>> HighWire Press, Inc.
>> 425 Broadway St, Redwood City, CA 94063
>> www.highwire.org
>>
>> Technology for Scholarly Communication
>>
>

Re: Is it possible to achieve "sticky" request routing?

[Jonathan Haddad <jo...@jonhaddad.com>]

Yep - Steve hit the nail on the head.  The odds of hitting the right server
with "sticky routing" goes down as your cluster size increases.  You end up
adding extra network hops instead of using token aware routing.

Unless you're trying to do a coordinator tier (and you're not, according to
your original post), this is a pretty bad idea and I'd advise you to push
back on that requirement.

On Tue, Apr 5, 2016 at 12:47 PM Steve Robenalt <sr...@highwire.org>
wrote:

> Aside from Jon's "why" question, I would point out that this only really
> works because you are running a 3 node cluster with RF=3. If your cluster
> is going to grow, you can't guarantee that any one server would have all
> records. I'd be pretty hesitant to put an invisible constraint like that on
> a cluster unless you're pretty sure it'll only ever be 3 nodes.
>
> On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
>> Why is this a requirement?  Honestly I don't know why you would do this.
>>
>>
>> On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan <we...@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> We currently have 3 Cassandra servers running in a single datacenter
>>> with a replication factor of 3 for our keyspace. We also use the
>>> SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing
>>> policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child.
>>> This overall configuration results in our client requests spreading equally
>>> across our 3 servers.
>>>
>>> However, we have a new requirement where we need to restrict a client's
>>> requests to a single server and only go to the other servers on failure of
>>> the previous server. This particular use case does not have high request
>>> traffic.
>>>
>>> Looking at the documentation the options we have seem to be:
>>>
>>> 1. Play with the snitching (e.g. place each server into its own DC or
>>> Rack) to ensure that requests always go to one server and failover to the
>>> others if required. I understand that this may also affect replica
>>> placement and we may need to run nodetool repair. So this is not our most
>>> preferred option.
>>>
>>> 2. Write a new load balancing policy that also uses the
>>> HostStateListener for tracking host up and down messages, that essentially
>>> accomplishes "sticky" request routing with failover to other nodes.
>>>
>>> Is option 2 the only clean way of accomplishing our requirement?
>>>
>>> Thanks,
>>> Micky
>>>
>>
>
>
> --
> Steve Robenalt
> Software Architect
> srobenalt@highwire.org <bz...@highwire.org>
> (office/cell): 916-505-1785
>
> HighWire Press, Inc.
> 425 Broadway St, Redwood City, CA 94063
> www.highwire.org
>
> Technology for Scholarly Communication
>

Re: Is it possible to achieve "sticky" request routing?

[Steve Robenalt <sr...@highwire.org>]

Aside from Jon's "why" question, I would point out that this only really
works because you are running a 3 node cluster with RF=3. If your cluster
is going to grow, you can't guarantee that any one server would have all
records. I'd be pretty hesitant to put an invisible constraint like that on
a cluster unless you're pretty sure it'll only ever be 3 nodes.

On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad <jo...@jonhaddad.com> wrote:

> Why is this a requirement?  Honestly I don't know why you would do this.
>
>
> On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan <we...@gmail.com>
> wrote:
>
>> Hello,
>>
>> We currently have 3 Cassandra servers running in a single datacenter with
>> a replication factor of 3 for our keyspace. We also use the SimpleSnitch
>> wiith DynamicSnitching enabled by default. Our load balancing policy is
>> TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child. This
>> overall configuration results in our client requests spreading equally
>> across our 3 servers.
>>
>> However, we have a new requirement where we need to restrict a client's
>> requests to a single server and only go to the other servers on failure of
>> the previous server. This particular use case does not have high request
>> traffic.
>>
>> Looking at the documentation the options we have seem to be:
>>
>> 1. Play with the snitching (e.g. place each server into its own DC or
>> Rack) to ensure that requests always go to one server and failover to the
>> others if required. I understand that this may also affect replica
>> placement and we may need to run nodetool repair. So this is not our most
>> preferred option.
>>
>> 2. Write a new load balancing policy that also uses the HostStateListener
>> for tracking host up and down messages, that essentially accomplishes
>> "sticky" request routing with failover to other nodes.
>>
>> Is option 2 the only clean way of accomplishing our requirement?
>>
>> Thanks,
>> Micky
>>
>


-- 
Steve Robenalt
Software Architect
srobenalt@highwire.org <bz...@highwire.org>
(office/cell): 916-505-1785

HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org

Technology for Scholarly Communication

Re: Is it possible to achieve "sticky" request routing?

[Jonathan Haddad <jo...@jonhaddad.com>]

Why is this a requirement?  Honestly I don't know why you would do this.

On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan <we...@gmail.com>
wrote:

> Hello,
>
> We currently have 3 Cassandra servers running in a single datacenter with
> a replication factor of 3 for our keyspace. We also use the SimpleSnitch
> wiith DynamicSnitching enabled by default. Our load balancing policy is
> TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child. This
> overall configuration results in our client requests spreading equally
> across our 3 servers.
>
> However, we have a new requirement where we need to restrict a client's
> requests to a single server and only go to the other servers on failure of
> the previous server. This particular use case does not have high request
> traffic.
>
> Looking at the documentation the options we have seem to be:
>
> 1. Play with the snitching (e.g. place each server into its own DC or
> Rack) to ensure that requests always go to one server and failover to the
> others if required. I understand that this may also affect replica
> placement and we may need to run nodetool repair. So this is not our most
> preferred option.
>
> 2. Write a new load balancing policy that also uses the HostStateListener
> for tracking host up and down messages, that essentially accomplishes
> "sticky" request routing with failover to other nodes.
>
> Is option 2 the only clean way of accomplishing our requirement?
>
> Thanks,
> Micky
>