I am looking for cross data center HA solution

[Santal Li <sa...@gmail.com>]

Greetings,

I am looking for cross data center HA solution for Cassandra.
I have do some evaluation with Cassandra's cross data center replication, my
test EVN as bellow:

  1. Deploy Node1, Node2 into data center A , and Node3 into data center B.
  2. Set each key have 3 replications ,  and using
"org.apache.cassandra.locator.RackAwareStrategy" for send one copy to remote
data center.
  3. If data center A was crashed, the data center B will have full copy,
our backup service in data center B will start, and connect to Node3.
  4. User will been redirect to this data center.

But Seems the RackAwareStrategy was not finished, some funtions was direct
throw out exception:

  eg:
       public EndPoint[] getReadStorageEndPoints(Token token, Map<Token,
EndPoint> tokenToEndPointMap)
       {
           throw new UnsupportedOperationException("This operation is not
currently supported");
       }

Was this strategy not finished or I should using some other Strategy, or I
should do some coding work to finish it?


There was another way for cross data center HA,  how about add some kind of
action hook, which can catch all the Cassandra.Iface function call(insert ,
batch_insert , remove ...) , then send the call to another Cassandra cluster
in remote data center? If so, there will be 1 Cassandra clusters in each 2
data centers, and have a extra application replicate data between the two
cluster. I prefer with this solution, because it looks more simple and
uncoupled 2 data center, BTW, from the source code, it looks was not very
hard to add a hook for this purpose , how do you think about it? is there
anyone meet the same requirements,  can you please give me some suggestion.


Regards
-Santal

Re: I am looking for cross data center HA solution

[Santal Li <sa...@gmail.com>]

Thanks for your quickly response, and also excellent job on this project.

For such kind of HA, we need do 2 way data sync. After failover to DC2, the
data change will be replicate to DC1 again, we had done same work with our
cross DC file system HA solution, so I prefer to this solution at beginning.
:)

Thanks for your suggestion, I think you are right, you guys do very good job
on the replication & failover with Cassandra, it was one of the best part of
this project. So let me try it. Thanks again.


Regards
-Santal



2009/10/14 Jonathan Ellis <jb...@gmail.com>

> On Wed, Oct 14, 2009 at 1:53 AM, Santal Li <sa...@gmail.com> wrote:
> > There was another way for cross data center HA,  how about add some kind
> of
> > action hook, which can catch all the Cassandra.Iface function call(insert
> ,
> > batch_insert , remove ...) , then send the call to another Cassandra
> cluster
> > in remote data center? If so, there will be 1 Cassandra clusters in each
> 2
> > data centers, and have a extra application replicate data between the two
> > cluster. I prefer with this solution, because it looks more simple and
> > uncoupled 2 data center, BTW, from the source code, it looks was not very
> > hard to add a hook for this purpose , how do you think about it? is there
> > anyone meet the same requirements,  can you please give me some
> suggestion.
>
> That works fine, until you actually have a failure.  Then how do you
> re-sync your clusters after you've done some writes in the second DC?
> The advantage to letting Cassandra handle replication + failover is
> all the hard parts are done already. :)
>
> -Jonathan
>

Re: I am looking for cross data center HA solution

[Jonathan Ellis <jb...@gmail.com>]

On Wed, Oct 14, 2009 at 1:53 AM, Santal Li <sa...@gmail.com> wrote:
> There was another way for cross data center HA,  how about add some kind of
> action hook, which can catch all the Cassandra.Iface function call(insert ,
> batch_insert , remove ...) , then send the call to another Cassandra cluster
> in remote data center? If so, there will be 1 Cassandra clusters in each 2
> data centers, and have a extra application replicate data between the two
> cluster. I prefer with this solution, because it looks more simple and
> uncoupled 2 data center, BTW, from the source code, it looks was not very
> hard to add a hook for this purpose , how do you think about it? is there
> anyone meet the same requirements,  can you please give me some suggestion.

That works fine, until you actually have a failure.  Then how do you
re-sync your clusters after you've done some writes in the second DC?
The advantage to letting Cassandra handle replication + failover is
all the hard parts are done already. :)

-Jonathan

Re: I am looking for cross data center HA solution

[Jonathan Ellis <jb...@gmail.com>]

On Wed, Oct 14, 2009 at 1:53 AM, Santal Li <sa...@gmail.com> wrote:
> But Seems the RackAwareStrategy was not finished, some funtions was direct
> throw out exception:
>
>   eg:
>        public EndPoint[] getReadStorageEndPoints(Token token, Map<Token,
> EndPoint> tokenToEndPointMap)
>        {
>            throw new UnsupportedOperationException("This operation is not
> currently supported");
>        }
>
> Was this strategy not finished or I should using some other Strategy, or I
> should do some coding work to finish it?

Everything should work fine with RackAware except bootstrapping.
We're working on finishing that for 0.5.

-Jonathan