mysterious data disappearance - what happened?

[Yiming Sun <yi...@gmail.com>]

Hello,

If two different instances of cassandra are running on separate machines,
but both are unfortunately configured to use the default cluster name, "Test
Cluster", do they gang up as one cluster (even though they were intended to
be two separate stand-alone instances), so that dropping keyspace on one
machine would result in the disappearance of data from the other?

We ran into some mysterious data disappearance situations -- and we are not
sure if this is something with our configuration.

We installed 2 instances of cassandra 0.8.2 on 2 separate machines, one is
silk.cs.our.domain and the other smoke.cs.our.domain.  Both the code and
data directories reside on the local partition of each machine, and we left
the the yaml file pretty much the default, except we changed the data
directory paths and also the listen addresses to point to each host.  What I
want to point out is, both files had the default cluster name "Test
Cluster".

Both cassandra instances were loaded with identical keyspaces, data sets,
and the intention was one would be used as production and the other testing
cassandra server.

Then yesterday we were trying to drop a keyspace on silk, which was the
testing server, and sometimes later, a user complained that he could not get
any data from the production machine.  We were intrigued but thought maybe
someone made a mistake.

Then the same thing happened again today.  So we began to suspect that two
servers somehow discovered each other.  In fact, after we shut down the
cassandra on silk, and then load a keyspace definition into smoke using
cassandra-cli, it even said unreachable destination and the IP was that of
silk.  I just want to confirm that the culprit was because we used the same
cluster name for both machines.  So in other words, if we want to launch two
separate instances of cassandra and keep them separate, we must make sure
each uses a different cluster name or else they will gang up into the same
cluster?  But how do they even discover each other?  Can someone enlighten
me please?  Thanks.


-- Y.

Re: mysterious data disappearance - what happened?

[Yiming Sun <yi...@gmail.com>]

Hi Peter,

Good call! I went and checked the seed, and indeed it left unchanged when we
copied the config yaml file from the first node to the second node.  Thanks!

-- Y.

On Fri, Sep 9, 2011 at 7:47 PM, Peter Schuller
<pe...@infidyne.com>wrote:

> > cluster name for both machines.  So in other words, if we want to launch
> two
> > separate instances of cassandra and keep them separate, we must make sure
> > each uses a different cluster name or else they will gang up into the
> same
> > cluster?  But how do they even discover each other?  Can someone
> enlighten
> > me please?  Thanks.
>
> It is highly recommended to use distinct cluster names, in particular
> because it can help avoid accidentally "merging" two independent
> clusters.
>
> As for how it happened: There is no magic discovery going on that
> would pick IP:s at random, but one could certainly e.g. accidentally
> configure the seed node on one to point to the other or some such.
>
> (1) does nodetool -h localhost ring show an unexpected node?
> (2) i'd suggest checking system.log on each node for the first
> appearance (if any) of the "unexpected" ip address and correlate (by
> time) with what happened on the other node (was it restarted at the
> time for example, potentially wth a bad conf?)
> (3) are these two single-instance cassandras that have never
> participated in another cluster?
>
> --
> / Peter Schuller (@scode on twitter)
>

Re: mysterious data disappearance - what happened?

[Peter Schuller <pe...@infidyne.com>]

> cluster name for both machines.  So in other words, if we want to launch two
> separate instances of cassandra and keep them separate, we must make sure
> each uses a different cluster name or else they will gang up into the same
> cluster?  But how do they even discover each other?  Can someone enlighten
> me please?  Thanks.

It is highly recommended to use distinct cluster names, in particular
because it can help avoid accidentally "merging" two independent
clusters.

As for how it happened: There is no magic discovery going on that
would pick IP:s at random, but one could certainly e.g. accidentally
configure the seed node on one to point to the other or some such.

(1) does nodetool -h localhost ring show an unexpected node?
(2) i'd suggest checking system.log on each node for the first
appearance (if any) of the "unexpected" ip address and correlate (by
time) with what happened on the other node (was it restarted at the
time for example, potentially wth a bad conf?)
(3) are these two single-instance cassandras that have never
participated in another cluster?

-- 
/ Peter Schuller (@scode on twitter)