You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Stephan Pfammatter <St...@logmein.com> on 2010/06/03 22:07:07 UTC

Cassandra Cluster Setup

I'm having difficulties setting up a 3 way cassandra cluster. Any comments/help would be appreciated.

My goal is that all data should be fully replicated amongst the 3 nodes. I want to simulate the failure of one node and proof that the test column family still can be accessed.
In a nutshell I did the following 3 steps. The excerpt below shows the changed storage-con.xml:

(1)    I started Cassandra-ca and worked fine. I added 3 rows to a cf cf_test.

(2)    Then I started Cassandra-or. It comes up ok but goes into "sleeping 30000".....mode

(3)    Nodetool streams doesn't reveal anything.



Node: Cassandra-ca
<AutoBootstrap>false</AutoBootstrap>
<Keyspace Name="ks_test">
                <ColumnFamily Name="cf_test" CompareWith="UTF8Type" />
        <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
                <ReplicationFactor>1</ReplicationFactor>
                <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
</Keyspace>
  <Seeds><Seed>cassandra-ca</Seed></Seeds>
  <ThriftAddress>cassandra-us</ThriftAddress>
 <ThriftPort>9160</ThriftPort>
<ListenAddress>cassandra-us</ListenAddress>

Cassandra-or
<AutoBootstrap>true</AutoBootstrap>
<Keyspace Name="ks_test">
                <ColumnFamily Name="cf_test" CompareWith="UTF8Type" />
        <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
                <ReplicationFactor>1</ReplicationFactor>
                <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
</Keyspace>
  <Seeds><Seed>cassandra-or</Seed></Seeds>
  <ThriftAddress>cassandra-or</ThriftAddress>
 <ThriftPort>9160</ThriftPort>
<ListenAddress>cassandra-or</ListenAddress>

Cassandra-az

Cassandra-or
<AutoBootstrap>true</AutoBootstrap>
<Keyspace Name="ks_test">
                <ColumnFamily Name="cf_test" CompareWith="UTF8Type" />
        <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
                <ReplicationFactor>1</ReplicationFactor>
                <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
</Keyspace>
  <Seeds><Seed>cassandra-az</Seed></Seeds>
  <ThriftAddress>cassandra-az</ThriftAddress>
 <ThriftPort>9160</ThriftPort>
<ListenAddress>cassandra-az</ListenAddress>




Re: Cassandra Cluster Setup

Posted by Gary Dusbabek <gd...@gmail.com>.
Your replication factor is only set to 1, which means that each key
will only live on a single node.  If you do wait for bootstrapping to
commence (takes 90s in trunk, I don't recall in 0.6), you should see
some keys moving unless your inserts were all into a small range.
Perhaps your being impatient.

If not, I recommend that you start over, set the replication factor to
3, wait a good while for all the nodes to fully join the cluster, and
then make sure the keys you write to are random.

Gary.


On Thu, Jun 3, 2010 at 13:07, Stephan Pfammatter
<St...@logmein.com> wrote:
> I’m having difficulties setting up a 3 way cassandra cluster. Any
> comments/help would be appreciated.
>
>
>
> My goal is that all data should be fully replicated amongst the 3 nodes. I
> want to simulate the failure of one node and proof that the test column
> family still can be accessed.
>
> In a nutshell I did the following 3 steps. The excerpt below shows the
> changed storage-con.xml:
>
> (1)    I started Cassandra-ca and worked fine. I added 3 rows to a cf
> cf_test.
>
> (2)    Then I started Cassandra-or. It comes up ok but goes into “sleeping
> 30000”…..mode
>
> (3)    Nodetool streams doesn’t reveal anything.
>
>
>
>
>
>
>
> Node: Cassandra-ca
>
> <AutoBootstrap>false</AutoBootstrap>
>
> <Keyspace Name="ks_test">
>
>                 <ColumnFamily Name="cf_test" CompareWith="UTF8Type" />
>
>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>
>                 <ReplicationFactor>1</ReplicationFactor>
>
>
> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>
> </Keyspace>
>
>   <Seeds><Seed>cassandra-ca</Seed></Seeds>
>
>   <ThriftAddress>cassandra-us</ThriftAddress>
>
>  <ThriftPort>9160</ThriftPort>
>
> <ListenAddress>cassandra-us</ListenAddress>
>
>
>
> Cassandra-or
>
> <AutoBootstrap>true</AutoBootstrap>
>
> <Keyspace Name="ks_test">
>
>                 <ColumnFamily Name="cf_test" CompareWith="UTF8Type" />
>
>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>
>                 <ReplicationFactor>1</ReplicationFactor>
>
>
> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>
> </Keyspace>
>
>   <Seeds><Seed>cassandra-or</Seed></Seeds>
>
>   <ThriftAddress>cassandra-or</ThriftAddress>
>
>  <ThriftPort>9160</ThriftPort>
>
> <ListenAddress>cassandra-or</ListenAddress>
>
>
>
> Cassandra-az
>
>
>
> Cassandra-or
>
> <AutoBootstrap>true</AutoBootstrap>
>
> <Keyspace Name="ks_test">
>
>                 <ColumnFamily Name="cf_test" CompareWith="UTF8Type" />
>
>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>
>                 <ReplicationFactor>1</ReplicationFactor>
>
>
> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>
> </Keyspace>
>
>   <Seeds><Seed>cassandra-az</Seed></Seeds>
>
>   <ThriftAddress>cassandra-az</ThriftAddress>
>
>  <ThriftPort>9160</ThriftPort>
>
> <ListenAddress>cassandra-az</ListenAddress>
>
>
>
>
>
>

RE: Cassandra Cluster Setup

Posted by Stephan Pfammatter <St...@logmein.com>.
Nice catch Gary. Just realized that replication factor is granular to key space. 

-----Original Message-----
From: Gary Dusbabek [mailto:gdusbabek@gmail.com] 
Sent: Friday, June 04, 2010 11:49 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra Cluster Setup

Great.  It looks as if the replication factor is still 1, not 3.  This
means that each key lives only on one node.  By increasing it to 3,
data will be replicated across all 3 nodes.

Gary.


On Fri, Jun 4, 2010 at 06:51, Stephan Pfammatter
<St...@logmein.com> wrote:
> Tx Nahor/Gary/Ben. I blew everything away, adjusted the seed to Cassandra-ca and changed the replication factor.
> Now I can see the sample keys being spread across the 3 nodes:
> Cassandra-ca holds rows (keys: 1,5,8)
> Cassandra-az holds rows (keys: 11,55)
> Cassandra-or hodls rows (keys: 88)
>
> I don't see the data being duplicated yet. Am I missing a storage-conf setting? I need to proof that if Cassandra-az fails that I can still provide key 11,55.
>
> -----Original Message-----
> From: Nahor [mailto:nahor.j+gmane@gmail.com]
> Sent: Thursday, June 03, 2010 8:25 PM
> To: user@cassandra.apache.org
> Subject: Re: Cassandra Cluster Setup
>
>  On 2010-06-03 13:07, Stephan Pfammatter wrote:
>> Cassandra-or
>> [...]
>> <Seeds><Seed>cassandra-or</Seed></Seeds>
>
>
> Aside from the replication factor noted by Gary, this should point to
> your existing node (cassandra-ca) otherwise, how will this node know
> where existing node is and where to get the data from?
>
>
>> Cassandra-az
>> [...]
>> <Seeds><Seed>cassandra-az</Seed></Seeds>
>
> Same here
>
>
>

Re: Cassandra Cluster Setup

Posted by Gary Dusbabek <gd...@gmail.com>.
It should take roughly 30 seconds (please clarify--not minutes) to
join the cluster.

Gary.


On Mon, Jun 7, 2010 at 08:23, Stephan Pfammatter
<St...@logmein.com> wrote:
> I still have one open issue: My cluster has only 3 local nodes with 1 CF (5 rows).
> The initial start works fine. I can see the replicated data and can do all admin tasks I want, successfully.
>
> Now I'm simulating a node failure by stopping Cassandra. I remove
> (a) commit log
> (b) content in system keyspace
> (c) application cf
>
> When I bring the node back up I can see the new token assignment, streaming notification (we are talking about 5 rows in 1 cf). Thrift port is not available for ~30 min. Why is that? I would expect it to be back in a matter of maybe <1' (this should be data quantity related).
>
> Thanks, Stephan
>
>
> -----Original Message-----
> From: Gary Dusbabek [mailto:gdusbabek@gmail.com]
> Sent: Friday, June 04, 2010 11:49 AM
> To: user@cassandra.apache.org
> Subject: Re: Cassandra Cluster Setup
>
> Great.  It looks as if the replication factor is still 1, not 3.  This
> means that each key lives only on one node.  By increasing it to 3,
> data will be replicated across all 3 nodes.
>
> Gary.
>
>
> On Fri, Jun 4, 2010 at 06:51, Stephan Pfammatter
> <St...@logmein.com> wrote:
>> Tx Nahor/Gary/Ben. I blew everything away, adjusted the seed to Cassandra-ca and changed the replication factor.
>> Now I can see the sample keys being spread across the 3 nodes:
>> Cassandra-ca holds rows (keys: 1,5,8)
>> Cassandra-az holds rows (keys: 11,55)
>> Cassandra-or hodls rows (keys: 88)
>>
>> I don't see the data being duplicated yet. Am I missing a storage-conf setting? I need to proof that if Cassandra-az fails that I can still provide key 11,55.
>>
>> -----Original Message-----
>> From: Nahor [mailto:nahor.j+gmane@gmail.com]
>> Sent: Thursday, June 03, 2010 8:25 PM
>> To: user@cassandra.apache.org
>> Subject: Re: Cassandra Cluster Setup
>>
>>  On 2010-06-03 13:07, Stephan Pfammatter wrote:
>>> Cassandra-or
>>> [...]
>>> <Seeds><Seed>cassandra-or</Seed></Seeds>
>>
>>
>> Aside from the replication factor noted by Gary, this should point to
>> your existing node (cassandra-ca) otherwise, how will this node know
>> where existing node is and where to get the data from?
>>
>>
>>> Cassandra-az
>>> [...]
>>> <Seeds><Seed>cassandra-az</Seed></Seeds>
>>
>> Same here
>>
>>
>>
>

RE: Cassandra Cluster Setup

Posted by Stephan Pfammatter <St...@logmein.com>.
I still have one open issue: My cluster has only 3 local nodes with 1 CF (5 rows).
The initial start works fine. I can see the replicated data and can do all admin tasks I want, successfully.

Now I'm simulating a node failure by stopping Cassandra. I remove 
(a) commit log
(b) content in system keyspace
(c) application cf

When I bring the node back up I can see the new token assignment, streaming notification (we are talking about 5 rows in 1 cf). Thrift port is not available for ~30 min. Why is that? I would expect it to be back in a matter of maybe <1' (this should be data quantity related).

Thanks, Stephan


-----Original Message-----
From: Gary Dusbabek [mailto:gdusbabek@gmail.com] 
Sent: Friday, June 04, 2010 11:49 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra Cluster Setup

Great.  It looks as if the replication factor is still 1, not 3.  This
means that each key lives only on one node.  By increasing it to 3,
data will be replicated across all 3 nodes.

Gary.


On Fri, Jun 4, 2010 at 06:51, Stephan Pfammatter
<St...@logmein.com> wrote:
> Tx Nahor/Gary/Ben. I blew everything away, adjusted the seed to Cassandra-ca and changed the replication factor.
> Now I can see the sample keys being spread across the 3 nodes:
> Cassandra-ca holds rows (keys: 1,5,8)
> Cassandra-az holds rows (keys: 11,55)
> Cassandra-or hodls rows (keys: 88)
>
> I don't see the data being duplicated yet. Am I missing a storage-conf setting? I need to proof that if Cassandra-az fails that I can still provide key 11,55.
>
> -----Original Message-----
> From: Nahor [mailto:nahor.j+gmane@gmail.com]
> Sent: Thursday, June 03, 2010 8:25 PM
> To: user@cassandra.apache.org
> Subject: Re: Cassandra Cluster Setup
>
>  On 2010-06-03 13:07, Stephan Pfammatter wrote:
>> Cassandra-or
>> [...]
>> <Seeds><Seed>cassandra-or</Seed></Seeds>
>
>
> Aside from the replication factor noted by Gary, this should point to
> your existing node (cassandra-ca) otherwise, how will this node know
> where existing node is and where to get the data from?
>
>
>> Cassandra-az
>> [...]
>> <Seeds><Seed>cassandra-az</Seed></Seeds>
>
> Same here
>
>
>

Re: Cassandra Cluster Setup

Posted by Gary Dusbabek <gd...@gmail.com>.
Great.  It looks as if the replication factor is still 1, not 3.  This
means that each key lives only on one node.  By increasing it to 3,
data will be replicated across all 3 nodes.

Gary.


On Fri, Jun 4, 2010 at 06:51, Stephan Pfammatter
<St...@logmein.com> wrote:
> Tx Nahor/Gary/Ben. I blew everything away, adjusted the seed to Cassandra-ca and changed the replication factor.
> Now I can see the sample keys being spread across the 3 nodes:
> Cassandra-ca holds rows (keys: 1,5,8)
> Cassandra-az holds rows (keys: 11,55)
> Cassandra-or hodls rows (keys: 88)
>
> I don't see the data being duplicated yet. Am I missing a storage-conf setting? I need to proof that if Cassandra-az fails that I can still provide key 11,55.
>
> -----Original Message-----
> From: Nahor [mailto:nahor.j+gmane@gmail.com]
> Sent: Thursday, June 03, 2010 8:25 PM
> To: user@cassandra.apache.org
> Subject: Re: Cassandra Cluster Setup
>
>  On 2010-06-03 13:07, Stephan Pfammatter wrote:
>> Cassandra-or
>> [...]
>> <Seeds><Seed>cassandra-or</Seed></Seeds>
>
>
> Aside from the replication factor noted by Gary, this should point to
> your existing node (cassandra-ca) otherwise, how will this node know
> where existing node is and where to get the data from?
>
>
>> Cassandra-az
>> [...]
>> <Seeds><Seed>cassandra-az</Seed></Seeds>
>
> Same here
>
>
>

RE: Cassandra Cluster Setup

Posted by Stephan Pfammatter <St...@logmein.com>.
Tx Nahor/Gary/Ben. I blew everything away, adjusted the seed to Cassandra-ca and changed the replication factor.
Now I can see the sample keys being spread across the 3 nodes:
Cassandra-ca holds rows (keys: 1,5,8)
Cassandra-az holds rows (keys: 11,55)
Cassandra-or hodls rows (keys: 88)

I don't see the data being duplicated yet. Am I missing a storage-conf setting? I need to proof that if Cassandra-az fails that I can still provide key 11,55. 

-----Original Message-----
From: Nahor [mailto:nahor.j+gmane@gmail.com] 
Sent: Thursday, June 03, 2010 8:25 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra Cluster Setup

  On 2010-06-03 13:07, Stephan Pfammatter wrote:
> Cassandra-or
> [...]
> <Seeds><Seed>cassandra-or</Seed></Seeds>


Aside from the replication factor noted by Gary, this should point to 
your existing node (cassandra-ca) otherwise, how will this node know 
where existing node is and where to get the data from?


> Cassandra-az
> [...]
> <Seeds><Seed>cassandra-az</Seed></Seeds>

Same here



Re: Cassandra Cluster Setup

Posted by Nahor <na...@gmail.com>.
  On 2010-06-03 13:07, Stephan Pfammatter wrote:
> Cassandra-or
> [...]
> <Seeds><Seed>cassandra-or</Seed></Seeds>


Aside from the replication factor noted by Gary, this should point to 
your existing node (cassandra-ca) otherwise, how will this node know 
where existing node is and where to get the data from?


> Cassandra-az
> [...]
> <Seeds><Seed>cassandra-az</Seed></Seeds>

Same here



Re: Cassandra Cluster Setup

Posted by Benjamin Black <b...@b3k.us>.
http://wiki.apache.org/cassandra/MultinodeCluster

On Thu, Jun 3, 2010 at 1:07 PM, Stephan Pfammatter
<St...@logmein.com> wrote:
> I’m having difficulties setting up a 3 way cassandra cluster. Any
> comments/help would be appreciated.
>
>
>
> My goal is that all data should be fully replicated amongst the 3 nodes. I
> want to simulate the failure of one node and proof that the test column
> family still can be accessed.
>
> In a nutshell I did the following 3 steps. The excerpt below shows the
> changed storage-con.xml:
>
> (1)    I started Cassandra-ca and worked fine. I added 3 rows to a cf
> cf_test.
>
> (2)    Then I started Cassandra-or. It comes up ok but goes into “sleeping
> 30000”…..mode
>
> (3)    Nodetool streams doesn’t reveal anything.
>
>
>
>
>
>
>
> Node: Cassandra-ca
>
> <AutoBootstrap>false</AutoBootstrap>
>
> <Keyspace Name="ks_test">
>
>                 <ColumnFamily Name="cf_test" CompareWith="UTF8Type" />
>
>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>
>                 <ReplicationFactor>1</ReplicationFactor>
>
>
> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>
> </Keyspace>
>
>   <Seeds><Seed>cassandra-ca</Seed></Seeds>
>
>   <ThriftAddress>cassandra-us</ThriftAddress>
>
>  <ThriftPort>9160</ThriftPort>
>
> <ListenAddress>cassandra-us</ListenAddress>
>
>
>
> Cassandra-or
>
> <AutoBootstrap>true</AutoBootstrap>
>
> <Keyspace Name="ks_test">
>
>                 <ColumnFamily Name="cf_test" CompareWith="UTF8Type" />
>
>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>
>                 <ReplicationFactor>1</ReplicationFactor>
>
>
> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>
> </Keyspace>
>
>   <Seeds><Seed>cassandra-or</Seed></Seeds>
>
>   <ThriftAddress>cassandra-or</ThriftAddress>
>
>  <ThriftPort>9160</ThriftPort>
>
> <ListenAddress>cassandra-or</ListenAddress>
>
>
>
> Cassandra-az
>
>
>
> Cassandra-or
>
> <AutoBootstrap>true</AutoBootstrap>
>
> <Keyspace Name="ks_test">
>
>                 <ColumnFamily Name="cf_test" CompareWith="UTF8Type" />
>
>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>
>                 <ReplicationFactor>1</ReplicationFactor>
>
>
> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>
> </Keyspace>
>
>   <Seeds><Seed>cassandra-az</Seed></Seeds>
>
>   <ThriftAddress>cassandra-az</ThriftAddress>
>
>  <ThriftPort>9160</ThriftPort>
>
> <ListenAddress>cassandra-az</ListenAddress>
>
>
>
>
>
>