You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Philippe <wa...@gmail.com> on 2013/04/30 18:15:03 UTC

SSTables not opened on new cluste

Hello,

I'm trying to bring up a copy of an existing 3-node cluster running 1.0.8
into a 3-node cluster running 1.0.11.
The new cluster has been configured to have the same tokens and the same
partitioner.

Initially, I copied the files in the data directory of each node into their
corresponding node on the new cluster. When starting the new cluster, it
didn't pick up the KS & CF.
So I remove the directories, created the schema, stopped cassandra and
copied the data back. None of the SSTables are opened at startup.

When I nodetool refresh them, it says "No new SSTables where found for
 XXX/YYY"
where X is my KS and Y is my CF

I'm totally stumped: any ideas as to why this is happening ? I've checked
that there are actual files there and that the permissions are correct.

Thanks.

Re: SSTables not opened on new cluste

Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, May 7, 2013 at 4:26 AM, Philippe <wa...@gmail.com> wrote :
> Definitely knew that for major releases, didn't expect it for a minor
> release at all.

This sort of incompatibility is definitely more common between major
versions, but not unheard of within minor series.

=Rob

Re: SSTables not opened on new cluste

Posted by Philippe <wa...@gmail.com>.
Definitely knew that for major releases, didn't expect it for a minor
release at all.
Le 6 mai 2013 19:22, "Robert Coli" <rc...@eventbrite.com> a écrit :

> On Sat, May 4, 2013 at 5:41 AM, Philippe <wa...@gmail.com> wrote:
> > After trying every possible combination of parameters, config and the
> rest,
> > I ended up downgrading the new node from 1.1.11 to 1.1.2 to match the
> > existing 3 nodes. And that solved the issue immediately : the schema was
> > propagated and the node started handling reads & writes.
>
> As you have discovered...
>
> Trying to upgrade Cassandra by :
>
> 1) Adding new node at new version
> 2) Upgrading old nodes
>
> Is far less likely to work than :
>
> 1) Add new node at old version
> 2) Upgrade all nodes
>
> =Rob
>

Re: SSTables not opened on new cluste

Posted by Robert Coli <rc...@eventbrite.com>.
On Sat, May 4, 2013 at 5:41 AM, Philippe <wa...@gmail.com> wrote:
> After trying every possible combination of parameters, config and the rest,
> I ended up downgrading the new node from 1.1.11 to 1.1.2 to match the
> existing 3 nodes. And that solved the issue immediately : the schema was
> propagated and the node started handling reads & writes.

As you have discovered...

Trying to upgrade Cassandra by :

1) Adding new node at new version
2) Upgrading old nodes

Is far less likely to work than :

1) Add new node at old version
2) Upgrade all nodes

=Rob

Re: SSTables not opened on new cluste

Posted by Philippe <wa...@gmail.com>.
After trying every possible combination of parameters, config and the rest,
I ended up downgrading the new node from 1.1.11 to 1.1.2 to match the
existing 3 nodes. And that solved the issue immediately : the schema was
propagated and the node started handling reads & writes.


2013/5/3 Philippe <wa...@gmail.com>

> Unfortunately not, I've moved on to trying to add the nodes the current
> cluster and then decommission the "old" ones.
>
> But even that is not working, this is the strangest of things : while
> trying to add a new node, I
>  - set its token to  an existing value+1
>  - ensure the yaml (clutser name, partitionner, etc) are the same
>  - verified I can connect to the :7000 ports across machines
>  - cleared the data and commitlog directory
>
> when I start the node, it goes through the bootstrap process but never
> "imports" the schema from the cluster (the data/ directory stays empty,
> permissions are correct) and I get errors when some reads come in after the
> bootstrap completes. I've tried restarting the node with -D replace.token
> The log is below. I've been at it all day so I've been adding the node &
> decommissioning it, assuming that would clear any state in the cluster.
>
>  INFO [main] 2013-05-03 21:10:22,126 StorageService.java (line 788)
> JOINING: waiting for ring information
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,421 Gossiper.java (line 838)
> InetAddress /172.16.0.42 is now dead.
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,427 StorageService.java (line
> 1303) Removing token Token(bytes[03abaa3000000000000001]) for /172.16.0.42
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,429 ColumnFamilyStore.java (line
> 674) Enqueuing flush of Memtable-LocationInfo@444462602(47/58
> serialized/live bytes, 2 ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:10:26,430 Memtable.java (line 264)
> Writing Memtable-LocationInfo@444462602(47/58 serialized/live bytes, 2
> ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:10:26,542 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db
> (160 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=769)
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,544 Gossiper.java (line 858)
> Node /{other_node_A} is now part of the cluster
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,545 Gossiper.java (line 824)
> InetAddress /{other_node_A} is now UP
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,547 ColumnFamilyStore.java (line
> 674) Enqueuing flush of Memtable-LocationInfo@1290155526(30/37
> serialized/live bytes, 1 ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:10:26,548 Memtable.java (line 264)
> Writing Memtable-LocationInfo@1290155526(30/37 serialized/live bytes, 1
> ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:10:26,662 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db
> (84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=862)
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,664 Gossiper.java (line 858)
> Node /{other_node_B} is now part of the cluster
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,665 Gossiper.java (line 824)
> InetAddress /{other_node_B} is now UP
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,666 ColumnFamilyStore.java (line
> 674) Enqueuing flush of Memtable-LocationInfo@899933215(30/37
> serialized/live bytes, 1 ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:10:26,667 Memtable.java (line 264)
> Writing Memtable-LocationInfo@899933215(30/37 serialized/live bytes, 1
> ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:10:26,779 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db
> (84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=955)
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,781 Gossiper.java (line 858)
> Node /{other_node_C} is now part of the cluster
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,782 Gossiper.java (line 824)
> InetAddress /{other_node_C} is now UP
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,784 ColumnFamilyStore.java (line
> 674) Enqueuing flush of Memtable-LocationInfo@1542631196(30/37
> serialized/live bytes, 1 ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:10:26,785 Memtable.java (line 264)
> Writing Memtable-LocationInfo@1542631196(30/37 serialized/live bytes, 1
> ops)
>
>  INFO [CompactionExecutor:4] 2013-05-03 21:10:26,789 CompactionTask.java
> (line 107) Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-1-Data.db')]
>
>  INFO [FlushWriter:1] 2013-05-03 21:10:26,939 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db
> (84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=1048)
>
>  INFO [GossipStage:1] 2013-05-03 21:10:26,942 Gossiper.java (line 838)
> InetAddress /172.16.0.39 is now dead.
>
>  INFO [CompactionExecutor:4] 2013-05-03 21:10:26,986 CompactionTask.java
> (line 232) Compacted to
> [/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db,].
> 566 to 458 (~80% of original) bytes for 4 keys at 0.002287MB/s.  Time:
> 191ms.
>
>  INFO [main] 2013-05-03 21:10:52,130 StorageService.java (line 788)
> JOINING: schema complete
>
>  INFO [main] 2013-05-03 21:10:52,130 StorageService.java (line 788)
> JOINING: waiting for pending range calculation
>
>  INFO [main] 2013-05-03 21:10:52,131 StorageService.java (line 788)
> JOINING: calculation complete, ready to bootstrap
>
>  INFO [main] 2013-05-03 21:10:52,131 StorageService.java (line 788)
> JOINING: getting bootstrap token
>
>  INFO [main] 2013-05-03 21:10:52,135 ColumnFamilyStore.java (line 674)
> Enqueuing flush of Memtable-LocationInfo@1692466348(31/38 serialized/live
> bytes, 1 ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:10:52,136 Memtable.java (line 264)
> Writing Memtable-LocationInfo@1692466348(31/38 serialized/live bytes, 1
> ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:10:52,252 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-7-Data.db
> (82 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=1139)
>
>  INFO [main] 2013-05-03 21:10:52,254 StorageService.java (line 788)
> JOINING: sleeping 30000 ms for pending range setup
>
>  INFO [main] 2013-05-03 21:11:22,255 StorageService.java (line 788)
> JOINING: Starting to bootstrap...
>
>  INFO [main] 2013-05-03 21:11:22,272 ColumnFamilyStore.java (line 674)
> Enqueuing flush of Memtable-LocationInfo@1458621244(48/60 serialized/live
> bytes, 2 ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:11:22,273 Memtable.java (line 264)
> Writing Memtable-LocationInfo@1458621244(48/60 serialized/live bytes, 2
> ops)
>
>  INFO [FlushWriter:1] 2013-05-03 21:11:22,386 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-8-Data.db
> (158 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
> position=1315)
>
>  INFO [CompactionExecutor:8] 2013-05-03 21:11:22,389 CompactionTask.java
> (line 107) Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-8-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-7-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db')]
>
>  INFO [main] 2013-05-03 21:11:22,396 StorageService.java (line 1133) Node /
> 5.135.140.99 state jump to normal
>
>  INFO [main] 2013-05-03 21:11:22,397 StorageService.java (line 701)
> Bootstrap/Replace/Move completed! Now serving reads.
>
>  INFO [main] 2013-05-03 21:11:22,415 CassandraDaemon.java (line 125)
> Binding thrift service to /0.0.0.0:9160
>
>  INFO [main] 2013-05-03 21:11:22,417 CassandraDaemon.java (line 134) Using
> TFastFramedTransport with a max frame size of 15728640 bytes.
>
>  INFO [main] 2013-05-03 21:11:22,419 CassandraDaemon.java (line 161) Using
> synchronous/threadpool thrift server on /0.0.0.0 : 9160
>
>  INFO [Thread-9] 2013-05-03 21:11:22,419 CassandraDaemon.java (line 213)
> Listening for thrift clients...
>
>  INFO [CompactionExecutor:8] 2013-05-03 21:11:22,542 CompactionTask.java
> (line 232) Compacted to
> [/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-9-Data.db,].
> 782 to 489 (~62% of original) bytes for 4 keys at 0.003088MB/s.  Time:
> 151ms.
>
>

Re: SSTables not opened on new cluste

Posted by Philippe <wa...@gmail.com>.
Unfortunately not, I've moved on to trying to add the nodes the current
cluster and then decommission the "old" ones.

But even that is not working, this is the strangest of things : while
trying to add a new node, I
 - set its token to  an existing value+1
 - ensure the yaml (clutser name, partitionner, etc) are the same
 - verified I can connect to the :7000 ports across machines
 - cleared the data and commitlog directory

when I start the node, it goes through the bootstrap process but never
"imports" the schema from the cluster (the data/ directory stays empty,
permissions are correct) and I get errors when some reads come in after the
bootstrap completes. I've tried restarting the node with -D replace.token
The log is below. I've been at it all day so I've been adding the node &
decommissioning it, assuming that would clear any state in the cluster.

 INFO [main] 2013-05-03 21:10:22,126 StorageService.java (line 788)
JOINING: waiting for ring information

 INFO [GossipStage:1] 2013-05-03 21:10:26,421 Gossiper.java (line 838)
InetAddress /172.16.0.42 is now dead.

 INFO [GossipStage:1] 2013-05-03 21:10:26,427 StorageService.java (line
1303) Removing token Token(bytes[03abaa3000000000000001]) for /172.16.0.42

 INFO [GossipStage:1] 2013-05-03 21:10:26,429 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@444462602(47/58
serialized/live bytes, 2 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,430 Memtable.java (line 264)
Writing Memtable-LocationInfo@444462602(47/58 serialized/live bytes, 2 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,542 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db
(160 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=769)

 INFO [GossipStage:1] 2013-05-03 21:10:26,544 Gossiper.java (line 858) Node
/{other_node_A} is now part of the cluster

 INFO [GossipStage:1] 2013-05-03 21:10:26,545 Gossiper.java (line 824)
InetAddress /{other_node_A} is now UP

 INFO [GossipStage:1] 2013-05-03 21:10:26,547 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@1290155526(30/37
serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,548 Memtable.java (line 264)
Writing Memtable-LocationInfo@1290155526(30/37 serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,662 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db
(84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=862)

 INFO [GossipStage:1] 2013-05-03 21:10:26,664 Gossiper.java (line 858) Node
/{other_node_B} is now part of the cluster

 INFO [GossipStage:1] 2013-05-03 21:10:26,665 Gossiper.java (line 824)
InetAddress /{other_node_B} is now UP

 INFO [GossipStage:1] 2013-05-03 21:10:26,666 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@899933215(30/37
serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,667 Memtable.java (line 264)
Writing Memtable-LocationInfo@899933215(30/37 serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,779 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db
(84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=955)

 INFO [GossipStage:1] 2013-05-03 21:10:26,781 Gossiper.java (line 858) Node
/{other_node_C} is now part of the cluster

 INFO [GossipStage:1] 2013-05-03 21:10:26,782 Gossiper.java (line 824)
InetAddress /{other_node_C} is now UP

 INFO [GossipStage:1] 2013-05-03 21:10:26,784 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@1542631196(30/37
serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,785 Memtable.java (line 264)
Writing Memtable-LocationInfo@1542631196(30/37 serialized/live bytes, 1 ops)

 INFO [CompactionExecutor:4] 2013-05-03 21:10:26,789 CompactionTask.java
(line 107) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-1-Data.db')]

 INFO [FlushWriter:1] 2013-05-03 21:10:26,939 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db
(84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=1048)

 INFO [GossipStage:1] 2013-05-03 21:10:26,942 Gossiper.java (line 838)
InetAddress /172.16.0.39 is now dead.

 INFO [CompactionExecutor:4] 2013-05-03 21:10:26,986 CompactionTask.java
(line 232) Compacted to
[/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db,].
566 to 458 (~80% of original) bytes for 4 keys at 0.002287MB/s.  Time:
191ms.

 INFO [main] 2013-05-03 21:10:52,130 StorageService.java (line 788)
JOINING: schema complete

 INFO [main] 2013-05-03 21:10:52,130 StorageService.java (line 788)
JOINING: waiting for pending range calculation

 INFO [main] 2013-05-03 21:10:52,131 StorageService.java (line 788)
JOINING: calculation complete, ready to bootstrap

 INFO [main] 2013-05-03 21:10:52,131 StorageService.java (line 788)
JOINING: getting bootstrap token

 INFO [main] 2013-05-03 21:10:52,135 ColumnFamilyStore.java (line 674)
Enqueuing flush of Memtable-LocationInfo@1692466348(31/38 serialized/live
bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:52,136 Memtable.java (line 264)
Writing Memtable-LocationInfo@1692466348(31/38 serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:52,252 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-7-Data.db
(82 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=1139)

 INFO [main] 2013-05-03 21:10:52,254 StorageService.java (line 788)
JOINING: sleeping 30000 ms for pending range setup

 INFO [main] 2013-05-03 21:11:22,255 StorageService.java (line 788)
JOINING: Starting to bootstrap...

 INFO [main] 2013-05-03 21:11:22,272 ColumnFamilyStore.java (line 674)
Enqueuing flush of Memtable-LocationInfo@1458621244(48/60 serialized/live
bytes, 2 ops)

 INFO [FlushWriter:1] 2013-05-03 21:11:22,273 Memtable.java (line 264)
Writing Memtable-LocationInfo@1458621244(48/60 serialized/live bytes, 2 ops)

 INFO [FlushWriter:1] 2013-05-03 21:11:22,386 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-8-Data.db
(158 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=1315)

 INFO [CompactionExecutor:8] 2013-05-03 21:11:22,389 CompactionTask.java
(line 107) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-8-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-7-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db')]

 INFO [main] 2013-05-03 21:11:22,396 StorageService.java (line 1133) Node /
5.135.140.99 state jump to normal

 INFO [main] 2013-05-03 21:11:22,397 StorageService.java (line 701)
Bootstrap/Replace/Move completed! Now serving reads.

 INFO [main] 2013-05-03 21:11:22,415 CassandraDaemon.java (line 125)
Binding thrift service to /0.0.0.0:9160

 INFO [main] 2013-05-03 21:11:22,417 CassandraDaemon.java (line 134) Using
TFastFramedTransport with a max frame size of 15728640 bytes.

 INFO [main] 2013-05-03 21:11:22,419 CassandraDaemon.java (line 161) Using
synchronous/threadpool thrift server on /0.0.0.0 : 9160

 INFO [Thread-9] 2013-05-03 21:11:22,419 CassandraDaemon.java (line 213)
Listening for thrift clients...

 INFO [CompactionExecutor:8] 2013-05-03 21:11:22,542 CompactionTask.java
(line 232) Compacted to
[/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-9-Data.db,].
782 to 489 (~62% of original) bytes for 4 keys at 0.003088MB/s.  Time:
151ms.

Re: SSTables not opened on new cluste

Posted by aaron morton <aa...@thelastpickle.com>.
> When I created the schema, the KS were created as directory in the .../data/ directory
Can you provide the logs from start up ? 


> I just noticed that when the new cluster nodes start up, auto_bootstrap is set to true. Given that it's no longer in the YAML file, I didn't set it to false; would that explain it ?
That will not cause any problems. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 1/05/2013, at 9:08 AM, Philippe <wa...@gmail.com> wrote:

> Hi Aaron,
> thanks for the response.
> 
> Permissions are correct : owner is cassandra (ubuntu) and permissions are drwxr-xr-x
> 
> When I created the schema, the KS were created as directory in the .../data/ directory
> When I use cassandra-cli, set the CL to QUORUM, ensure two instances are up (nodetool ring) and set a bogus column, I get an Unavailable exception which is weird.
> 
> I just noticed that when the new cluster nodes start up, auto_bootstrap is set to true. Given that it's no longer in the YAML file, I didn't set it to false; would that explain it ?
> 
> Once I removed it and set logging to TRACE, I get these relevant log lines
> 
> SliceQueryFilter.java (line 123) collecting 1 of 2147483647: XXX:5123@1367333147034
> a bunch of Schema.java (line 382) Adding org.apache.cassandra.config.CFMetaData@45f8acdc[cfId=1009,ksName=XXX,cfName=YYY...
> Removing compacted SSTable files from XXX
> No bootstrapping, leaving or moving nodes -> empty pending ranges for XXX (my KS)
> Table.java (line 317) Initializing XXX.YYY (KS and CF)
> 
> Any ideas ?
> 
> 
> 2013/4/30 aaron morton <aa...@thelastpickle.com>
> Double check the file permissions ?
> 
> Write some data (using cqlsh or cassandra-cli) and flush to make sure the new files are created where you expect them to be.
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 1/05/2013, at 4:15 AM, Philippe <wa...@gmail.com> wrote:
> 
> > Hello,
> >
> > I'm trying to bring up a copy of an existing 3-node cluster running 1.0.8 into a 3-node cluster running 1.0.11.
> > The new cluster has been configured to have the same tokens and the same partitioner.
> >
> > Initially, I copied the files in the data directory of each node into their corresponding node on the new cluster. When starting the new cluster, it didn't pick up the KS & CF.
> > So I remove the directories, created the schema, stopped cassandra and copied the data back. None of the SSTables are opened at startup.
> >
> > When I nodetool refresh them, it says "No new SSTables where found for  XXX/YYY" where X is my KS and Y is my CF
> >
> > I'm totally stumped: any ideas as to why this is happening ? I've checked that there are actual files there and that the permissions are correct.
> >
> > Thanks.
> >
> 
> 


Re: SSTables not opened on new cluste

Posted by Philippe <wa...@gmail.com>.
Hi Aaron,
thanks for the response.

Permissions are correct : owner is cassandra (ubuntu) and permissions
are drwxr-xr-x

When I created the schema, the KS were created as directory in the
.../data/ directory
When I use cassandra-cli, set the CL to QUORUM, ensure two instances are up
(nodetool ring) and set a bogus column, I get an Unavailable exception
which is weird.

I just noticed that when the new cluster nodes start up, auto_bootstrap is
set to true. Given that it's no longer in the YAML file, I didn't set it to
false; would that explain it ?

Once I removed it and set logging to TRACE, I get these relevant log lines

SliceQueryFilter.java (line 123) collecting 1 of 2147483647:
XXX:5123@1367333147034
a bunch of Schema.java (line 382) Adding
org.apache.cassandra.config.CFMetaData@45f8acdc
[cfId=1009,ksName=XXX,cfName=YYY...
Removing compacted SSTable files from XXX
No bootstrapping, leaving or moving nodes -> empty pending ranges for XXX
(my KS)
Table.java (line 317) Initializing XXX.YYY (KS and CF)

Any ideas ?


2013/4/30 aaron morton <aa...@thelastpickle.com>

> Double check the file permissions ?
>
> Write some data (using cqlsh or cassandra-cli) and flush to make sure the
> new files are created where you expect them to be.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 1/05/2013, at 4:15 AM, Philippe <wa...@gmail.com> wrote:
>
> > Hello,
> >
> > I'm trying to bring up a copy of an existing 3-node cluster running
> 1.0.8 into a 3-node cluster running 1.0.11.
> > The new cluster has been configured to have the same tokens and the same
> partitioner.
> >
> > Initially, I copied the files in the data directory of each node into
> their corresponding node on the new cluster. When starting the new cluster,
> it didn't pick up the KS & CF.
> > So I remove the directories, created the schema, stopped cassandra and
> copied the data back. None of the SSTables are opened at startup.
> >
> > When I nodetool refresh them, it says "No new SSTables where found for
>  XXX/YYY" where X is my KS and Y is my CF
> >
> > I'm totally stumped: any ideas as to why this is happening ? I've
> checked that there are actual files there and that the permissions are
> correct.
> >
> > Thanks.
> >
>
>

Re: SSTables not opened on new cluste

Posted by aaron morton <aa...@thelastpickle.com>.
Double check the file permissions ? 

Write some data (using cqlsh or cassandra-cli) and flush to make sure the new files are created where you expect them to be. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 1/05/2013, at 4:15 AM, Philippe <wa...@gmail.com> wrote:

> Hello,
> 
> I'm trying to bring up a copy of an existing 3-node cluster running 1.0.8 into a 3-node cluster running 1.0.11.
> The new cluster has been configured to have the same tokens and the same partitioner. 
> 
> Initially, I copied the files in the data directory of each node into their corresponding node on the new cluster. When starting the new cluster, it didn't pick up the KS & CF.
> So I remove the directories, created the schema, stopped cassandra and copied the data back. None of the SSTables are opened at startup.
> 
> When I nodetool refresh them, it says "No new SSTables where found for  XXX/YYY" where X is my KS and Y is my CF
> 
> I'm totally stumped: any ideas as to why this is happening ? I've checked that there are actual files there and that the permissions are correct.
> 
> Thanks.
>