You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by "olek.stasiak@gmail.com" <ol...@gmail.com> on 2014/03/11 13:30:44 UTC

Problems with adding datacenter and schema version disagreement

Hi All,
I've faced an issue with cassandra 2.0.5.
I've 6 node cluster with random partitioner, still using tokens
instead of vnodes.
Cause we're changing hardware we decide to migrate cluster to 6 new
machines and change partitioning options to vnode rather then
token-based.
I've followed instruction on site:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html
and started cassandra on 6 new nodes in new DC. Everything seems to
work correctly, nodes were seen from all others as up and normal.
Then i performed nodetool repair -pr on the first of new nodes.
But process falls into infinite loop, sending/receiving merkle trees
over and over. It hangs on one very small KS it there were no hope it
will stop sometime (process was running whole night).
So I decided to stop the repair and restart cass on this particular
new node. after restart 'Ive tried repair one more time with another
small KS, but it also falls into infinite loop.
So i decided to break the procedure of adding datacenter, remove nodes
from new DC and start all from scratch.
After running removenode on all new nodes I've wiped data dir and
start cassandra on new node once again. During the start messages
"org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=98bb99a2-42f2-3fcd-af67-208a4faae5fa"
appears in logs. Google said, that they may mean problems with schema
versions consistency, so I performed describe cluster in cassandra-cli
and i get:
Cluster Information:
Name: Metadata Cluster
Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
76198f8b-663f-3434-8860-251ebc6f50c4: [150.254.164.4]

f48d3512-e299-3508-a29d-0844a0293f3a: [150.254.164.3]

16ad2e35-1eef-32f0-995c-e2cbd4c18abf: [150.254.164.6]

72352017-9b0d-3b29-8c55-ed86f30363c5: [150.254.164.1]

7f1faa84-0821-3311-9232-9407500591cc: [150.254.164.5]

85cd0ebc-5d33-3bec-a682-8c5880ee2fa1: [150.254.164.2]

So now I have 6 diff schema version for cluster. But how it can
happened? How can I take my cluster to consistent state?
What did I wrong during extending cluster, so nodetool falls into infinite loop?
At the first sight data looks ok, I can read from cluster and I'm
getting expected output.
best regards
Aleksander

Re: Problems with adding datacenter and schema version disagreement

Posted by Umut Kocasaraç <uk...@gmail.com>.

We have upgraded our cassandra version to 2.0.7 and problem has been solved.

Thanks Russ,


On Wed, Apr 16, 2014 at 6:46 PM, Russell Hatch <rh...@datastax.com> wrote:

> I think you might be seeing the issue reported in
> https://issues.apache.org/jira/browse/CASSANDRA-6971
>
> If that's the case, it looks like a fix will be in 2.0.7
>
> Thanks,
>
> Russ
>
>
> On Tue, Apr 15, 2014 at 11:48 PM, Umut Kocasaraç <uk...@gmail.com>wrote:
>
>> Hi Olek,
>>
>> Could you solve the problem. Because we are experiencing exactly same
>> issue. We have 4 nodes and all of them are in different schema. Our
>> cassandra version is 2.0.6.
>>
>> Umut
>>
>>
>> On Fri, Mar 21, 2014 at 12:26 AM, Robert Coli <rc...@eventbrite.com>wrote:
>>
>>> On Thu, Mar 20, 2014 at 2:23 PM, olek.stasiak@gmail.com <
>>> olek.stasiak@gmail.com> wrote:
>>>
>>>> Bump one more time, could anybody help me?
>>>>
>>>
>>> Unfortunately, while I can advise you how to resolve "my cluster has
>>> multiple schema versions", I have no assistance to offer for "my cluster
>>> ends up with split schema versions when I do normal operations."
>>>
>>> I suggest opening up a JIRA in the apache cassandra JIRA describing what
>>> you did and how the result differed from what you expected.
>>>
>>> =Rob
>>>
>>>
>>
>>
>

Re: Problems with adding datacenter and schema version disagreement

Posted by Russell Hatch <rh...@datastax.com>.

I think you might be seeing the issue reported in
https://issues.apache.org/jira/browse/CASSANDRA-6971

If that's the case, it looks like a fix will be in 2.0.7

Thanks,

Russ


On Tue, Apr 15, 2014 at 11:48 PM, Umut Kocasaraç <uk...@gmail.com>wrote:

> Hi Olek,
>
> Could you solve the problem. Because we are experiencing exactly same
> issue. We have 4 nodes and all of them are in different schema. Our
> cassandra version is 2.0.6.
>
> Umut
>
>
> On Fri, Mar 21, 2014 at 12:26 AM, Robert Coli <rc...@eventbrite.com>wrote:
>
>> On Thu, Mar 20, 2014 at 2:23 PM, olek.stasiak@gmail.com <
>> olek.stasiak@gmail.com> wrote:
>>
>>> Bump one more time, could anybody help me?
>>>
>>
>> Unfortunately, while I can advise you how to resolve "my cluster has
>> multiple schema versions", I have no assistance to offer for "my cluster
>> ends up with split schema versions when I do normal operations."
>>
>> I suggest opening up a JIRA in the apache cassandra JIRA describing what
>> you did and how the result differed from what you expected.
>>
>> =Rob
>>
>>
>
>

Re: Problems with adding datacenter and schema version disagreement

Posted by Umut Kocasaraç <uk...@gmail.com>.

Hi Olek,

Could you solve the problem. Because we are experiencing exactly same
issue. We have 4 nodes and all of them are in different schema. Our
cassandra version is 2.0.6.

Umut

On Fri, Mar 21, 2014 at 12:26 AM, Robert Coli <rc...@eventbrite.com> wrote:

> On Thu, Mar 20, 2014 at 2:23 PM, olek.stasiak@gmail.com <
> olek.stasiak@gmail.com> wrote:
>
>> Bump one more time, could anybody help me?
>>
>
> Unfortunately, while I can advise you how to resolve "my cluster has
> multiple schema versions", I have no assistance to offer for "my cluster
> ends up with split schema versions when I do normal operations."
>
> I suggest opening up a JIRA in the apache cassandra JIRA describing what
> you did and how the result differed from what you expected.
>
> =Rob
>
>