You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Jay Zhuang <ja...@yahoo.com.INVALID> on 2017/05/30 21:22:53 UTC
Potential block issue for 3.0.13: schema version id mismatch while
upgrading
Hi,
While upgrading to 3.0.13 we found that the schema id is changed for the
same schema. Which could cause cassandra unable to start and other
issues related to UnknownColumnFamilyException. Ticket: CASSANDRA-13559
The problem is because the order of SchemaKeyspace tables is changed.
Then the digest for the same schema is also changed:
https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/schema/SchemaKeyspace.java#L311
I would suggest to have the older list back for digest calculation. But
it also means 3.0.13 -> 3.0.14 upgrade will have the same problem. Any
suggestion on that?
Thanks,
Jay
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Potential block issue for 3.0.13: schema version id mismatch while upgrading
Posted by "J. D. Jordan" <je...@gmail.com>.
I would guess there are way fewer people who have already upgraded than people who haven't. So better to fix this as fast as possible and cut a release than wait longer and not. At the very least we should pull 3.0.13 so even less people can/have upgrade to it.
> On May 31, 2017, at 7:45 AM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> The number of people isn't zero. Before we commit and cut a new release, we need to be able to explain what that upgrade path is.
>
> If this is just migration storms then it's not worth cutting a new release - 3.0 was doing that anyway until very recently. If it fails to start that's a much bigger problem, but I don't see why (without testing this myself first) why it would fail to start on the minor version upgrade. Can someone explain why we get unknown cfid just be changing the digest order?
>
>
> --
> Jeff Jirsa
>
>
>> On May 31, 2017, at 12:34 AM, Stefania Alborghetti <st...@datastax.com> wrote:
>>
>> The patch for 13559 is available if we want to release it in 3.0.14. The
>> tests are still running but I don't expect problems. We need a reviewer
>> however.
>>
>> Releasing the patch in 3.0.14 will cause problems for people that deployed
>> 3.0.13 and upgrade upwards, but the schema will agree once the rolling
>> upgrade is over. There shouldn't be too many people with 3.0.13 in
>> production however, since upgrading to 3.0.13 is broken in the first place.
>>
>> Our upgrade tests totally missed this, so they should be checked as to why
>> this is the case.
>>
>>
>>> On Wed, May 31, 2017 at 12:34 PM, kurt greaves <ku...@instaclustr.com> wrote:
>>>
>>> On 31 May 2017 at 03:41, Jeremiah D Jordan <je...@gmail.com>
>>> wrote:
>>>
>>>> If 3.0.13 causes schema mismatch on upgrade, then maybe we should pull
>>>> that and release 3.0.14 once 13559 is fixed. As that is a pretty bad
>>> place
>>>> to get into.
>>>
>>>
>>> i think that would be a good idea
>>>
>>
>>
>>
>> --
>>
>> <http://www.datastax.com/>
>>
>> STEFANIA ALBORGHETTI
>>
>> Software engineer | +852 6114 9265 | stefania.alborghetti@datastax.com
>>
>>
>> [image: http://www.datastax.com/cloud-applications]
>> <http://www.datastax.com/cloud-applications>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Potential block issue for 3.0.13: schema version id mismatch while upgrading
Posted by Jeff Jirsa <jj...@gmail.com>.
The number of people isn't zero. Before we commit and cut a new release, we need to be able to explain what that upgrade path is.
If this is just migration storms then it's not worth cutting a new release - 3.0 was doing that anyway until very recently. If it fails to start that's a much bigger problem, but I don't see why (without testing this myself first) why it would fail to start on the minor version upgrade. Can someone explain why we get unknown cfid just be changing the digest order?
--
Jeff Jirsa
> On May 31, 2017, at 12:34 AM, Stefania Alborghetti <st...@datastax.com> wrote:
>
> The patch for 13559 is available if we want to release it in 3.0.14. The
> tests are still running but I don't expect problems. We need a reviewer
> however.
>
> Releasing the patch in 3.0.14 will cause problems for people that deployed
> 3.0.13 and upgrade upwards, but the schema will agree once the rolling
> upgrade is over. There shouldn't be too many people with 3.0.13 in
> production however, since upgrading to 3.0.13 is broken in the first place.
>
> Our upgrade tests totally missed this, so they should be checked as to why
> this is the case.
>
>
>> On Wed, May 31, 2017 at 12:34 PM, kurt greaves <ku...@instaclustr.com> wrote:
>>
>> On 31 May 2017 at 03:41, Jeremiah D Jordan <je...@gmail.com>
>> wrote:
>>
>>> If 3.0.13 causes schema mismatch on upgrade, then maybe we should pull
>>> that and release 3.0.14 once 13559 is fixed. As that is a pretty bad
>> place
>>> to get into.
>>
>>
>> i think that would be a good idea
>>
>
>
>
> --
>
> <http://www.datastax.com/>
>
> STEFANIA ALBORGHETTI
>
> Software engineer | +852 6114 9265 | stefania.alborghetti@datastax.com
>
>
> [image: http://www.datastax.com/cloud-applications]
> <http://www.datastax.com/cloud-applications>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Potential block issue for 3.0.13: schema version id mismatch
while upgrading
Posted by Stefan Podkowinski <sp...@apache.org>.
On 31.05.2017 09:34, Stefania Alborghetti wrote:
> There shouldn't be too many people with 3.0.13 in
> production however, since upgrading to 3.0.13 is broken in the first place.
Keep in mind that there are always people upgrading from 2.x, especially
since we had a couple of important bug fixes for 2.x -> 3.0 that people
might have been waiting for before taking the plunge. We should at least
be so kind to let these users know about potential issues in NEWS.txt.
If I understand correctly, this will cause schema migration storms
across nodes for .13 -> .14. I'd rather see us pulling 3.0.13 from
downloads now and take some more time to figure out what can be done for
users already running 3.0.13 on production, instead of rushing .14 and
risk throwing .13 users under the bus.
E.g. can we create a nodetool command to disable execution of schema
migration tasks, which can be run during cluster upgrades in cases like
this?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Potential block issue for 3.0.13: schema version id mismatch
while upgrading
Posted by Stefania Alborghetti <st...@datastax.com>.
The patch for 13559 is available if we want to release it in 3.0.14. The
tests are still running but I don't expect problems. We need a reviewer
however.
Releasing the patch in 3.0.14 will cause problems for people that deployed
3.0.13 and upgrade upwards, but the schema will agree once the rolling
upgrade is over. There shouldn't be too many people with 3.0.13 in
production however, since upgrading to 3.0.13 is broken in the first place.
Our upgrade tests totally missed this, so they should be checked as to why
this is the case.
On Wed, May 31, 2017 at 12:34 PM, kurt greaves <ku...@instaclustr.com> wrote:
> On 31 May 2017 at 03:41, Jeremiah D Jordan <je...@gmail.com>
> wrote:
>
> > If 3.0.13 causes schema mismatch on upgrade, then maybe we should pull
> > that and release 3.0.14 once 13559 is fixed. As that is a pretty bad
> place
> > to get into.
>
>
> i think that would be a good idea
>
--
<http://www.datastax.com/>
STEFANIA ALBORGHETTI
Software engineer | +852 6114 9265 | stefania.alborghetti@datastax.com
[image: http://www.datastax.com/cloud-applications]
<http://www.datastax.com/cloud-applications>
Re: Potential block issue for 3.0.13: schema version id mismatch
while upgrading
Posted by kurt greaves <ku...@instaclustr.com>.
On 31 May 2017 at 03:41, Jeremiah D Jordan <je...@gmail.com>
wrote:
> If 3.0.13 causes schema mismatch on upgrade, then maybe we should pull
> that and release 3.0.14 once 13559 is fixed. As that is a pretty bad place
> to get into.
i think that would be a good idea
Re: Potential block issue for 3.0.13: schema version id mismatch
while upgrading
Posted by Jeremiah D Jordan <je...@gmail.com>.
If 3.0.13 causes schema mismatch on upgrade, then maybe we should pull that and release 3.0.14 once 13559 is fixed. As that is a pretty bad place to get into.
> On May 30, 2017, at 6:39 PM, Jay Zhuang <zj...@uber.com> wrote:
>
> Seems the mail is marked as spam. So try forwarding with another email
> account.
>
> Thanks,
> Jay
>
> ---------- Forwarded message ----------
> From: Jay Zhuang <ja...@yahoo.com.invalid>
> Date: Tue, May 30, 2017 at 2:22 PM
> Subject: Potential block issue for 3.0.13: schema version id mismatch while
> upgrading
> To: dev@cassandra.apache.org
>
>
> Hi,
>
> While upgrading to 3.0.13 we found that the schema id is changed for the
> same schema. Which could cause cassandra unable to start and other issues
> related to UnknownColumnFamilyException. Ticket: CASSANDRA-13559
>
> The problem is because the order of SchemaKeyspace tables is changed. Then
> the digest for the same schema is also changed:
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/
> java/org/apache/cassandra/schema/SchemaKeyspace.java#L311
>
> I would suggest to have the older list back for digest calculation. But it
> also means 3.0.13 -> 3.0.14 upgrade will have the same problem. Any
> suggestion on that?
>
> Thanks,
> Jay
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Fwd: Potential block issue for 3.0.13: schema version id mismatch
while upgrading
Posted by Jay Zhuang <zj...@uber.com>.
Seems the mail is marked as spam. So try forwarding with another email
account.
Thanks,
Jay
---------- Forwarded message ----------
From: Jay Zhuang <ja...@yahoo.com.invalid>
Date: Tue, May 30, 2017 at 2:22 PM
Subject: Potential block issue for 3.0.13: schema version id mismatch while
upgrading
To: dev@cassandra.apache.org
Hi,
While upgrading to 3.0.13 we found that the schema id is changed for the
same schema. Which could cause cassandra unable to start and other issues
related to UnknownColumnFamilyException. Ticket: CASSANDRA-13559
The problem is because the order of SchemaKeyspace tables is changed. Then
the digest for the same schema is also changed:
https://github.com/apache/cassandra/blob/cassandra-3.0/src/
java/org/apache/cassandra/schema/SchemaKeyspace.java#L311
I would suggest to have the older list back for digest calculation. But it
also means 3.0.13 -> 3.0.14 upgrade will have the same problem. Any
suggestion on that?
Thanks,
Jay
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org