You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Paul Chandler <pa...@redshots.com> on 2020/02/13 14:43:06 UTC
Corruption of frozen UDT during upgrade
Hi all,
I have looked at the release notes for the up coming release 3.11.6 and seen the part about corruption of frozen UDT types during upgrade from 3.0.
We have a number of cluster using UDT and have been upgrading to 3.11.4 and haven’t noticed any problems.
In the ticket ( CASSANDRA-15035 ) it does not seem to specify how to reproduce this problem, so I tried using the following definition:
CREATE TYPE supplier_holiday_udt (
holiday_type int,
holiday_start date,
holiday_end date
);
CREATE TABLE supplier (
supplier_id int PRIMARY KEY,
calendar frozen<supplier_holiday_udt>
)
I performed an upgrade from 3.0.15 to 3.11.4, including running a nodetool upgradesstables.
There were no errors during the process and I can still read the data in supplier table.
Can anyone tell me how I reproduce this problem, or check that the clusters we have already upgraded do not have any problems .
Thanks
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Corruption of frozen UDT during upgrade
Posted by Andrew Prudhomme <an...@gmail.com>.
What does
http://cassandra.apache.org/doc/latest/tools/sstable/sstablemetadata.html show
as the type in the upgraded table? Also, the ticket mentions serialization
ordering being incorrect between single and multi-cell column data.
Perhaps there needs to be other non-PK fields to trigger this condition?
On Fri, Feb 14, 2020 at 6:34 AM Paul Chandler <pa...@redshots.com> wrote:
> Erick,
>
> Thank you for your help.
>
> I am still having problems reproducing this, so I am wondering if I have
> created the tables correctly to create this issue.
>
> I have looked at the sstabledumps and they seem exactly the same.
>
> This is the pre upgrade version 3.0.14 ( a snapshotted version )
>
> sstabledump mc-1-big-Data.db
> [
> {
> "partition" : {
> "key" : [ "1" ],
> "position" : 0
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 48,
> "liveness_info" : { "tstamp" : "2020-02-13T10:48:53.856469Z" },
> "cells" : [
> { "name" : "calendar", "value" : {"holiday_type": 1,
> "holiday_start": "2020-01-01", "holiday_end": "2020-01-02"} }
> ]
> }
> ]
> }
> ]
>
> This is the post upgrade 3.1.4 version:
>
> sstabledump md-2-big-Data.db
> [
> {
> "partition" : {
> "key" : [ "1" ],
> "position" : 0
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 48,
> "liveness_info" : { "tstamp" : "2020-02-13T10:48:53.856469Z" },
> "cells" : [
> { "name" : "calendar", "value" : {"holiday_type": 1,
> "holiday_start": "2020-01-01", "holiday_end": "2020-01-02"} }
> ]
> }
> ]
> }
> ]
>
>
> I can see the sstable has been upgraded, as the name has changed
> from mc-1-big-Data.db to md-2-big-Data.db
>
> Thanks
>
> Paul
>
>
>
>
> On 14 Feb 2020, at 00:47, Erick Ramirez <er...@datastax.com>
> wrote:
>
> Paul, if you do a sstabledump in C* 3.0 (before upgrading) and compare it
> to the dump output after upgrading to C* 3.11 then you will see that the
> cell names in the outputs are different. This is the symptom of the broken
> serialization header which leads to various exceptions during compactions
> and reads.
>
> CASSANDRA-15035 <https://issues.apache.org/jira/browse/CASSANDRA-15035>
> has been fixed but is not yet included in a released version of C*
> (earmarked for C* 3.11.6, 4.0). The patched version of sstablescrub
> includes a new flag "-e" which rewrites the SSTable serialization headers
> to include the missing info for the frozen UDTs. See NEWS.txt
> <https://github.com/apache/cassandra/blob/trunk/NEWS.txt#L116-L133> for
> more details.
>
> If you want to run a verification test on your SSTables, you can follow
> this procedure as a workaround:
> - copy the SSTables to another server that's not part of any C* cluster
> - download the DSE 5.1 (equivalent to C* 3.11) tarball from
> https://downloads.datastax.com/enterprise/dse-5.1.17-bin.tar.gz
> - unpack the tarball (more details here
> <https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/install/installTARdse.html>
> )
> - run sstablescrub -e fix-only to just fix the headers without doing a
> normal scrub
>
> If the headers are fine, the scrub will be a no-op. Otherwise, it will
> report that new metadata files are being written. For more details, see
> https://support.datastax.com/hc/en-us/articles/360025955351. Cheers!
>
> Erick Ramirez | Developer Relations
> erick.ramirez@datastax.com | datastax.com <http://www.datastax.com/>
> <https://www.linkedin.com/company/datastax>
> <https://www.facebook.com/datastax> <https://twitter.com/datastax>
> <http://feeds.feedburner.com/datastax> <https://github.com/datastax/>
> <https://www.datastax.com/accelerate>
>
>
> On Fri, 14 Feb 2020 at 01:43, Paul Chandler <pa...@redshots.com> wrote:
>
>> Hi all,
>>
>> I have looked at the release notes for the up coming release 3.11.6 and
>> seen the part about corruption of frozen UDT types during upgrade from 3.0.
>>
>> We have a number of cluster using UDT and have been upgrading to 3.11.4
>> and haven’t noticed any problems.
>>
>> In the ticket ( CASSANDRA-15035 ) it does not seem to specify how to
>> reproduce this problem, so I tried using the following definition:
>>
>> CREATE TYPE supplier_holiday_udt (
>> holiday_type int,
>> holiday_start date,
>> holiday_end date
>> );
>>
>> CREATE TABLE supplier (
>> supplier_id int PRIMARY KEY,
>> calendar frozen<supplier_holiday_udt>
>> )
>>
>> I performed an upgrade from 3.0.15 to 3.11.4, including running a
>> nodetool upgradesstables.
>>
>> There were no errors during the process and I can still read the data in
>> supplier table.
>>
>> Can anyone tell me how I reproduce this problem, or check that the
>> clusters we have already upgraded do not have any problems .
>>
>> Thanks
>>
>> Paul
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>
Re: Corruption of frozen UDT during upgrade
Posted by Paul Chandler <pa...@redshots.com>.
Thanks Erick, looks like I have a bit of detective work to do on Monday, to work out which of my list of clusters started out as 2.* or DSE 4.* and whether they had UDT’s at that time.
> On 15 Feb 2020, at 00:50, Erick Ramirez <er...@datastax.com> wrote:
>
> I am still having problems reproducing this, so I am wondering if I have created the tables correctly to create this issue.
>
> Paul, I've since had clarification on the bug and I hope I can explain it correctly here (happy to be corrected if anyone else has insight on the issue). When you create a table with a frozen UDT in C* 2.1 or 2.2, the serialisation header does not get written correctly when you upgrade to C* 3.0. So if you want to replicate it, you'd need to create the table in C* 2.1/2.2.
>
> Consequently, you would have seen the notification from Michael Shuler a few minutes ago about C* 3.11.6 being released which contains the fix for CASSANDRA-15035 <https://github.com/apache/cassandra/blob/cassandra-3.11.6/CHANGES.txt#L2>. This means that you can use the C* 3.11.6 sstablescrub which contains the new header-fix flag for your testing. Cheers!
Re: Corruption of frozen UDT during upgrade
Posted by Erick Ramirez <er...@datastax.com>.
>
> I am still having problems reproducing this, so I am wondering if I have
> created the tables correctly to create this issue.
Paul, I've since had clarification on the bug and I hope I can explain it
correctly here (happy to be corrected if anyone else has insight on the
issue). When you create a table with a frozen UDT in C* 2.1 or 2.2, the
serialisation header does not get written correctly when you upgrade to C*
3.0. So if you want to replicate it, you'd need to create the table in C*
2.1/2.2.
Consequently, you would have seen the notification from Michael Shuler a
few minutes ago about C* 3.11.6 being released which contains the fix for
CASSANDRA-15035
<https://github.com/apache/cassandra/blob/cassandra-3.11.6/CHANGES.txt#L2>.
This means that you can use the C* 3.11.6 sstablescrub which contains the
new header-fix flag for your testing. Cheers!
Re: Corruption of frozen UDT during upgrade
Posted by Paul Chandler <pa...@redshots.com>.
Erick,
Thank you for your help.
I am still having problems reproducing this, so I am wondering if I have created the tables correctly to create this issue.
I have looked at the sstabledumps and they seem exactly the same.
This is the pre upgrade version 3.0.14 ( a snapshotted version )
sstabledump mc-1-big-Data.db
[
{
"partition" : {
"key" : [ "1" ],
"position" : 0
},
"rows" : [
{
"type" : "row",
"position" : 48,
"liveness_info" : { "tstamp" : "2020-02-13T10:48:53.856469Z" },
"cells" : [
{ "name" : "calendar", "value" : {"holiday_type": 1, "holiday_start": "2020-01-01", "holiday_end": "2020-01-02"} }
]
}
]
}
]
This is the post upgrade 3.1.4 version:
sstabledump md-2-big-Data.db
[
{
"partition" : {
"key" : [ "1" ],
"position" : 0
},
"rows" : [
{
"type" : "row",
"position" : 48,
"liveness_info" : { "tstamp" : "2020-02-13T10:48:53.856469Z" },
"cells" : [
{ "name" : "calendar", "value" : {"holiday_type": 1, "holiday_start": "2020-01-01", "holiday_end": "2020-01-02"} }
]
}
]
}
]
I can see the sstable has been upgraded, as the name has changed from mc-1-big-Data.db to md-2-big-Data.db
Thanks
Paul
> On 14 Feb 2020, at 00:47, Erick Ramirez <er...@datastax.com> wrote:
>
> Paul, if you do a sstabledump in C* 3.0 (before upgrading) and compare it to the dump output after upgrading to C* 3.11 then you will see that the cell names in the outputs are different. This is the symptom of the broken serialization header which leads to various exceptions during compactions and reads.
>
> CASSANDRA-15035 <https://issues.apache.org/jira/browse/CASSANDRA-15035> has been fixed but is not yet included in a released version of C* (earmarked for C* 3.11.6, 4.0). The patched version of sstablescrub includes a new flag "-e" which rewrites the SSTable serialization headers to include the missing info for the frozen UDTs. See NEWS.txt <https://github.com/apache/cassandra/blob/trunk/NEWS.txt#L116-L133> for more details.
>
> If you want to run a verification test on your SSTables, you can follow this procedure as a workaround:
> - copy the SSTables to another server that's not part of any C* cluster
> - download the DSE 5.1 (equivalent to C* 3.11) tarball from https://downloads.datastax.com/enterprise/dse-5.1.17-bin.tar.gz <https://downloads.datastax.com/enterprise/dse-5.1.17-bin.tar.gz>
> - unpack the tarball (more details here <https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/install/installTARdse.html>)
> - run sstablescrub -e fix-only to just fix the headers without doing a normal scrub
>
> If the headers are fine, the scrub will be a no-op. Otherwise, it will report that new metadata files are being written. For more details, see https://support.datastax.com/hc/en-us/articles/360025955351 <https://support.datastax.com/hc/en-us/articles/360025955351>. Cheers!
>
> Erick Ramirez | Developer Relations
> erick.ramirez@datastax.com <ma...@datastax.com> | datastax.com <http://www.datastax.com/> <https://www.linkedin.com/company/datastax> <https://www.facebook.com/datastax> <https://twitter.com/datastax> <http://feeds.feedburner.com/datastax> <https://github.com/datastax/>
> <https://www.datastax.com/accelerate>
>
>
> On Fri, 14 Feb 2020 at 01:43, Paul Chandler <paul@redshots.com <ma...@redshots.com>> wrote:
> Hi all,
>
> I have looked at the release notes for the up coming release 3.11.6 and seen the part about corruption of frozen UDT types during upgrade from 3.0.
>
> We have a number of cluster using UDT and have been upgrading to 3.11.4 and haven’t noticed any problems.
>
> In the ticket ( CASSANDRA-15035 ) it does not seem to specify how to reproduce this problem, so I tried using the following definition:
>
> CREATE TYPE supplier_holiday_udt (
> holiday_type int,
> holiday_start date,
> holiday_end date
> );
>
> CREATE TABLE supplier (
> supplier_id int PRIMARY KEY,
> calendar frozen<supplier_holiday_udt>
> )
>
> I performed an upgrade from 3.0.15 to 3.11.4, including running a nodetool upgradesstables.
>
> There were no errors during the process and I can still read the data in supplier table.
>
> Can anyone tell me how I reproduce this problem, or check that the clusters we have already upgraded do not have any problems .
>
> Thanks
>
> Paul
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
>
Re: Corruption of frozen UDT during upgrade
Posted by Erick Ramirez <er...@datastax.com>.
Paul, if you do a sstabledump in C* 3.0 (before upgrading) and compare it
to the dump output after upgrading to C* 3.11 then you will see that the
cell names in the outputs are different. This is the symptom of the broken
serialization header which leads to various exceptions during compactions
and reads.
CASSANDRA-15035 <https://issues.apache.org/jira/browse/CASSANDRA-15035> has
been fixed but is not yet included in a released version of C* (earmarked
for C* 3.11.6, 4.0). The patched version of sstablescrub includes a new
flag "-e" which rewrites the SSTable serialization headers to include the
missing info for the frozen UDTs. See NEWS.txt
<https://github.com/apache/cassandra/blob/trunk/NEWS.txt#L116-L133> for
more details.
If you want to run a verification test on your SSTables, you can follow
this procedure as a workaround:
- copy the SSTables to another server that's not part of any C* cluster
- download the DSE 5.1 (equivalent to C* 3.11) tarball from
https://downloads.datastax.com/enterprise/dse-5.1.17-bin.tar.gz
- unpack the tarball (more details here
<https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/install/installTARdse.html>
)
- run sstablescrub -e fix-only to just fix the headers without doing a
normal scrub
If the headers are fine, the scrub will be a no-op. Otherwise, it will
report that new metadata files are being written. For more details, see
https://support.datastax.com/hc/en-us/articles/360025955351. Cheers!
Erick Ramirez | Developer Relations
erick.ramirez@datastax.com | datastax.com <http://www.datastax.com>
<https://www.linkedin.com/company/datastax>
<https://www.facebook.com/datastax> <https://twitter.com/datastax>
<http://feeds.feedburner.com/datastax> <https://github.com/datastax/>
<https://www.datastax.com/accelerate>
On Fri, 14 Feb 2020 at 01:43, Paul Chandler <pa...@redshots.com> wrote:
> Hi all,
>
> I have looked at the release notes for the up coming release 3.11.6 and
> seen the part about corruption of frozen UDT types during upgrade from 3.0.
>
> We have a number of cluster using UDT and have been upgrading to 3.11.4
> and haven’t noticed any problems.
>
> In the ticket ( CASSANDRA-15035 ) it does not seem to specify how to
> reproduce this problem, so I tried using the following definition:
>
> CREATE TYPE supplier_holiday_udt (
> holiday_type int,
> holiday_start date,
> holiday_end date
> );
>
> CREATE TABLE supplier (
> supplier_id int PRIMARY KEY,
> calendar frozen<supplier_holiday_udt>
> )
>
> I performed an upgrade from 3.0.15 to 3.11.4, including running a nodetool
> upgradesstables.
>
> There were no errors during the process and I can still read the data in
> supplier table.
>
> Can anyone tell me how I reproduce this problem, or check that the
> clusters we have already upgraded do not have any problems .
>
> Thanks
>
> Paul
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>