[jira] [Comment Edited] (CASSANDRA-12144) Undeletable rows after upgrading from 2.2.4 to 3.0.7

["Stanislav Vishnevskiy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364659#comment-15364659 ] 

Stanislav Vishnevskiy edited comment on CASSANDRA-12144 at 7/6/16 5:16 PM:
---------------------------------------------------------------------------

The writetime on the rows is in the GitHub gist.


{noformat}
cqlsh>  SELECT WRITETIME(since), WRITETIME(type) FROM discord_relationships.relationships WHERE  user_id = 116138050710536192 AND id = 153047019424972800;

 writetime(since) | writetime(type)
------------------+------------------
 1464619988173052 | 1464619988173052
{noformat}

So that was written on May 30th.

This cluster is half a year old and exhibited zero issues until yesterday pretty much right after the upgrade finished. We also are noticing a weird key cache hit rate right from when the upgrade finished.

http://i.imgur.com/JDihdGO.png

The schema is as follows.

{noformat}
CREATE KEYSPACE discord_relationships WITH replication = {'class': 'NetworkTopologyStrategy', 'us-east1': '3'}  AND durable_writes = true;

CREATE TABLE discord_relationships.relationships (
    user_id bigint,
    id bigint,
    since timestamp,
    type tinyint,
    PRIMARY KEY (user_id, id)
) WITH CLUSTERING ORDER BY (id ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';
{noformat}

Thanks.


was (Author: stanislav):
The writetime on the rows is in the GitHub


{noformat}
cqlsh>  SELECT WRITETIME(since), WRITETIME(type) FROM discord_relationships.relationships WHERE  user_id = 116138050710536192 AND id = 153047019424972800;

 writetime(since) | writetime(type)
------------------+------------------
 1464619988173052 | 1464619988173052
{noformat}

So that was written on May 30th.

This cluster is half a year old and exhibited zero issues until yesterday pretty much right after the upgrade finished. We also are noticing a weird key cache hit rate right from when the upgrade finished.

http://i.imgur.com/JDihdGO.png

The schema is as follows.

{noformat}
CREATE KEYSPACE discord_relationships WITH replication = {'class': 'NetworkTopologyStrategy', 'us-east1': '3'}  AND durable_writes = true;

CREATE TABLE discord_relationships.relationships (
    user_id bigint,
    id bigint,
    since timestamp,
    type tinyint,
    PRIMARY KEY (user_id, id)
) WITH CLUSTERING ORDER BY (id ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';
{noformat}

Thanks.

> Undeletable rows after upgrading from 2.2.4 to 3.0.7
> ----------------------------------------------------
>
>                 Key: CASSANDRA-12144
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12144
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Stanislav Vishnevskiy
>
> We upgraded our cluster today and now have a some rows that refuse to delete.
> Here are some example traces.
> https://gist.github.com/vishnevskiy/36aa18c468344ea22d14f9fb9b99171d
> Even weirder.
> Updating the row and querying it back results in 2 rows even though the id is the clustering key.
> {noformat}
> user_id            | id                 | since                    | type
> -------------------+--------------------+--------------------------+------
> 116138050710536192 | 153047019424972800 |                     null |    0
> 116138050710536192 | 153047019424972800 | 2016-05-30 14:53:08+0000 |    2
> {noformat}
> And then deleting it again only removes the new one.
> {noformat}
> cqlsh:discord_relationships> DELETE FROM relationships WHERE user_id = 116138050710536192 AND id = 153047019424972800;
> cqlsh:discord_relationships> SELECT * FROM relationships WHERE user_id = 116138050710536192 AND id = 153047019424972800;
>  user_id            | id                 | since                    | type
> --------------------+--------------------+--------------------------+------
>  116138050710536192 | 153047019424972800 | 2016-05-30 14:53:08+0000 |    2
> {noformat}
> We tried repairing, compacting, scrubbing. No Luck.
> Not sure what to do. Is anyone aware of this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)