You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sergey (JIRA)" <ji...@apache.org> on 2015/08/20 13:14:45 UTC
[jira] [Updated] (CASSANDRA-10137) Consistency problem
[ https://issues.apache.org/jira/browse/CASSANDRA-10137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sergey updated CASSANDRA-10137:
-------------------------------
Description:
I have 2 dc and 3 node:
dc1: 2 node;
dc2: 1 node;
Exist keyspace
KEYSPACE itm_dhcp_test WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '2', 'DC2': '1'} AND durable_writes = true;
and CF:
TABLE itm_dhcp_test.lock (
name text PRIMARY KEY,
reason text,
time timestamp,
who text
) WITH bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 'max_threshold': '32'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
Periodically there is a problem with deleting records. For example execute query:
INSERT INTO lock (name, reason, time, who) values ('unitTest4', 'CassandraClusterLockTest', dateof(now()), 'I') IF NOT EXISTS USING TTL 60
SELECT * FROM lock WHERE name='unitTest4'
DELETE FROM lock WHERE name='unitTest4'
SELECT * FROM lock WHERE name='unitTest4'
20% - 30% of cases last SELECT returns not empty record. Most often when coordinator node1-dc2.
In trace I see the message:
| Parsing DELETE FROM lock WHERE name='unitTest4' | node1.dc2 | 45 | SharedPool-Worker-3
| Preparing statement | node1.dc2 | 151 | SharedPool-Worker-3
| Executing single-partition query on users | node1.dc2 | 588 | SharedPool-Worker-1
| Acquiring sstable references | node1.dc2 | 601 | SharedPool-Worker-1
| Merging memtable tombstones | node1.dc2 | 634 | SharedPool-Worker-1
| Key cache hit for sstable 2 | node1.dc2| 668 | SharedPool-Worker-1
| Seeking to partition beginning in data file | node1.dc2 | 674 | SharedPool-Worker-1
| Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 737 | SharedPool-Worker-1
| Merging data from memtables and 1 sstables | node1.dc2 | 743 | SharedPool-Worker-1
| Read 1 live and 0 tombstoned cells | node1.dc2 | 795 | SharedPool-Worker-1
| Executing single-partition query on permissions | node1.dc2 | 1653 | SharedPool-Worker-1
| Acquiring sstable references | node1.dc2 | 1662 | SharedPool-Worker-1
| Merging memtable tombstones | node1.dc2 | 1690 | SharedPool-Worker-1
| Key cache hit for sstable 5 | node1.dc2| 1737 | SharedPool-Worker-1
| Seeking to partition indexed section in data file | node1.dc2 | 1742 | SharedPool-Worker-1
| Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 1797 | SharedPool-Worker-1
| Merging data from memtables and 1 sstables | node1.dc2 | 1805 | SharedPool-Worker-1
| Read 0 live and 0 tombstoned cells | node1.dc2 | 1819 | SharedPool-Worker-1
| Executing single-partition query on users | node1.dc2 | 2798 | SharedPool-Worker-4
| Acquiring sstable references | node1.dc2 | 2808 | SharedPool-Worker-4
| Merging memtable tombstones | node1.dc2 | 2851 | SharedPool-Worker-4
| Key cache hit for sstable 2 | node1.dc2| 2896 | SharedPool-Worker-4
| Seeking to partition beginning in data file | node1.dc2 | 2903 | SharedPool-Worker-4
| Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 2948 | SharedPool-Worker-4
| Merging data from memtables and 1 sstables | node1.dc2 | 2954 | SharedPool-Worker-4
| Read 1 live and 0 tombstoned cells | node1.dc2 | 3004 | SharedPool-Worker-4
| Executing single-partition query on permissions | node1.dc2 | 3807 | SharedPool-Worker-4
| Acquiring sstable references | node1.dc2 | 3820 | SharedPool-Worker-4
| Merging memtable tombstones | node1.dc2 | 3861 | SharedPool-Worker-4
| Key cache hit for sstable 5 | node1.dc2| 3927 | SharedPool-Worker-4
| Seeking to partition indexed section in data file | node1.dc2 | 3933 | SharedPool-Worker-4
|
Please help understand, why the record is not deleted?
was:
I have 2 dc and 3 node:
dc1: 2 node;
dc2: 1 node;
Exist keyspace
KEYSPACE itm_dhcp_test WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '2', 'DC2': '1'} AND durable_writes = true;
and CF:
TABLE itm_dhcp_test.lock (
name text PRIMARY KEY,
reason text,
time timestamp,
who text
) WITH bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 'max_threshold': '32'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
Periodically there is a problem with deleting records. For example execute query:
INSERT INTO lock (name, reason, time, who) values ('unitTest4', 'CassandraClusterLockTest', dateof(now()), 'I') IF NOT EXISTS USING TTL 60
SELECT * FROM lock WHERE name='unitTest4'
DELETE FROM lock WHERE name='unitTest4'
SELECT * FROM lock WHERE name='unitTest4'
20% - 30% of cases last SELECT returns not empty record. Most often when coordinator node1-dc2.
In trace I see the message:
Parsing DELETE FROM lock WHERE name='unitTest4' | node1.dc2 | 45 | SharedPool-Worker-3
| Preparing statement | node1.dc2 | 151 | SharedPool-Worker-3
| Executing single-partition query on users | node1.dc2 | 588 | SharedPool-Worker-1
| Acquiring sstable references | node1.dc2 | 601 | SharedPool-Worker-1
| Merging memtable tombstones | node1.dc2 | 634 | SharedPool-Worker-1
| Key cache hit for sstable 2 | node1.dc2| 668 | SharedPool-Worker-1
| Seeking to partition beginning in data file | node1.dc2 | 674 | SharedPool-Worker-1
| Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 737 | SharedPool-Worker-1
| Merging data from memtables and 1 sstables | node1.dc2 | 743 | SharedPool-Worker-1
| Read 1 live and 0 tombstoned cells | node1.dc2 | 795 | SharedPool-Worker-1
| Executing single-partition query on permissions | node1.dc2 | 1653 | SharedPool-Worker-1
| Acquiring sstable references | node1.dc2 | 1662 | SharedPool-Worker-1
| Merging memtable tombstones | node1.dc2 | 1690 | SharedPool-Worker-1
| Key cache hit for sstable 5 | node1.dc2| 1737 | SharedPool-Worker-1
| Seeking to partition indexed section in data file | node1.dc2 | 1742 | SharedPool-Worker-1
| Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 1797 | SharedPool-Worker-1
| Merging data from memtables and 1 sstables | node1.dc2 | 1805 | SharedPool-Worker-1
| Read 0 live and 0 tombstoned cells | node1.dc2 | 1819 | SharedPool-Worker-1
| Executing single-partition query on users | node1.dc2 | 2798 | SharedPool-Worker-4
| Acquiring sstable references | node1.dc2 | 2808 | SharedPool-Worker-4
| Merging memtable tombstones | node1.dc2 | 2851 | SharedPool-Worker-4
| Key cache hit for sstable 2 | node1.dc2| 2896 | SharedPool-Worker-4
| Seeking to partition beginning in data file | node1.dc2 | 2903 | SharedPool-Worker-4
| Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 2948 | SharedPool-Worker-4
| Merging data from memtables and 1 sstables | node1.dc2 | 2954 | SharedPool-Worker-4
| Read 1 live and 0 tombstoned cells | node1.dc2 | 3004 | SharedPool-Worker-4
| Executing single-partition query on permissions | node1.dc2 | 3807 | SharedPool-Worker-4
| Acquiring sstable references | node1.dc2 | 3820 | SharedPool-Worker-4
| Merging memtable tombstones | node1.dc2 | 3861 | SharedPool-Worker-4
| Key cache hit for sstable 5 | node1.dc2| 3927 | SharedPool-Worker-4
| Seeking to partition indexed section in data file | node1.dc2 | 3933 | SharedPool-Worker-4
Please help understand, why the record is not deleted?
> Consistency problem
> -------------------
>
> Key: CASSANDRA-10137
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10137
> Project: Cassandra
> Issue Type: Bug
> Reporter: Sergey
>
> I have 2 dc and 3 node:
> dc1: 2 node;
> dc2: 1 node;
> Exist keyspace
> KEYSPACE itm_dhcp_test WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '2', 'DC2': '1'} AND durable_writes = true;
> and CF:
> TABLE itm_dhcp_test.lock (
> name text PRIMARY KEY,
> reason text,
> time timestamp,
> who text
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 'max_threshold': '32'}
> AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> Periodically there is a problem with deleting records. For example execute query:
> INSERT INTO lock (name, reason, time, who) values ('unitTest4', 'CassandraClusterLockTest', dateof(now()), 'I') IF NOT EXISTS USING TTL 60
> SELECT * FROM lock WHERE name='unitTest4'
> DELETE FROM lock WHERE name='unitTest4'
> SELECT * FROM lock WHERE name='unitTest4'
> 20% - 30% of cases last SELECT returns not empty record. Most often when coordinator node1-dc2.
> In trace I see the message:
> | Parsing DELETE FROM lock WHERE name='unitTest4' | node1.dc2 | 45 | SharedPool-Worker-3
> | Preparing statement | node1.dc2 | 151 | SharedPool-Worker-3
> | Executing single-partition query on users | node1.dc2 | 588 | SharedPool-Worker-1
> | Acquiring sstable references | node1.dc2 | 601 | SharedPool-Worker-1
> | Merging memtable tombstones | node1.dc2 | 634 | SharedPool-Worker-1
> | Key cache hit for sstable 2 | node1.dc2| 668 | SharedPool-Worker-1
> | Seeking to partition beginning in data file | node1.dc2 | 674 | SharedPool-Worker-1
> | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 737 | SharedPool-Worker-1
> | Merging data from memtables and 1 sstables | node1.dc2 | 743 | SharedPool-Worker-1
> | Read 1 live and 0 tombstoned cells | node1.dc2 | 795 | SharedPool-Worker-1
> | Executing single-partition query on permissions | node1.dc2 | 1653 | SharedPool-Worker-1
> | Acquiring sstable references | node1.dc2 | 1662 | SharedPool-Worker-1
> | Merging memtable tombstones | node1.dc2 | 1690 | SharedPool-Worker-1
> | Key cache hit for sstable 5 | node1.dc2| 1737 | SharedPool-Worker-1
> | Seeking to partition indexed section in data file | node1.dc2 | 1742 | SharedPool-Worker-1
> | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 1797 | SharedPool-Worker-1
> | Merging data from memtables and 1 sstables | node1.dc2 | 1805 | SharedPool-Worker-1
> | Read 0 live and 0 tombstoned cells | node1.dc2 | 1819 | SharedPool-Worker-1
> | Executing single-partition query on users | node1.dc2 | 2798 | SharedPool-Worker-4
> | Acquiring sstable references | node1.dc2 | 2808 | SharedPool-Worker-4
> | Merging memtable tombstones | node1.dc2 | 2851 | SharedPool-Worker-4
> | Key cache hit for sstable 2 | node1.dc2| 2896 | SharedPool-Worker-4
> | Seeking to partition beginning in data file | node1.dc2 | 2903 | SharedPool-Worker-4
> | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 2948 | SharedPool-Worker-4
> | Merging data from memtables and 1 sstables | node1.dc2 | 2954 | SharedPool-Worker-4
> | Read 1 live and 0 tombstoned cells | node1.dc2 | 3004 | SharedPool-Worker-4
> | Executing single-partition query on permissions | node1.dc2 | 3807 | SharedPool-Worker-4
> | Acquiring sstable references | node1.dc2 | 3820 | SharedPool-Worker-4
> | Merging memtable tombstones | node1.dc2 | 3861 | SharedPool-Worker-4
> | Key cache hit for sstable 5 | node1.dc2| 3927 | SharedPool-Worker-4
> | Seeking to partition indexed section in data file | node1.dc2 | 3933 | SharedPool-Worker-4
> |
> Please help understand, why the record is not deleted?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)