You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sylvain Lebresne (JIRA)" <ji...@apache.org> on 2016/02/08 11:34:40 UTC
[jira] [Commented] (CASSANDRA-11044) query under certain partition
key takes much more time than expected
[ https://issues.apache.org/jira/browse/CASSANDRA-11044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136792#comment-15136792 ]
Sylvain Lebresne commented on CASSANDRA-11044:
----------------------------------------------
If you see {{gatherTombstones.getNext}} iterating 9000+ times, then it definitively sounds like the partition is very tombstone heavy and that might just be the source of the problem. It might be related to CASSANDRA-8546 but I'm not sure because that ticket is mostly about inefficiencies in reverse queries and it doesn't seem that it's what you're doing and again, lots of iteration inside {{gatherTombstones.getNext}} simply suggest a huge amount of tombstones which is enough to explain the slowness.
So the good question would be why you have a huge amount of tombstones in that partition. That could be due to the way your insert (you have a workload that simply insert too many tombstone), in which case you'd have to fix that. But one other issue that comes to mind is CASSANDRA-7953 which had the bad consequence of multiplying range tombstones, so that could be the underlying reason. Now you seem to be on 2.2.4 so CASSANDRA-7953 should be fixed, but there can still be lots of tombstones accumulated from before your upgrade that haven't been removed by compaction.
Overall, trying to compact with a low {{gc_grace_seconds}} is definitively something to try and if you can report on the result of that experiment that would be appreciated.
> query under certain partition key takes much more time than expected
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-11044
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11044
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Reporter: xiaost
> Attachments: tracing.log
>
>
> In my cluster, all the nodes is under low workload,
> but query under certain partition key (we found one) takes much more time than expected.
> we write & updates about 3 times per row in one day,
> reads are much more than writes.
> HARDWARD:
> 6*nodes(E5-2630, 1*ssd with 5GB data)
> TABLE DESCRIBE:
> {noformat}
> CREATE TABLE album.user_updates (
> user_id bigint,
> time_uuid bigint,
> key ascii,
> PRIMARY KEY (user_id, time_uuid)
> ) WITH CLUSTERING ORDER BY (time_uuid ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
> AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {noformat}
> QUERYs:
> {noformat}
> select * from user_updates where user_id = 1432138730701829 limit 100;
> select count(1) from user_updates where user_id = 1432138730701829;
> {noformat}
> RESULT: (takes about 3.5 minutes)
> {noformat}
> count
> -------
> 1058
> (1 rows)
> {noformat}
> check attachments for the tracing log
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)