You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Alain RODRIGUEZ <ar...@gmail.com> on 2013/02/08 14:03:40 UTC
High read latency cluster
Hi,
I have some big latencies (OpsCenter homepage shows an average about 30-60
ms), inducing instability in my front servers, stacking queries, waiting
for C* to answer, in the following 1.1.6 C* cluster:
10.208.45.173 eu-west 1b Up Normal 297.02 GB
100.00% 0
10.208.40.6 eu-west 1b Up Normal 292.91 GB
100.00% 56713727820156407428984779325531226112
10.208.47.135 eu-west 1b Up Normal 307.96 GB
100.00% 113427455640312814857969558651062452224
I run on 3 AWS m1.xLarge with mostly the Datastax AMI default node
configuration (But with the following
options MAX_HEAP_SIZE="8G" HEAP_NEWSIZE="400M", I was under regular memory
pressure with the default 4GB heap, maybe because of bloomfilters). RF = 3,
CL = QUORUM r/w.
I have a high load from 4 to 15 with an average of 8 (mainly because of
iowait which can reach up to 40-60%).
extract from "iostat -mx 5 10" :
avg-cpu: %user %nice %system %iowait %steal %idle
16.66 0.00 4.82 35.47 0.21 42.85
I use compression and Size Tiered Compaction Strategy for any of my CF.
A typical CF :
create column family active_product
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'UTF8Type'
and key_validation_class = 'UTF8Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 12
and replicate_on_write = true
and compaction_strategy =
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'KEYS_ONLY'
and bloom_filter_fp_chance = 0.01
and compression_options = {'sstable_compression' :
'org.apache.cassandra.io.compress.SnappyCompressor'};
And there is a typical counter CF:
create column family algo_product_view
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'CounterColumnType'
and key_validation_class = 'UTF8Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 12
and replicate_on_write = true
and compaction_strategy =
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'KEYS_ONLY'
and bloom_filter_fp_chance = 0.01
and compression_options = {'sstable_compression' :
'org.apache.cassandra.io.compress.SnappyCompressor'};
I attach my cfhistograms, proxyhistograms, cfstats and tpstats hopping a
clue is somewhere in there, even if I was unable to learn something there
by myself.
cfstats: http://pastebin.com/z3sAshjP
tpstats: http://pastebin.com/LETPqfLV
proxyhistograms: http://pastebin.com/FqwMFrxG
cfhistograms (from the 2 most read / highest latencies):
http://pastebin.com/BCsdc50z & http://pastebin.com/CGZZpydL
These latencies are quite annoying, hope you'll help me figuring out what I
am doing wrong or how I can tune Cassandra better.
Alain
Re: High read latency cluster
Posted by Edward Capriolo <ed...@gmail.com>.
300 GB is a lot of data for cloud machines (especially with their
weaker performance in general). If you are unhappy with performance
why not scale the cluster out to more servers, with that much data you
are usually contending with the physics of spinning disks. Three nodes
+ replication factor 3 also means all your data is replicated to every
node and your not getting any scale out benefits.
On Fri, Feb 8, 2013 at 8:03 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> Hi,
>
> I have some big latencies (OpsCenter homepage shows an average about 30-60
> ms), inducing instability in my front servers, stacking queries, waiting for
> C* to answer, in the following 1.1.6 C* cluster:
>
> 10.208.45.173 eu-west 1b Up Normal 297.02 GB
> 100.00% 0
> 10.208.40.6 eu-west 1b Up Normal 292.91 GB
> 100.00% 56713727820156407428984779325531226112
> 10.208.47.135 eu-west 1b Up Normal 307.96 GB
> 100.00% 113427455640312814857969558651062452224
>
> I run on 3 AWS m1.xLarge with mostly the Datastax AMI default node
> configuration (But with the following options MAX_HEAP_SIZE="8G"
> HEAP_NEWSIZE="400M", I was under regular memory pressure with the default
> 4GB heap, maybe because of bloomfilters). RF = 3, CL = QUORUM r/w.
>
> I have a high load from 4 to 15 with an average of 8 (mainly because of
> iowait which can reach up to 40-60%).
>
> extract from "iostat -mx 5 10" :
> avg-cpu: %user %nice %system %iowait %steal %idle
> 16.66 0.00 4.82 35.47 0.21 42.85
>
>
> I use compression and Size Tiered Compaction Strategy for any of my CF.
>
> A typical CF :
>
> create column family active_product
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'UTF8Type'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 12
> and replicate_on_write = true
> and compaction_strategy =
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and bloom_filter_fp_chance = 0.01
> and compression_options = {'sstable_compression' :
> 'org.apache.cassandra.io.compress.SnappyCompressor'};
>
> And there is a typical counter CF:
>
> create column family algo_product_view
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'CounterColumnType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 12
> and replicate_on_write = true
> and compaction_strategy =
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and bloom_filter_fp_chance = 0.01
> and compression_options = {'sstable_compression' :
> 'org.apache.cassandra.io.compress.SnappyCompressor'};
>
> I attach my cfhistograms, proxyhistograms, cfstats and tpstats hopping a
> clue is somewhere in there, even if I was unable to learn something there by
> myself.
>
> cfstats: http://pastebin.com/z3sAshjP
> tpstats: http://pastebin.com/LETPqfLV
> proxyhistograms: http://pastebin.com/FqwMFrxG
> cfhistograms (from the 2 most read / highest latencies):
> http://pastebin.com/BCsdc50z & http://pastebin.com/CGZZpydL
>
> These latencies are quite annoying, hope you'll help me figuring out what I
> am doing wrong or how I can tune Cassandra better.
>
> Alain
Re: High read latency cluster
Posted by Bill de hÓra <bi...@dehora.net>.
> FlushWriter 0 0 8252 0 299
If you are not suffering from gc pressure/pauses (possibly not, because you don't seem to have a lot of read failures in tpstats or outlier latency on the histograms), then the flush writer errors are suggestive of memtable pressure, which may be followed by compactions that grind the disk.
> maybe because of bloomfilters
If you think bloom filters or indexes are occupying heap on startup, then you could alleviate things for a while with memtable/cache tuning, resampling the index interval, or increasing the heap to 10G (yes, not generally recommended). Not enough working ram can also impact the key cache which then puts more pressure on disk - check info to see if your caches are being resized. If then disk I/O is simply be falling behind in conjunction with a server that doesn't have much memory headroom, then you'll want to expand the cluster at some point to spread out the load.
Bill
On 8 Feb 2013, at 13:03, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> Hi,
>
> I have some big latencies (OpsCenter homepage shows an average about 30-60 ms), inducing instability in my front servers, stacking queries, waiting for C* to answer, in the following 1.1.6 C* cluster:
>
> 10.208.45.173 eu-west 1b Up Normal 297.02 GB 100.00% 0
> 10.208.40.6 eu-west 1b Up Normal 292.91 GB 100.00% 56713727820156407428984779325531226112
> 10.208.47.135 eu-west 1b Up Normal 307.96 GB 100.00% 113427455640312814857969558651062452224
>
> I run on 3 AWS m1.xLarge with mostly the Datastax AMI default node configuration (But with the following options MAX_HEAP_SIZE="8G" HEAP_NEWSIZE="400M", I was under regular memory pressure with the default 4GB heap, maybe because of bloomfilters). RF = 3, CL = QUORUM r/w.
>
> I have a high load from 4 to 15 with an average of 8 (mainly because of iowait which can reach up to 40-60%).
>
> extract from "iostat -mx 5 10" :
> avg-cpu: %user %nice %system %iowait %steal %idle
> 16.66 0.00 4.82 35.47 0.21 42.85
>
>
> I use compression and Size Tiered Compaction Strategy for any of my CF.
>
> A typical CF :
>
> create column family active_product
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'UTF8Type'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 12
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and bloom_filter_fp_chance = 0.01
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
>
> And there is a typical counter CF:
>
> create column family algo_product_view
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'CounterColumnType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 12
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and bloom_filter_fp_chance = 0.01
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
>
> I attach my cfhistograms, proxyhistograms, cfstats and tpstats hopping a clue is somewhere in there, even if I was unable to learn something there by myself.
>
> cfstats: http://pastebin.com/z3sAshjP
> tpstats: http://pastebin.com/LETPqfLV
> proxyhistograms: http://pastebin.com/FqwMFrxG
> cfhistograms (from the 2 most read / highest latencies): http://pastebin.com/BCsdc50z & http://pastebin.com/CGZZpydL
>
> These latencies are quite annoying, hope you'll help me figuring out what I am doing wrong or how I can tune Cassandra better.
>
> Alain