You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Alexander Shutyaev <sh...@gmail.com> on 2013/12/05 13:33:50 UTC
cassandra performance problems
Hi all,
We have a 3 node cluster setup, single keyspace, about 500 tables. The
hardware is 2 cores + 16 GB RAM (Cassandra chose to have 4GB). Cassandra
version is 2.0.3. Our replication factor is 3, read/write consistency is
QUORUM. We've plugged it into our production environment as a cache in
front of postgres. Everything worked fine, we even stressed it by
explicitly propagating about 30G (10G/node) data from postgres to cassandra.
Then the problems came. Our nodes began showing high cpu usage (around 20).
The funny thing is that they were actually doing it one after another and
there was always only node with high cpu usage. Using OpsCenter we saw that
when the CPU was beginning to go high the node in question was performing
compaction. But even after the compaction was performed the cpu remained
still high, and in some cases didn't go down for hours. Our jmx monitoring
showed that it was presumably in constant garbage collection. During that
time cluster read latency goes from 2ms to 200ms
What can be the reason? Can it be high number of tables? Do we need to
adjust some settings for this setup? Is it ok to have so many tables?
Theoretically we can stuck them all in 3-4 tables.
Thanks in advance,
Alexander
Re: cassandra performance problems
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
Do you mean high CPU usage or high load avg? (20 indicates load avg to
me). High load avg means the CPU is waiting on something.
Check "iostat -dmx 1 100" to check your disk stats, you'll see the columns
that indicate mb/s read & write as well as % utilization.
Once you understand the bottleneck we can start to narrow down the cause.
On Thu, Dec 5, 2013 at 4:33 AM, Alexander Shutyaev <sh...@gmail.com>wrote:
> Hi all,
>
> We have a 3 node cluster setup, single keyspace, about 500 tables. The
> hardware is 2 cores + 16 GB RAM (Cassandra chose to have 4GB). Cassandra
> version is 2.0.3. Our replication factor is 3, read/write consistency is
> QUORUM. We've plugged it into our production environment as a cache in
> front of postgres. Everything worked fine, we even stressed it by
> explicitly propagating about 30G (10G/node) data from postgres to cassandra.
>
> Then the problems came. Our nodes began showing high cpu usage (around
> 20). The funny thing is that they were actually doing it one after another
> and there was always only node with high cpu usage. Using OpsCenter we saw
> that when the CPU was beginning to go high the node in question was
> performing compaction. But even after the compaction was performed the cpu
> remained still high, and in some cases didn't go down for hours. Our jmx
> monitoring showed that it was presumably in constant garbage collection.
> During that time cluster read latency goes from 2ms to 200ms
>
> What can be the reason? Can it be high number of tables? Do we need to
> adjust some settings for this setup? Is it ok to have so many tables?
> Theoretically we can stuck them all in 3-4 tables.
>
> Thanks in advance,
> Alexander
>
--
Jon Haddad
http://www.rustyrazorblade.com
skype: rustyrazorblade
Re: cassandra performance problems
Posted by Alexander Shutyaev <sh...@gmail.com>.
Thanks for your answers,
Jonathan, yes it was load avg and iowait was lower than 2% all that time -
the only load was the user one.
Robert, we had -Xmx4012m which was automatically calculated by the default
cassandra-env.sh (1/4 of total memory - 16G) - we didn't change that.
2013/12/5 Robert Coli <rc...@eventbrite.com>
> On Thu, Dec 5, 2013 at 4:33 AM, Alexander Shutyaev <sh...@gmail.com>wrote:
>
>> Cassandra version is 2.0.3. ... We've plugged it into our production
>> environment as a cache in front of postgres.
>>
>
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>
>
>> What can be the reason? Can it be high number of tables? Do we need to
>> adjust some settings for this setup? Is it ok to have so many tables?
>> Theoretically we can stuck them all in 3-4 tables.
>>
>
> 500 column families ("tables") is "a lot" but not necessarily "too many."
>
> During that time cluster read latency goes from 2ms to 200ms
>>
>
> In perfectly normal operation, things operating in the JVM may exhibit a
> few hundreds of milliseconds of latency, when the JVM pauses for garbage
> collection. If you absolutely require consistent latency below this time
> for all requests, something running in the JVM may not be for you.
>
> If compaction triggers "runaway GC" then you probably have too much steady
> state heap consumption. You don't mention anything about the hardware
> environment, how much heap is available, etc.. ?
>
> =Rob
>
Re: cassandra performance problems
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Dec 5, 2013 at 4:33 AM, Alexander Shutyaev <sh...@gmail.com>wrote:
> Cassandra version is 2.0.3. ... We've plugged it into our production
> environment as a cache in front of postgres.
>
https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
> What can be the reason? Can it be high number of tables? Do we need to
> adjust some settings for this setup? Is it ok to have so many tables?
> Theoretically we can stuck them all in 3-4 tables.
>
500 column families ("tables") is "a lot" but not necessarily "too many."
During that time cluster read latency goes from 2ms to 200ms
>
In perfectly normal operation, things operating in the JVM may exhibit a
few hundreds of milliseconds of latency, when the JVM pauses for garbage
collection. If you absolutely require consistent latency below this time
for all requests, something running in the JVM may not be for you.
If compaction triggers "runaway GC" then you probably have too much steady
state heap consumption. You don't mention anything about the hardware
environment, how much heap is available, etc.. ?
=Rob
Re: cassandra performance problems
Posted by Alexander Shutyaev <sh...@gmail.com>.
Thanks all for your responses. We've downgraded from 2.0.3 to 2.0.0 and
everything became normal.
2013/12/8 Nate McCall <na...@thelastpickle.com>
> If you are really set on using Cassandra as a cache, I would recommend
> disabling durable writes for the keyspace(s)[0]. This will bypass the
> commitlog (the flushing/rotation of which my be a good-sized portion of
> your performance problems given the number of tables).
>
> [0]
> http://www.datastax.com/documentation/cql/3.0/webhelp/index.html#cql/cql_reference/alter_keyspace_r.html
>
>
> On Fri, Dec 6, 2013 at 12:42 PM, J. Ryan Earl <os...@jryanearl.us> wrote:
>
>> On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev <sh...@gmail.com>wrote:
>>
>>> We've plugged it into our production environment as a cache in front of
>>> postgres. Everything worked fine, we even stressed it by explicitly
>>> propagating about 30G (10G/node) data from postgres to cassandra.
>>>
>>
>> If you just want a caching layer, why wouldn't you use Memcached or Redis
>> instead? Cassandra is designed to be a persist store and not so much
>> designed as caching layer. If you were replacing your use of Postgres
>> completely, that would be appropriate.
>>
>
>
>
> --
> -----------------
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
Re: cassandra performance problems
Posted by Nate McCall <na...@thelastpickle.com>.
If you are really set on using Cassandra as a cache, I would recommend
disabling durable writes for the keyspace(s)[0]. This will bypass the
commitlog (the flushing/rotation of which my be a good-sized portion of
your performance problems given the number of tables).
[0]
http://www.datastax.com/documentation/cql/3.0/webhelp/index.html#cql/cql_reference/alter_keyspace_r.html
On Fri, Dec 6, 2013 at 12:42 PM, J. Ryan Earl <os...@jryanearl.us> wrote:
> On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev <sh...@gmail.com>wrote:
>
>> We've plugged it into our production environment as a cache in front of
>> postgres. Everything worked fine, we even stressed it by explicitly
>> propagating about 30G (10G/node) data from postgres to cassandra.
>>
>
> If you just want a caching layer, why wouldn't you use Memcached or Redis
> instead? Cassandra is designed to be a persist store and not so much
> designed as caching layer. If you were replacing your use of Postgres
> completely, that would be appropriate.
>
--
-----------------
Nate McCall
Austin, TX
@zznate
Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
Re: cassandra performance problems
Posted by "J. Ryan Earl" <os...@jryanearl.us>.
On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev <sh...@gmail.com>wrote:
> We've plugged it into our production environment as a cache in front of
> postgres. Everything worked fine, we even stressed it by explicitly
> propagating about 30G (10G/node) data from postgres to cassandra.
>
If you just want a caching layer, why wouldn't you use Memcached or Redis
instead? Cassandra is designed to be a persist store and not so much
designed as caching layer. If you were replacing your use of Postgres
completely, that would be appropriate.