You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Yiming Sun <yi...@gmail.com> on 2012/05/16 21:59:36 UTC
how can we get (a lot) more performance from cassandra
Hello,
I asked the question as a follow-up under a different thread, so I figure I
should ask here instead in case the other one gets buried, and besides, I
have a little more information.
"We find the lack of performance disturbing" as we are only able to get
about 3-4MB/sec read performance out of Cassandra.
We are using cassandra as the backend for an IR repository of digital
texts. It is a read-mostly repository with occasional writes. Each row
represents a book volume, and each column of a row represents a page of the
volume. Granted the data size is small -- the average size of a column
text is 2-3KB, and each row has about 250 columns (varies quite a bit from
one volume to another).
Currently we are running a 3-node cluster, and will soon be upgraded to a
6-node setup. Each node is a VM with 4 cores and 16GB of memory. All VMs
use SAN as disk storage.
To retrieve a volume, a slice query is used via Hector that specifies the
row key (the volume), and a list of column keys (pages), and the
consistency level is set to ONE. It is typical to retrieve multiple
volumes per request.
The read rate that I have been seeing is about 3-4 MB/sec, and that is
reading the raw bytes... using string serializer the rate is even lower,
about 2.2MB/sec.
The server log shows the GC ParNew frequently gets longer than 200ms, often
in the range of 4-5seconds. But nowhere near 15 seconds (which is an
indication that JVM heap is being swapped out).
Currently we have not added JNA. From a blog post, it seems JNA is able to
increase the performance by 13%, and we are hoping to increase the
performance by something more like 1300% (3-4 MB/sec is just disturbingly
low). And we are hesitant to disable swap entirely since one of the nodes
is running a couple other services
Do you have any suggestions on how we may boost the performance? Thanks!
-- Y.
Re: how can we get (a lot) more performance from cassandra
Posted by Yiming Sun <yi...@gmail.com>.
Ah, never thought I would be quoting Luke's "No, that's not true... that's
impossible~~" here... sigh.
But seriously, thanks Mike. Instead of using memcached, would it help to
turn on row cache?
An even more philosophical question: what would be a better choice for
read-heavy loads? a major part of the reason we went for Cassandra was to
not deal with millions of small files, but we completely overlooked the
assumed workload. Thanks!
-- Y.
On Wed, May 16, 2012 at 4:07 PM, Mike Peters <cassandra@softwareprojects.com
> wrote:
> Hi Yiming,
>
> Cassandra is optimized for write-heavy environments.
>
> If you have a read-heavy application, you shouldn't be running your reads
> through Cassandra.
>
> On the bright side - Cassandra read throughput will remain consistent,
> regardless of your volume. But you are going to have to "wrap" your reads
> with memcache (or redis), so that the bulk of your reads can be served from
> memory.
>
>
> Thanks,
> Mike Peters
>
>
> On 5/16/2012 3:59 PM, Yiming Sun wrote:
>
> Hello,
>
> I asked the question as a follow-up under a different thread, so I
> figure I should ask here instead in case the other one gets buried, and
> besides, I have a little more information.
>
> "We find the lack of performance disturbing" as we are only able to get
> about 3-4MB/sec read performance out of Cassandra.
>
> We are using cassandra as the backend for an IR repository of digital
> texts. It is a read-mostly repository with occasional writes. Each row
> represents a book volume, and each column of a row represents a page of the
> volume. Granted the data size is small -- the average size of a column
> text is 2-3KB, and each row has about 250 columns (varies quite a bit from
> one volume to another).
>
> Currently we are running a 3-node cluster, and will soon be upgraded to
> a 6-node setup. Each node is a VM with 4 cores and 16GB of memory. All
> VMs use SAN as disk storage.
>
> To retrieve a volume, a slice query is used via Hector that specifies
> the row key (the volume), and a list of column keys (pages), and the
> consistency level is set to ONE. It is typical to retrieve multiple
> volumes per request.
>
> The read rate that I have been seeing is about 3-4 MB/sec, and that is
> reading the raw bytes... using string serializer the rate is even lower,
> about 2.2MB/sec.
>
> The server log shows the GC ParNew frequently gets longer than 200ms,
> often in the range of 4-5seconds. But nowhere near 15 seconds (which is an
> indication that JVM heap is being swapped out).
>
> Currently we have not added JNA. From a blog post, it seems JNA is able
> to increase the performance by 13%, and we are hoping to increase the
> performance by something more like 1300% (3-4 MB/sec is just disturbingly
> low). And we are hesitant to disable swap entirely since one of the nodes
> is running a couple other services
>
> Do you have any suggestions on how we may boost the performance? Thanks!
>
> -- Y.
>
>
>
>
Re: how can we get (a lot) more performance from cassandra
Posted by Yiming Sun <yi...@gmail.com>.
Will do, Oleg. Again, thanks for the information.
-- Y.
On Wed, May 16, 2012 at 4:44 PM, Oleg Dulin <ol...@gmail.com> wrote:
> **
>
> Please do keep us posted. We have a somewhat similar Cassandra utilization
> pattern, and I would like to know what your solution is...
>
>
>
> On 2012-05-16 20:38:37 +0000, Yiming Sun said:
>
>
> Thanks Oleg. Another caveat from our side is, we have a very large data
> space (imaging picking 100 items out of 3 million, the chance of having 2
> items from the same bin is pretty low). We will experiment with row cache,
> and hopefully it will help, not the opposite (the tuning guide says row
> cache could be detrimental in some circumstances).
>
>
> -- Y.
>
>
> On Wed, May 16, 2012 at 4:25 PM, Oleg Dulin <ol...@gmail.com> wrote:
>
> Indeed. This is how we are trying to solve this problem.
>
>
> Our application has a built-in cache that resembles a supercolumn or
> standardcolumn data structure and has API that resembles a combination of
> Pelops selector and mutator. You can do something like that for Hector.
>
>
> The cache is constrained and uses LRU to purge unused items and keep
> memory usage steady.
>
>
> It is not perfect and we have bugs still but it cuts down on 90% of
> cassandra reads.
>
>
>
> On 2012-05-16 20:07:11 +0000, Mike Peters said:
>
>
> Hi Yiming,
>
>
> Cassandra is optimized for write-heavy environments.
>
>
> If you have a read-heavy application, you shouldn't be running your reads
> through Cassandra.
>
>
> On the bright side - Cassandra read throughput will remain consistent,
> regardless of your volume. But you are going to have to "wrap" your reads
> with memcache (or redis), so that the bulk of your reads can be served from
> memory.
>
>
>
> Thanks,
>
> Mike Peters
>
>
> On 5/16/2012 3:59 PM, Yiming Sun wrote:
>
> Hello,
>
>
> I asked the question as a follow-up under a different thread, so I figure
> I should ask here instead in case the other one gets buried, and besides, I
> have a little more information.
>
>
> "We find the lack of performance disturbing" as we are only able to get
> about 3-4MB/sec read performance out of Cassandra.
>
>
> We are using cassandra as the backend for an IR repository of digital
> texts. It is a read-mostly repository with occasional writes. Each row
> represents a book volume, and each column of a row represents a page of the
> volume. Granted the data size is small -- the average size of a column
> text is 2-3KB, and each row has about 250 columns (varies quite a bit from
> one volume to another).
>
>
> Currently we are running a 3-node cluster, and will soon be upgraded to a
> 6-node setup. Each node is a VM with 4 cores and 16GB of memory. All VMs
> use SAN as disk storage.
>
>
> To retrieve a volume, a slice query is used via Hector that specifies the
> row key (the volume), and a list of column keys (pages), and the
> consistency level is set to ONE. It is typical to retrieve multiple
> volumes per request.
>
>
> The read rate that I have been seeing is about 3-4 MB/sec, and that is
> reading the raw bytes... using string serializer the rate is even lower,
> about 2.2MB/sec.
>
>
> The server log shows the GC ParNew frequently gets longer than 200ms,
> often in the range of 4-5seconds. But nowhere near 15 seconds (which is an
> indication that JVM heap is being swapped out).
>
>
> Currently we have not added JNA. From a blog post, it seems JNA is able
> to increase the performance by 13%, and we are hoping to increase the
> performance by something more like 1300% (3-4 MB/sec is just disturbingly
> low). And we are hesitant to disable swap entirely since one of the nodes
> is running a couple other services
>
>
> Do you have any suggestions on how we may boost the performance? Thanks!
>
>
> -- Y.
>
>
>
Re: how can we get (a lot) more performance from cassandra
Posted by Oleg Dulin <ol...@gmail.com>.
Please do keep us posted. We have a somewhat similar Cassandra
utilization pattern, and I would like to know what your solution is...
On 2012-05-16 20:38:37 +0000, Yiming Sun said:
> Thanks Oleg. Another caveat from our side is, we have a very large
> data space (imaging picking 100 items out of 3 million, the chance of
> having 2 items from the same bin is pretty low). We will experiment
> with row cache, and hopefully it will help, not the opposite (the
> tuning guide says row cache could be detrimental in some circumstances).
>
> -- Y.
>
> On Wed, May 16, 2012 at 4:25 PM, Oleg Dulin <ol...@gmail.com> wrote:
> Indeed. This is how we are trying to solve this problem.
>
> Our application has a built-in cache that resembles a supercolumn or
> standardcolumn data structure and has API that resembles a combination
> of Pelops selector and mutator. You can do something like that for
> Hector.
>
> The cache is constrained and uses LRU to purge unused items and keep
> memory usage steady.
>
> It is not perfect and we have bugs still but it cuts down on 90% of
> cassandra reads.
>
>
> On 2012-05-16 20:07:11 +0000, Mike Peters said:
>
> Hi Yiming,
>
> Cassandra is optimized for write-heavy environments.
>
> If you have a read-heavy application, you shouldn't be running your
> reads through Cassandra.
>
> On the bright side - Cassandra read throughput will remain consistent,
> regardless of your volume. But you are going to have to "wrap" your
> reads with memcache (or redis), so that the bulk of your reads can be
> served from memory.
>
>
> Thanks,
> Mike Peters
>
> On 5/16/2012 3:59 PM, Yiming Sun wrote:
> Hello,
>
> I asked the question as a follow-up under a different thread, so I
> figure I should ask here instead in case the other one gets buried, and
> besides, I have a little more information.
>
> "We find the lack of performance disturbing" as we are only able to get
> about 3-4MB/sec read performance out of Cassandra.
>
> We are using cassandra as the backend for an IR repository of digital
> texts. It is a read-mostly repository with occasional writes. Each row
> represents a book volume, and each column of a row represents a page of
> the volume. Granted the data size is small -- the average size of a
> column text is 2-3KB, and each row has about 250 columns (varies quite
> a bit from one volume to another).
>
> Currently we are running a 3-node cluster, and will soon be upgraded to
> a 6-node setup. Each node is a VM with 4 cores and 16GB of memory.
> All VMs use SAN as disk storage.
>
> To retrieve a volume, a slice query is used via Hector that specifies
> the row key (the volume), and a list of column keys (pages), and the
> consistency level is set to ONE. It is typical to retrieve multiple
> volumes per request.
>
> The read rate that I have been seeing is about 3-4 MB/sec, and that is
> reading the raw bytes... using string serializer the rate is even
> lower, about 2.2MB/sec.
>
> The server log shows the GC ParNew frequently gets longer than 200ms,
> often in the range of 4-5seconds. But nowhere near 15 seconds (which
> is an indication that JVM heap is being swapped out).
>
> Currently we have not added JNA. From a blog post, it seems JNA is
> able to increase the performance by 13%, and we are hoping to increase
> the performance by something more like 1300% (3-4 MB/sec is just
> disturbingly low). And we are hesitant to disable swap entirely since
> one of the nodes is running a couple other services
>
> Do you have any suggestions on how we may boost the performance? Thanks!
>
> -- Y.
Re: how can we get (a lot) more performance from cassandra
Posted by Yiming Sun <yi...@gmail.com>.
Thanks Oleg. Another caveat from our side is, we have a very large data
space (imaging picking 100 items out of 3 million, the chance of having 2
items from the same bin is pretty low). We will experiment with row cache,
and hopefully it will help, not the opposite (the tuning guide says row
cache could be detrimental in some circumstances).
-- Y.
On Wed, May 16, 2012 at 4:25 PM, Oleg Dulin <ol...@gmail.com> wrote:
> Indeed. This is how we are trying to solve this problem.
>
> Our application has a built-in cache that resembles a supercolumn or
> standardcolumn data structure and has API that resembles a combination of
> Pelops selector and mutator. You can do something like that for Hector.
>
> The cache is constrained and uses LRU to purge unused items and keep
> memory usage steady.
>
> It is not perfect and we have bugs still but it cuts down on 90% of
> cassandra reads.
>
>
> On 2012-05-16 20:07:11 +0000, Mike Peters said:
>
> Hi Yiming,
>>
>> Cassandra is optimized for write-heavy environments.
>>
>> If you have a read-heavy application, you shouldn't be running your reads
>> through Cassandra.
>>
>> On the bright side - Cassandra read throughput will remain consistent,
>> regardless of your volume. But you are going to have to "wrap" your reads
>> with memcache (or redis), so that the bulk of your reads can be served from
>> memory.
>>
>>
>> Thanks,
>> Mike Peters
>>
>> On 5/16/2012 3:59 PM, Yiming Sun wrote:
>> Hello,
>>
>> I asked the question as a follow-up under a different thread, so I figure
>> I should ask here instead in case the other one gets buried, and besides, I
>> have a little more information.
>>
>> "We find the lack of performance disturbing" as we are only able to get
>> about 3-4MB/sec read performance out of Cassandra.
>>
>> We are using cassandra as the backend for an IR repository of digital
>> texts. It is a read-mostly repository with occasional writes. Each row
>> represents a book volume, and each column of a row represents a page of the
>> volume. Granted the data size is small -- the average size of a column
>> text is 2-3KB, and each row has about 250 columns (varies quite a bit from
>> one volume to another).
>>
>> Currently we are running a 3-node cluster, and will soon be upgraded to a
>> 6-node setup. Each node is a VM with 4 cores and 16GB of memory. All VMs
>> use SAN as disk storage.
>>
>> To retrieve a volume, a slice query is used via Hector that specifies the
>> row key (the volume), and a list of column keys (pages), and the
>> consistency level is set to ONE. It is typical to retrieve multiple
>> volumes per request.
>>
>> The read rate that I have been seeing is about 3-4 MB/sec, and that is
>> reading the raw bytes... using string serializer the rate is even lower,
>> about 2.2MB/sec.
>>
>> The server log shows the GC ParNew frequently gets longer than 200ms,
>> often in the range of 4-5seconds. But nowhere near 15 seconds (which is an
>> indication that JVM heap is being swapped out).
>>
>> Currently we have not added JNA. From a blog post, it seems JNA is able
>> to increase the performance by 13%, and we are hoping to increase the
>> performance by something more like 1300% (3-4 MB/sec is just disturbingly
>> low). And we are hesitant to disable swap entirely since one of the nodes
>> is running a couple other services
>>
>> Do you have any suggestions on how we may boost the performance? Thanks!
>>
>> -- Y.
>>
>
>
>
>
Re: how can we get (a lot) more performance from cassandra
Posted by Oleg Dulin <ol...@gmail.com>.
Indeed. This is how we are trying to solve this problem.
Our application has a built-in cache that resembles a supercolumn or
standardcolumn data structure and has API that resembles a combination
of Pelops selector and mutator. You can do something like that for
Hector.
The cache is constrained and uses LRU to purge unused items and keep
memory usage steady.
It is not perfect and we have bugs still but it cuts down on 90% of
cassandra reads.
On 2012-05-16 20:07:11 +0000, Mike Peters said:
> Hi Yiming,
>
> Cassandra is optimized for write-heavy environments.
>
> If you have a read-heavy application, you shouldn't be running your
> reads through Cassandra.
>
> On the bright side - Cassandra read throughput will remain consistent,
> regardless of your volume. But you are going to have to "wrap" your
> reads with memcache (or redis), so that the bulk of your reads can be
> served from memory.
>
>
> Thanks,
> Mike Peters
>
> On 5/16/2012 3:59 PM, Yiming Sun wrote:
> Hello,
>
> I asked the question as a follow-up under a different thread, so I
> figure I should ask here instead in case the other one gets buried, and
> besides, I have a little more information.
>
> "We find the lack of performance disturbing" as we are only able to get
> about 3-4MB/sec read performance out of Cassandra.
>
> We are using cassandra as the backend for an IR repository of digital
> texts. It is a read-mostly repository with occasional writes. Each row
> represents a book volume, and each column of a row represents a page of
> the volume. Granted the data size is small -- the average size of a
> column text is 2-3KB, and each row has about 250 columns (varies quite
> a bit from one volume to another).
>
> Currently we are running a 3-node cluster, and will soon be upgraded to
> a 6-node setup. Each node is a VM with 4 cores and 16GB of memory.
> All VMs use SAN as disk storage.
>
> To retrieve a volume, a slice query is used via Hector that specifies
> the row key (the volume), and a list of column keys (pages), and the
> consistency level is set to ONE. It is typical to retrieve multiple
> volumes per request.
>
> The read rate that I have been seeing is about 3-4 MB/sec, and that is
> reading the raw bytes... using string serializer the rate is even
> lower, about 2.2MB/sec.
>
> The server log shows the GC ParNew frequently gets longer than 200ms,
> often in the range of 4-5seconds. But nowhere near 15 seconds (which
> is an indication that JVM heap is being swapped out).
>
> Currently we have not added JNA. From a blog post, it seems JNA is
> able to increase the performance by 13%, and we are hoping to increase
> the performance by something more like 1300% (3-4 MB/sec is just
> disturbingly low). And we are hesitant to disable swap entirely since
> one of the nodes is running a couple other services
>
> Do you have any suggestions on how we may boost the performance? Thanks!
>
> -- Y.
Re: how can we get (a lot) more performance from cassandra
Posted by Mike Peters <ca...@softwareprojects.com>.
Hi Yiming,
Cassandra is optimized for write-heavy environments.
If you have a read-heavy application, you shouldn't be running your
reads through Cassandra.
On the bright side - Cassandra read throughput will remain consistent,
regardless of your volume. But you are going to have to "wrap" your
reads with memcache (or redis), so that the bulk of your reads can be
served from memory.
Thanks,
Mike Peters
On 5/16/2012 3:59 PM, Yiming Sun wrote:
> Hello,
>
> I asked the question as a follow-up under a different thread, so I
> figure I should ask here instead in case the other one gets buried,
> and besides, I have a little more information.
>
> "We find the lack of performance disturbing" as we are only able to
> get about 3-4MB/sec read performance out of Cassandra.
>
> We are using cassandra as the backend for an IR repository of digital
> texts. It is a read-mostly repository with occasional writes. Each
> row represents a book volume, and each column of a row represents a
> page of the volume. Granted the data size is small -- the average
> size of a column text is 2-3KB, and each row has about 250 columns
> (varies quite a bit from one volume to another).
>
> Currently we are running a 3-node cluster, and will soon be upgraded
> to a 6-node setup. Each node is a VM with 4 cores and 16GB of memory.
> All VMs use SAN as disk storage.
>
> To retrieve a volume, a slice query is used via Hector that specifies
> the row key (the volume), and a list of column keys (pages), and the
> consistency level is set to ONE. It is typical to retrieve multiple
> volumes per request.
>
> The read rate that I have been seeing is about 3-4 MB/sec, and that is
> reading the raw bytes... using string serializer the rate is even
> lower, about 2.2MB/sec.
>
> The server log shows the GC ParNew frequently gets longer than 200ms,
> often in the range of 4-5seconds. But nowhere near 15 seconds (which
> is an indication that JVM heap is being swapped out).
>
> Currently we have not added JNA. From a blog post, it seems JNA is
> able to increase the performance by 13%, and we are hoping to increase
> the performance by something more like 1300% (3-4 MB/sec is just
> disturbingly low). And we are hesitant to disable swap entirely since
> one of the nodes is running a couple other services
>
> Do you have any suggestions on how we may boost the performance? Thanks!
>
> -- Y.
>
>
Re: how can we get (a lot) more performance from cassandra
Posted by aaron morton <aa...@thelastpickle.com>.
> I would look into the problems you are having with GC...
When ParNew runs the jvm pauses https://blogs.oracle.com/jonthecollector/entry/our_collectors . If it's pausing for 4 seconds it's not processing queries.
> Then check the throughput on the san and the steal on the VM's.
Check to see if the IO system is overloaded / has a long latencies see http://spyced.blogspot.co.nz/2010/01/linux-performance-basics.html
Steal time is the time your virtual cpu is waiting to get the physical cpu, you can see it in top (http://manpages.ubuntu.com/manpages/lucid/man1/top.1.html) and vmstat
> Also try to isolate the issue to "it takes this long for a single thread to make this call"
Things like "3-4 MB/sec" don't explain what you are doing. If you say "we are making this sort of query, on a single thread, and it takes this long or we only get this many a second" it's easier.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 22/05/2012, at 8:15 AM, Yiming Sun wrote:
> Hi Aaron,
>
> I don't know if you could elaborate a bit more on each of the points you suggested. Thanks.
>
> -- Y.
>
> On Sun, May 20, 2012 at 7:29 PM, aaron morton <aa...@thelastpickle.com> wrote:
> I would look into the problems you are having with GC...
>
>> The server log shows the GC ParNew frequently gets longer than 200ms, often in the range of 4-5seconds. But nowhere near 15 seconds (which is an indication that JVM heap is being swapped out).
>
> Then check the throughput on the san and the steal on the VM's.
>
> Also try to isolate the issue to "it takes this long for a single thread to make this call"
>
> In a low write environment reads should be flying along.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 17/05/2012, at 1:44 PM, Yiming Sun wrote:
>
>> Hi Aaron T., No, actually we haven't, but this sounds like a good suggestion. I can definitely try THIS before jumping into other things such as enabling row cache etc. Thanks!
>>
>> -- Y.
>>
>> On Wed, May 16, 2012 at 9:38 PM, Aaron Turner <sy...@gmail.com> wrote:
>> On Wed, May 16, 2012 at 12:59 PM, Yiming Sun <yi...@gmail.com> wrote:
>> > Hello,
>> >
>> > I asked the question as a follow-up under a different thread, so I figure I
>> > should ask here instead in case the other one gets buried, and besides, I
>> > have a little more information.
>> >
>> > "We find the lack of performance disturbing" as we are only able to get
>> > about 3-4MB/sec read performance out of Cassandra.
>> >
>> > We are using cassandra as the backend for an IR repository of digital texts.
>> > It is a read-mostly repository with occasional writes. Each row represents
>> > a book volume, and each column of a row represents a page of the volume.
>> > Granted the data size is small -- the average size of a column text is
>> > 2-3KB, and each row has about 250 columns (varies quite a bit from one
>> > volume to another).
>> >
>> > Currently we are running a 3-node cluster, and will soon be upgraded to a
>> > 6-node setup. Each node is a VM with 4 cores and 16GB of memory. All VMs
>> > use SAN as disk storage.
>> >
>> > To retrieve a volume, a slice query is used via Hector that specifies the
>> > row key (the volume), and a list of column keys (pages), and the consistency
>> > level is set to ONE. It is typical to retrieve multiple volumes per
>> > request.
>> >
>> > The read rate that I have been seeing is about 3-4 MB/sec, and that is
>> > reading the raw bytes... using string serializer the rate is even lower,
>> > about 2.2MB/sec.
>> >
>> > The server log shows the GC ParNew frequently gets longer than 200ms, often
>> > in the range of 4-5seconds. But nowhere near 15 seconds (which is an
>> > indication that JVM heap is being swapped out).
>> >
>> > Currently we have not added JNA. From a blog post, it seems JNA is able to
>> > increase the performance by 13%, and we are hoping to increase the
>> > performance by something more like 1300% (3-4 MB/sec is just disturbingly
>> > low). And we are hesitant to disable swap entirely since one of the nodes
>> > is running a couple other services
>> >
>> > Do you have any suggestions on how we may boost the performance? Thanks!
>>
>> Have you tried using more threads on the client side? Generally
>> speaking, when I need faster read/write performance I look for ways to
>> parallelize my requests and it scales pretty much linearly.
>>
>>
>> --
>> Aaron Turner
>> http://synfin.net/ Twitter: @synfinatic
>> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
>> Those who would give up essential Liberty, to purchase a little temporary
>> Safety, deserve neither Liberty nor Safety.
>> -- Benjamin Franklin
>> "carpe diem quam minimum credula postero"
>>
>
>
Re: how can we get (a lot) more performance from cassandra
Posted by Yiming Sun <yi...@gmail.com>.
Hi Aaron,
I don't know if you could elaborate a bit more on each of the points you
suggested. Thanks.
-- Y.
On Sun, May 20, 2012 at 7:29 PM, aaron morton <aa...@thelastpickle.com>wrote:
> I would look into the problems you are having with GC...
>
> The server log shows the GC ParNew frequently gets longer than 200ms,
> often in the range of 4-5seconds. But nowhere near 15 seconds (which is an
> indication that JVM heap is being swapped out).
>
>
> Then check the throughput on the san and the steal on the VM's.
>
> Also try to isolate the issue to "it takes this long for a single thread
> to make this call"
>
> In a low write environment reads should be flying along.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 17/05/2012, at 1:44 PM, Yiming Sun wrote:
>
> Hi Aaron T., No, actually we haven't, but this sounds like a good
> suggestion. I can definitely try THIS before jumping into other things
> such as enabling row cache etc. Thanks!
>
> -- Y.
>
> On Wed, May 16, 2012 at 9:38 PM, Aaron Turner <sy...@gmail.com>wrote:
>
>> On Wed, May 16, 2012 at 12:59 PM, Yiming Sun <yi...@gmail.com>
>> wrote:
>> > Hello,
>> >
>> > I asked the question as a follow-up under a different thread, so I
>> figure I
>> > should ask here instead in case the other one gets buried, and besides,
>> I
>> > have a little more information.
>> >
>> > "We find the lack of performance disturbing" as we are only able to get
>> > about 3-4MB/sec read performance out of Cassandra.
>> >
>> > We are using cassandra as the backend for an IR repository of digital
>> texts.
>> > It is a read-mostly repository with occasional writes. Each row
>> represents
>> > a book volume, and each column of a row represents a page of the volume.
>> > Granted the data size is small -- the average size of a column text is
>> > 2-3KB, and each row has about 250 columns (varies quite a bit from one
>> > volume to another).
>> >
>> > Currently we are running a 3-node cluster, and will soon be upgraded to
>> a
>> > 6-node setup. Each node is a VM with 4 cores and 16GB of memory. All
>> VMs
>> > use SAN as disk storage.
>> >
>> > To retrieve a volume, a slice query is used via Hector that specifies
>> the
>> > row key (the volume), and a list of column keys (pages), and the
>> consistency
>> > level is set to ONE. It is typical to retrieve multiple volumes per
>> > request.
>> >
>> > The read rate that I have been seeing is about 3-4 MB/sec, and that is
>> > reading the raw bytes... using string serializer the rate is even lower,
>> > about 2.2MB/sec.
>> >
>> > The server log shows the GC ParNew frequently gets longer than 200ms,
>> often
>> > in the range of 4-5seconds. But nowhere near 15 seconds (which is an
>> > indication that JVM heap is being swapped out).
>> >
>> > Currently we have not added JNA. From a blog post, it seems JNA is
>> able to
>> > increase the performance by 13%, and we are hoping to increase the
>> > performance by something more like 1300% (3-4 MB/sec is just
>> disturbingly
>> > low). And we are hesitant to disable swap entirely since one of the
>> nodes
>> > is running a couple other services
>> >
>> > Do you have any suggestions on how we may boost the performance?
>> Thanks!
>>
>> Have you tried using more threads on the client side? Generally
>> speaking, when I need faster read/write performance I look for ways to
>> parallelize my requests and it scales pretty much linearly.
>>
>>
>> --
>> Aaron Turner
>> http://synfin.net/ Twitter: @synfinatic
>> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix &
>> Windows
>> Those who would give up essential Liberty, to purchase a little temporary
>> Safety, deserve neither Liberty nor Safety.
>> -- Benjamin Franklin
>> "carpe diem quam minimum credula postero"
>>
>
>
>
Re: how can we get (a lot) more performance from cassandra
Posted by aaron morton <aa...@thelastpickle.com>.
I would look into the problems you are having with GC...
> The server log shows the GC ParNew frequently gets longer than 200ms, often in the range of 4-5seconds. But nowhere near 15 seconds (which is an indication that JVM heap is being swapped out).
Then check the throughput on the san and the steal on the VM's.
Also try to isolate the issue to "it takes this long for a single thread to make this call"
In a low write environment reads should be flying along.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 17/05/2012, at 1:44 PM, Yiming Sun wrote:
> Hi Aaron T., No, actually we haven't, but this sounds like a good suggestion. I can definitely try THIS before jumping into other things such as enabling row cache etc. Thanks!
>
> -- Y.
>
> On Wed, May 16, 2012 at 9:38 PM, Aaron Turner <sy...@gmail.com> wrote:
> On Wed, May 16, 2012 at 12:59 PM, Yiming Sun <yi...@gmail.com> wrote:
> > Hello,
> >
> > I asked the question as a follow-up under a different thread, so I figure I
> > should ask here instead in case the other one gets buried, and besides, I
> > have a little more information.
> >
> > "We find the lack of performance disturbing" as we are only able to get
> > about 3-4MB/sec read performance out of Cassandra.
> >
> > We are using cassandra as the backend for an IR repository of digital texts.
> > It is a read-mostly repository with occasional writes. Each row represents
> > a book volume, and each column of a row represents a page of the volume.
> > Granted the data size is small -- the average size of a column text is
> > 2-3KB, and each row has about 250 columns (varies quite a bit from one
> > volume to another).
> >
> > Currently we are running a 3-node cluster, and will soon be upgraded to a
> > 6-node setup. Each node is a VM with 4 cores and 16GB of memory. All VMs
> > use SAN as disk storage.
> >
> > To retrieve a volume, a slice query is used via Hector that specifies the
> > row key (the volume), and a list of column keys (pages), and the consistency
> > level is set to ONE. It is typical to retrieve multiple volumes per
> > request.
> >
> > The read rate that I have been seeing is about 3-4 MB/sec, and that is
> > reading the raw bytes... using string serializer the rate is even lower,
> > about 2.2MB/sec.
> >
> > The server log shows the GC ParNew frequently gets longer than 200ms, often
> > in the range of 4-5seconds. But nowhere near 15 seconds (which is an
> > indication that JVM heap is being swapped out).
> >
> > Currently we have not added JNA. From a blog post, it seems JNA is able to
> > increase the performance by 13%, and we are hoping to increase the
> > performance by something more like 1300% (3-4 MB/sec is just disturbingly
> > low). And we are hesitant to disable swap entirely since one of the nodes
> > is running a couple other services
> >
> > Do you have any suggestions on how we may boost the performance? Thanks!
>
> Have you tried using more threads on the client side? Generally
> speaking, when I need faster read/write performance I look for ways to
> parallelize my requests and it scales pretty much linearly.
>
>
> --
> Aaron Turner
> http://synfin.net/ Twitter: @synfinatic
> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
> -- Benjamin Franklin
> "carpe diem quam minimum credula postero"
>
Re: how can we get (a lot) more performance from cassandra
Posted by Yiming Sun <yi...@gmail.com>.
Hi Aaron T., No, actually we haven't, but this sounds like a good
suggestion. I can definitely try THIS before jumping into other things
such as enabling row cache etc. Thanks!
-- Y.
On Wed, May 16, 2012 at 9:38 PM, Aaron Turner <sy...@gmail.com> wrote:
> On Wed, May 16, 2012 at 12:59 PM, Yiming Sun <yi...@gmail.com> wrote:
> > Hello,
> >
> > I asked the question as a follow-up under a different thread, so I
> figure I
> > should ask here instead in case the other one gets buried, and besides, I
> > have a little more information.
> >
> > "We find the lack of performance disturbing" as we are only able to get
> > about 3-4MB/sec read performance out of Cassandra.
> >
> > We are using cassandra as the backend for an IR repository of digital
> texts.
> > It is a read-mostly repository with occasional writes. Each row
> represents
> > a book volume, and each column of a row represents a page of the volume.
> > Granted the data size is small -- the average size of a column text is
> > 2-3KB, and each row has about 250 columns (varies quite a bit from one
> > volume to another).
> >
> > Currently we are running a 3-node cluster, and will soon be upgraded to a
> > 6-node setup. Each node is a VM with 4 cores and 16GB of memory. All
> VMs
> > use SAN as disk storage.
> >
> > To retrieve a volume, a slice query is used via Hector that specifies the
> > row key (the volume), and a list of column keys (pages), and the
> consistency
> > level is set to ONE. It is typical to retrieve multiple volumes per
> > request.
> >
> > The read rate that I have been seeing is about 3-4 MB/sec, and that is
> > reading the raw bytes... using string serializer the rate is even lower,
> > about 2.2MB/sec.
> >
> > The server log shows the GC ParNew frequently gets longer than 200ms,
> often
> > in the range of 4-5seconds. But nowhere near 15 seconds (which is an
> > indication that JVM heap is being swapped out).
> >
> > Currently we have not added JNA. From a blog post, it seems JNA is able
> to
> > increase the performance by 13%, and we are hoping to increase the
> > performance by something more like 1300% (3-4 MB/sec is just disturbingly
> > low). And we are hesitant to disable swap entirely since one of the
> nodes
> > is running a couple other services
> >
> > Do you have any suggestions on how we may boost the performance? Thanks!
>
> Have you tried using more threads on the client side? Generally
> speaking, when I need faster read/write performance I look for ways to
> parallelize my requests and it scales pretty much linearly.
>
>
> --
> Aaron Turner
> http://synfin.net/ Twitter: @synfinatic
> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix &
> Windows
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
> -- Benjamin Franklin
> "carpe diem quam minimum credula postero"
>
Re: how can we get (a lot) more performance from cassandra
Posted by Aaron Turner <sy...@gmail.com>.
On Wed, May 16, 2012 at 12:59 PM, Yiming Sun <yi...@gmail.com> wrote:
> Hello,
>
> I asked the question as a follow-up under a different thread, so I figure I
> should ask here instead in case the other one gets buried, and besides, I
> have a little more information.
>
> "We find the lack of performance disturbing" as we are only able to get
> about 3-4MB/sec read performance out of Cassandra.
>
> We are using cassandra as the backend for an IR repository of digital texts.
> It is a read-mostly repository with occasional writes. Each row represents
> a book volume, and each column of a row represents a page of the volume.
> Granted the data size is small -- the average size of a column text is
> 2-3KB, and each row has about 250 columns (varies quite a bit from one
> volume to another).
>
> Currently we are running a 3-node cluster, and will soon be upgraded to a
> 6-node setup. Each node is a VM with 4 cores and 16GB of memory. All VMs
> use SAN as disk storage.
>
> To retrieve a volume, a slice query is used via Hector that specifies the
> row key (the volume), and a list of column keys (pages), and the consistency
> level is set to ONE. It is typical to retrieve multiple volumes per
> request.
>
> The read rate that I have been seeing is about 3-4 MB/sec, and that is
> reading the raw bytes... using string serializer the rate is even lower,
> about 2.2MB/sec.
>
> The server log shows the GC ParNew frequently gets longer than 200ms, often
> in the range of 4-5seconds. But nowhere near 15 seconds (which is an
> indication that JVM heap is being swapped out).
>
> Currently we have not added JNA. From a blog post, it seems JNA is able to
> increase the performance by 13%, and we are hoping to increase the
> performance by something more like 1300% (3-4 MB/sec is just disturbingly
> low). And we are hesitant to disable swap entirely since one of the nodes
> is running a couple other services
>
> Do you have any suggestions on how we may boost the performance? Thanks!
Have you tried using more threads on the client side? Generally
speaking, when I need faster read/write performance I look for ways to
parallelize my requests and it scales pretty much linearly.
--
Aaron Turner
http://synfin.net/ Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
-- Benjamin Franklin
"carpe diem quam minimum credula postero"