You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rishi Bhardwaj <kh...@yahoo.com> on 2010/06/11 02:42:20 UTC
Cassandra Write Performance, CPU usage
Hi
I am investigating Cassandra write performance and see very heavy CPU usage from Cassandra. I have a single node Cassandra instance running on a dual core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are being generated from the same server using BatchMutate(). The client makes exactly one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2 MB of data and once it is acknowledged by Cassandra, the next RPC is done. Cassandra has two separate disks, one for commitlog with a sequential b/w of 130MBps and the other a solid state disk for data with b/w of 90MBps. Tuning various parameters, I observe that I am able to attain a maximum write performance of about 45 to 50 MBps from Cassandra. I see that the Cassandra java process consistently uses 100% to 150% of CPU resources (as shown by top) during the entire write operation. Also, iostat clearly shows that the max disk bandwidth is not reached anytime during the write
operation, every now and then the i/o activity on "commitlog" disk and the data disk spike but it is never consistently maintained by cassandra close to their peak. I would imagine that the CPU is probably the bottleneck here. Does anyone have any idea why Cassandra beats the heck out of the CPU here? Any suggestions on how to go about finding the exact bottleneck here?
Some more information about the writes: I have 2 column families, the data though is mostly written in one column family with column sizes of around 32k and each row having around 256 or 512 columns. I would really appreciate any help here.
Thanks,
Rishi
Re: Cassandra Write Performance, CPU usage
Posted by Jonathan Shook <js...@gmail.com>.
Rishi,
I am not yet knowledgeable enough to answer your question in more
detail. I would like to know more about the specifics as well.
There are counters you can use via JMX to show logical events, but
this will not always translate to good baseline information that you
can use in scaling estimates.
I would like to see a good analysis that characterizes the scaling
factors of different parts of the system, both from load
characterization and from an algorithmic perspective.
This is a common area of inquiry. Maybe we should start
http://wiki.apache.org/cassandra/ScalabilityFactors
On Thu, Jun 10, 2010 at 11:05 PM, Rishi Bhardwaj <kh...@yahoo.com> wrote:
> Hi Jonathan
> Thanks for such an informative reply. My application may end up doing such
> continuous bulk writes to Cassandra and thus I was interested in such a
> performance case. I was wondering as to what are all the CPU overheads for
> each row/column written to Cassandra? You mentioned updating of bloom
> filters, would that be the main CPU overhead, there may even be copying of
> data happening? I want to investigate about all the factors in play here and
> if there is a possibility for improvement. Is it possible to profile
> cassandra and see what maybe the bottleneck here. The auxiliary I/O you had
> mentioned for the Bloom filters, wouldn't that occur with the I/O for the
> SSTable, in which case the extra I/O for the bloom filter gets piggybacked
> with the SSTable I/O? I guess I don't understand the Cassandra internals too
> well but wanted to see how much can Cassandra achieve for continuous bulk
> writes.
> Has anyone done any bulk write experiments with Cassandra? Is Cassandra
> performance always expected to be bottlenecked by CPU when doing continuous
> bulk writes?
> Thanks for all the help,
> Rishi
> ________________________________
> From: Jonathan Shook <js...@gmail.com>
> To: user@cassandra.apache.org
> Sent: Thu, June 10, 2010 7:39:24 PM
> Subject: Re: Cassandra Write Performance, CPU usage
>
> You are testing Cassandra in a way that it was not designed to be used.
> Bandwidth to disk is not a meaningful example for nearly anything
> except for filesystem benchmarking and things very nearly the same as
> filesystem benchmarking.
> Unless the usage patterns of your application match your test data,
> there is not a good reason to expect a strong correlation between this
> test and actual performance.
>
> Cassandra is not simply shuffling data through IO when you write.
> There are calculations that have to be done as writes filter their way
> through various stages of processing. The point of this is to minimize
> the overall effort Cassandra has to make in order to retrieve the data
> again. One example would be bloom filters. Each column that is written
> requires bloom filter processing and potentially auxiliary IO. Some of
> these steps are allowed to happen in the background, but if you try,
> you can cause them to stack up on top of the available CPU and memory
> resources.
>
> In such a case (continuous bulk writes), you are causing all of these
> costs to be taken in more of a synchronous (not delayed) fashion. You
> are not allowing the background processing that helps reduce client
> blocking (by deferring some processing) to do its magic.
>
>
>
> On Thu, Jun 10, 2010 at 7:42 PM, Rishi Bhardwaj <kh...@yahoo.com>
> wrote:
>> Hi
>> I am investigating Cassandra write performance and see very heavy CPU
>> usage
>> from Cassandra. I have a single node Cassandra instance running on a dual
>> core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are
>> being
>> generated from the same server using BatchMutate(). The client makes
>> exactly
>> one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2 MB
>> of
>> data and once it is acknowledged by Cassandra, the next RPC is done.
>> Cassandra has two separate disks, one for commitlog with a sequential b/w
>> of
>> 130MBps and the other a solid state disk for data with b/w of 90MBps.
>> Tuning
>> various parameters, I observe that I am able to attain a maximum write
>> performance of about 45 to 50 MBps from Cassandra. I see that the
>> Cassandra
>> java process consistently uses 100% to 150% of CPU resources (as shown by
>> top) during the entire write operation. Also, iostat clearly shows that
>> the
>> max disk bandwidth is not reached anytime during the write operation,
>> every
>> now and then the i/o activity on "commitlog" disk and the data disk spike
>> but it is never consistently maintained by cassandra close to their
>> peak. I
>> would imagine that the CPU is probably the bottleneck here. Does anyone
>> have
>> any idea why Cassandra beats the heck out of the CPU here? Any suggestions
>> on how to go about finding the exact bottleneck here?
>> Some more information about the writes: I have 2 column families, the data
>> though is mostly written in one column family with column sizes of around
>> 32k and each row having around 256 or 512 columns. I would really
>> appreciate
>> any help here.
>> Thanks,
>> Rishi
>>
>>
>
>
Re: Cassandra Write Performance, CPU usage
Posted by Rishi Bhardwaj <kh...@yahoo.com>.
Hi Jonathan
Thanks for such an informative reply. My application may end up doing such continuous bulk writes to Cassandra and thus I was interested in such a performance case. I was wondering as to what are all the CPU overheads for each row/column written to Cassandra? You mentioned updating of bloom filters, would that be the main CPU overhead, there may even be copying of data happening? I want to investigate about all the factors in play here and if there is a possibility for improvement. Is it possible to profile cassandra and see what maybe the bottleneck here. The auxiliary I/O you had mentioned for the Bloom filters, wouldn't that occur with the I/O for the SSTable, in which case the extra I/O for the bloom filter gets piggybacked with the SSTable I/O? I guess I don't understand the Cassandra internals too well but wanted to see how much can Cassandra achieve for continuous bulk writes.
Has anyone done any bulk write experiments with Cassandra? Is Cassandra performance always expected to be bottlenecked by CPU when doing continuous bulk writes?
Thanks for all the help,
Rishi
________________________________
From: Jonathan Shook <js...@gmail.com>
To: user@cassandra.apache.org
Sent: Thu, June 10, 2010 7:39:24 PM
Subject: Re: Cassandra Write Performance, CPU usage
You are testing Cassandra in a way that it was not designed to be used.
Bandwidth to disk is not a meaningful example for nearly anything
except for filesystem benchmarking and things very nearly the same as
filesystem benchmarking.
Unless the usage patterns of your application match your test data,
there is not a good reason to expect a strong correlation between this
test and actual performance.
Cassandra is not simply shuffling data through IO when you write.
There are calculations that have to be done as writes filter their way
through various stages of processing. The point of this is to minimize
the overall effort Cassandra has to make in order to retrieve the data
again. One example would be bloom filters. Each column that is written
requires bloom filter processing and potentially auxiliary IO. Some of
these steps are allowed to happen in the background, but if you try,
you can cause them to stack up on top of the available CPU and memory
resources.
In such a case (continuous bulk writes), you are causing all of these
costs to be taken in more of a synchronous (not delayed) fashion. You
are not allowing the background processing that helps reduce client
blocking (by deferring some processing) to do its magic.
On Thu, Jun 10, 2010 at 7:42 PM, Rishi Bhardwaj <kh...@yahoo.com> wrote:
> Hi
> I am investigating Cassandra write performance and see very heavy CPU usage
> from Cassandra. I have a single node Cassandra instance running on a dual
> core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are being
> generated from the same server using BatchMutate(). The client makes exactly
> one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2 MB of
> data and once it is acknowledged by Cassandra, the next RPC is done.
> Cassandra has two separate disks, one for commitlog with a sequential b/w of
> 130MBps and the other a solid state disk for data with b/w of 90MBps. Tuning
> various parameters, I observe that I am able to attain a maximum write
> performance of about 45 to 50 MBps from Cassandra. I see that the Cassandra
> java process consistently uses 100% to 150% of CPU resources (as shown by
> top) during the entire write operation. Also, iostat clearly shows that the
> max disk bandwidth is not reached anytime during the write operation, every
> now and then the i/o activity on "commitlog" disk and the data disk spike
> but it is never consistently maintained by cassandra close to their peak. I
> would imagine that the CPU is probably the bottleneck here. Does anyone have
> any idea why Cassandra beats the heck out of the CPU here? Any suggestions
> on how to go about finding the exact bottleneck here?
> Some more information about the writes: I have 2 column families, the data
> though is mostly written in one column family with column sizes of around
> 32k and each row having around 256 or 512 columns. I would really appreciate
> any help here.
> Thanks,
> Rishi
>
>
Re: Cassandra Write Performance, CPU usage
Posted by Rishi Bhardwaj <kh...@yahoo.com>.
Can we configure/optimize the write path to lower CPU bound and improve performance? I am wondering if I should investigate and see what is eating up so much of CPU (memory/data copying? bloom filters? etc.). Would this be a worthwhile investigation to see if we can improve on things or is there an inherent reason why the high CPU bound on continuous bulk writes can't be improved?
Thanks for all the help,
Rishi
________________________________
From: Jonathan Ellis <jb...@gmail.com>
To: user@cassandra.apache.org
Sent: Fri, June 11, 2010 12:22:39 PM
Subject: Re: Cassandra Write Performance, CPU usage
yes, it is expected that writes are cpu-bound.
On Fri, Jun 11, 2010 at 11:29 AM, Rishi Bhardwaj <kh...@yahoo.com> wrote:
> I think it would be a good exercise to know what the CPU bottleneck is on
> the write path. The fact that Cassandra optimizes disk I/O for writes would
> only go so far if the CPU becomes a big bottleneck on continuous writes. I
> am fairly new to Java ecosystem performance profiling but I would give it a
> try and see if I can pinpoint the problem area here. I am also thinking
> about making concurrent writes to cassandra instead of only one write at a
> time. This would probably make Cassandra beat the hell out of all CPU
> resources and confirm that Cassandra is CPU bound on continuous writes.
> Again, I would love to hear from Cassandra experts here and see what they
> think of this. Are Cassandra continuous bulk writes expected to be
> bottlenecked by CPU? If this is definitely the case and thats what it seems
> right now, then it would be a good thing to look at the algorithms in the
> write path.
> Thanks,
> Rishi
> ________________________________
> From: Mike Malone <mi...@simplegeo.com>
> To: user@cassandra.apache.org
> Sent: Fri, June 11, 2010 9:20:06 AM
> Subject: Re: Cassandra Write Performance, CPU usage
>
> Jonathan, while I agree with you re: this being an unusual load for the
> system, it is interesting that he's found at least one use-case where
> Cassandra is CPU-bound, not IO-bound. I'd definitely be interested in
> learning what his critical path is and seeing if there's some low-hanging
> fruit that may improve performance overall. I have also noticed very high
> CPU usage during high write loads and have wondered whether write speed and
> throughput could be improved by improving some of the algorithms along that
> path.
> I'm nowhere near being an expert on the whole Java ecosystem, but I've had
> good luck with the `jvisualvm` tool that comes with Java SE 6. It's a nice
> lightweight CPU and memory profiling tool that can attach to a running
> process like Cassandra and dump stats in real time.
> Mike
>
> On Thu, Jun 10, 2010 at 7:39 PM, Jonathan Shook <js...@gmail.com> wrote:
>>
>> You are testing Cassandra in a way that it was not designed to be used.
>> Bandwidth to disk is not a meaningful example for nearly anything
>> except for filesystem benchmarking and things very nearly the same as
>> filesystem benchmarking.
>> Unless the usage patterns of your application match your test data,
>> there is not a good reason to expect a strong correlation between this
>> test and actual performance.
>>
>> Cassandra is not simply shuffling data through IO when you write.
>> There are calculations that have to be done as writes filter their way
>> through various stages of processing. The point of this is to minimize
>> the overall effort Cassandra has to make in order to retrieve the data
>> again. One example would be bloom filters. Each column that is written
>> requires bloom filter processing and potentially auxiliary IO. Some of
>> these steps are allowed to happen in the background, but if you try,
>> you can cause them to stack up on top of the available CPU and memory
>> resources.
>>
>> In such a case (continuous bulk writes), you are causing all of these
>> costs to be taken in more of a synchronous (not delayed) fashion. You
>> are not allowing the background processing that helps reduce client
>> blocking (by deferring some processing) to do its magic.
>>
>>
>>
>> On Thu, Jun 10, 2010 at 7:42 PM, Rishi Bhardwaj <kh...@yahoo.com>
>> wrote:
>> > Hi
>> > I am investigating Cassandra write performance and see very heavy CPU
>> > usage
>> > from Cassandra. I have a single node Cassandra instance running on a
>> > dual
>> > core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are
>> > being
>> > generated from the same server using BatchMutate(). The client makes
>> > exactly
>> > one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2
>> > MB of
>> > data and once it is acknowledged by Cassandra, the next RPC is done.
>> > Cassandra has two separate disks, one for commitlog with a sequential
>> > b/w of
>> > 130MBps and the other a solid state disk for data with b/w of 90MBps.
>> > Tuning
>> > various parameters, I observe that I am able to attain a maximum write
>> > performance of about 45 to 50 MBps from Cassandra. I see that the
>> > Cassandra
>> > java process consistently uses 100% to 150% of CPU resources (as shown
>> > by
>> > top) during the entire write operation. Also, iostat clearly shows that
>> > the
>> > max disk bandwidth is not reached anytime during the write operation,
>> > every
>> > now and then the i/o activity on "commitlog" disk and the data disk
>> > spike
>> > but it is never consistently maintained by cassandra close to their
>> > peak. I
>> > would imagine that the CPU is probably the bottleneck here. Does anyone
>> > have
>> > any idea why Cassandra beats the heck out of the CPU here? Any
>> > suggestions
>> > on how to go about finding the exact bottleneck here?
>> > Some more information about the writes: I have 2 column families, the
>> > data
>> > though is mostly written in one column family with column sizes of
>> > around
>> > 32k and each row having around 256 or 512 columns. I would really
>> > appreciate
>> > any help here.
>> > Thanks,
>> > Rishi
>> >
>> >
>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Cassandra Write Performance, CPU usage
Posted by Jonathan Ellis <jb...@gmail.com>.
yes, it is expected that writes are cpu-bound.
On Fri, Jun 11, 2010 at 11:29 AM, Rishi Bhardwaj <kh...@yahoo.com> wrote:
> I think it would be a good exercise to know what the CPU bottleneck is on
> the write path. The fact that Cassandra optimizes disk I/O for writes would
> only go so far if the CPU becomes a big bottleneck on continuous writes. I
> am fairly new to Java ecosystem performance profiling but I would give it a
> try and see if I can pinpoint the problem area here. I am also thinking
> about making concurrent writes to cassandra instead of only one write at a
> time. This would probably make Cassandra beat the hell out of all CPU
> resources and confirm that Cassandra is CPU bound on continuous writes.
> Again, I would love to hear from Cassandra experts here and see what they
> think of this. Are Cassandra continuous bulk writes expected to be
> bottlenecked by CPU? If this is definitely the case and thats what it seems
> right now, then it would be a good thing to look at the algorithms in the
> write path.
> Thanks,
> Rishi
> ________________________________
> From: Mike Malone <mi...@simplegeo.com>
> To: user@cassandra.apache.org
> Sent: Fri, June 11, 2010 9:20:06 AM
> Subject: Re: Cassandra Write Performance, CPU usage
>
> Jonathan, while I agree with you re: this being an unusual load for the
> system, it is interesting that he's found at least one use-case where
> Cassandra is CPU-bound, not IO-bound. I'd definitely be interested in
> learning what his critical path is and seeing if there's some low-hanging
> fruit that may improve performance overall. I have also noticed very high
> CPU usage during high write loads and have wondered whether write speed and
> throughput could be improved by improving some of the algorithms along that
> path.
> I'm nowhere near being an expert on the whole Java ecosystem, but I've had
> good luck with the `jvisualvm` tool that comes with Java SE 6. It's a nice
> lightweight CPU and memory profiling tool that can attach to a running
> process like Cassandra and dump stats in real time.
> Mike
>
> On Thu, Jun 10, 2010 at 7:39 PM, Jonathan Shook <js...@gmail.com> wrote:
>>
>> You are testing Cassandra in a way that it was not designed to be used.
>> Bandwidth to disk is not a meaningful example for nearly anything
>> except for filesystem benchmarking and things very nearly the same as
>> filesystem benchmarking.
>> Unless the usage patterns of your application match your test data,
>> there is not a good reason to expect a strong correlation between this
>> test and actual performance.
>>
>> Cassandra is not simply shuffling data through IO when you write.
>> There are calculations that have to be done as writes filter their way
>> through various stages of processing. The point of this is to minimize
>> the overall effort Cassandra has to make in order to retrieve the data
>> again. One example would be bloom filters. Each column that is written
>> requires bloom filter processing and potentially auxiliary IO. Some of
>> these steps are allowed to happen in the background, but if you try,
>> you can cause them to stack up on top of the available CPU and memory
>> resources.
>>
>> In such a case (continuous bulk writes), you are causing all of these
>> costs to be taken in more of a synchronous (not delayed) fashion. You
>> are not allowing the background processing that helps reduce client
>> blocking (by deferring some processing) to do its magic.
>>
>>
>>
>> On Thu, Jun 10, 2010 at 7:42 PM, Rishi Bhardwaj <kh...@yahoo.com>
>> wrote:
>> > Hi
>> > I am investigating Cassandra write performance and see very heavy CPU
>> > usage
>> > from Cassandra. I have a single node Cassandra instance running on a
>> > dual
>> > core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are
>> > being
>> > generated from the same server using BatchMutate(). The client makes
>> > exactly
>> > one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2
>> > MB of
>> > data and once it is acknowledged by Cassandra, the next RPC is done.
>> > Cassandra has two separate disks, one for commitlog with a sequential
>> > b/w of
>> > 130MBps and the other a solid state disk for data with b/w of 90MBps.
>> > Tuning
>> > various parameters, I observe that I am able to attain a maximum write
>> > performance of about 45 to 50 MBps from Cassandra. I see that the
>> > Cassandra
>> > java process consistently uses 100% to 150% of CPU resources (as shown
>> > by
>> > top) during the entire write operation. Also, iostat clearly shows that
>> > the
>> > max disk bandwidth is not reached anytime during the write operation,
>> > every
>> > now and then the i/o activity on "commitlog" disk and the data disk
>> > spike
>> > but it is never consistently maintained by cassandra close to their
>> > peak. I
>> > would imagine that the CPU is probably the bottleneck here. Does anyone
>> > have
>> > any idea why Cassandra beats the heck out of the CPU here? Any
>> > suggestions
>> > on how to go about finding the exact bottleneck here?
>> > Some more information about the writes: I have 2 column families, the
>> > data
>> > though is mostly written in one column family with column sizes of
>> > around
>> > 32k and each row having around 256 or 512 columns. I would really
>> > appreciate
>> > any help here.
>> > Thanks,
>> > Rishi
>> >
>> >
>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Cassandra Write Performance, CPU usage
Posted by Rishi Bhardwaj <kh...@yahoo.com>.
I think it would be a good exercise to know what the CPU bottleneck is on the write path. The fact that Cassandra optimizes disk I/O for writes would only go so far if the CPU becomes a big bottleneck on continuous writes. I am fairly new to Java ecosystem performance profiling but I would give it a try and see if I can pinpoint the problem area here. I am also thinking about making concurrent writes to cassandra instead of only one write at a time. This would probably make Cassandra beat the hell out of all CPU resources and confirm that Cassandra is CPU bound on continuous writes.
Again, I would love to hear from Cassandra experts here and see what they think of this. Are Cassandra continuous bulk writes expected to be bottlenecked by CPU? If this is definitely the case and thats what it seems right now, then it would be a good thing to look at the algorithms in the write path.
Thanks,
Rishi
________________________________
From: Mike Malone <mi...@simplegeo.com>
To: user@cassandra.apache.org
Sent: Fri, June 11, 2010 9:20:06 AM
Subject: Re: Cassandra Write Performance, CPU usage
Jonathan, while I agree with you re: this being an unusual load for the system, it is interesting that he's found at least one use-case where Cassandra is CPU-bound, not IO-bound. I'd definitely be interested in learning what his critical path is and seeing if there's some low-hanging fruit that may improve performance overall. I have also noticed very high CPU usage during high write loads and have wondered whether write speed and throughput could be improved by improving some of the algorithms along that path.
I'm nowhere near being an expert on the whole Java ecosystem, but I've had good luck with the `jvisualvm` tool that comes with Java SE 6. It's a nice lightweight CPU and memory profiling tool that can attach to a running process like Cassandra and dump stats in real time.
Mike
On Thu, Jun 10, 2010 at 7:39 PM, Jonathan Shook <js...@gmail.com> wrote:
>You are testing Cassandra in a way that it was not designed to be used.
>>Bandwidth to disk is not a meaningful example for nearly anything
>>except for filesystem benchmarking and things very nearly the same as
>>filesystem benchmarking.
>>Unless the usage patterns of your application match your test data,
>>there is not a good reason to expect a strong correlation between this
>>test and actual performance.
>
>>Cassandra is not simply shuffling data through IO when you write.
>>There are calculations that have to be done as writes filter their way
>>through various stages of processing. The point of this is to minimize
>>the overall effort Cassandra has to make in order to retrieve the data
>>again. One example would be bloom filters. Each column that is written
>>requires bloom filter processing and potentially auxiliary IO. Some of
>>these steps are allowed to happen in the background, but if you try,
>>you can cause them to stack up on top of the available CPU and memory
>>resources.
>
>>In such a case (continuous bulk writes), you are causing all of these
>>costs to be taken in more of a synchronous (not delayed) fashion. You
>>are not allowing the background processing that helps reduce client
>>blocking (by deferring some processing) to do its magic.
>
>
>
>
>>On Thu, Jun 10, 2010 at 7:42 PM, Rishi Bhardwaj <kh...@yahoo.com> wrote:
>>> Hi
>>> I am investigating Cassandra write performance and see very heavy CPU usage
>>> from Cassandra. I have a single node Cassandra instance running on a dual
>>> core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are being
>>> generated from the same server using BatchMutate(). The client makes exactly
>>> one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2 MB of
>>> data and once it is acknowledged by Cassandra, the next RPC is done.
>>> Cassandra has two separate disks, one for commitlog with a sequential b/w of
>>> 130MBps and the other a solid state disk for data with b/w of 90MBps. Tuning
>>> various parameters, I observe that I am able to attain a maximum write
>>> performance of about 45 to 50 MBps from Cassandra. I see that the Cassandra
>>> java process consistently uses 100% to 150% of CPU resources (as shown by
>>> top) during the entire write operation. Also, iostat clearly shows that the
>>> max disk bandwidth is not reached anytime during the write operation, every
>>> now and then the i/o activity on "commitlog" disk and the data disk spike
>>> but it is never consistently maintained by cassandra close to their peak. I
>>> would imagine that the CPU is probably the bottleneck here. Does anyone have
>>> any idea why Cassandra beats the heck out of the CPU here? Any suggestions
>>> on how to go about finding the exact bottleneck here?
>>> Some more information about the writes: I have 2 column families, the data
>>> though is mostly written in one column family with column sizes of around
>>> 32k and each row having around 256 or 512 columns. I would really appreciate
>>> any help here.
>>> Thanks,
>>> Rishi
>>>
>>>
>
Re: Cassandra Write Performance, CPU usage
Posted by Mike Malone <mi...@simplegeo.com>.
Jonathan, while I agree with you re: this being an unusual load for the
system, it is interesting that he's found at least one use-case where
Cassandra is CPU-bound, not IO-bound. I'd definitely be interested in
learning what his critical path is and seeing if there's some low-hanging
fruit that may improve performance overall. I have also noticed very high
CPU usage during high write loads and have wondered whether write speed and
throughput could be improved by improving some of the algorithms along that
path.
I'm nowhere near being an expert on the whole Java ecosystem, but I've had
good luck with the `jvisualvm` tool that comes with Java SE 6. It's a nice
lightweight CPU and memory profiling tool that can attach to a running
process like Cassandra and dump stats in real time.
Mike
On Thu, Jun 10, 2010 at 7:39 PM, Jonathan Shook <js...@gmail.com> wrote:
> You are testing Cassandra in a way that it was not designed to be used.
> Bandwidth to disk is not a meaningful example for nearly anything
> except for filesystem benchmarking and things very nearly the same as
> filesystem benchmarking.
> Unless the usage patterns of your application match your test data,
> there is not a good reason to expect a strong correlation between this
> test and actual performance.
>
> Cassandra is not simply shuffling data through IO when you write.
> There are calculations that have to be done as writes filter their way
> through various stages of processing. The point of this is to minimize
> the overall effort Cassandra has to make in order to retrieve the data
> again. One example would be bloom filters. Each column that is written
> requires bloom filter processing and potentially auxiliary IO. Some of
> these steps are allowed to happen in the background, but if you try,
> you can cause them to stack up on top of the available CPU and memory
> resources.
>
> In such a case (continuous bulk writes), you are causing all of these
> costs to be taken in more of a synchronous (not delayed) fashion. You
> are not allowing the background processing that helps reduce client
> blocking (by deferring some processing) to do its magic.
>
>
>
> On Thu, Jun 10, 2010 at 7:42 PM, Rishi Bhardwaj <kh...@yahoo.com>
> wrote:
> > Hi
> > I am investigating Cassandra write performance and see very heavy CPU
> usage
> > from Cassandra. I have a single node Cassandra instance running on a dual
> > core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are
> being
> > generated from the same server using BatchMutate(). The client makes
> exactly
> > one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2 MB
> of
> > data and once it is acknowledged by Cassandra, the next RPC is done.
> > Cassandra has two separate disks, one for commitlog with a sequential b/w
> of
> > 130MBps and the other a solid state disk for data with b/w of 90MBps.
> Tuning
> > various parameters, I observe that I am able to attain a maximum write
> > performance of about 45 to 50 MBps from Cassandra. I see that the
> Cassandra
> > java process consistently uses 100% to 150% of CPU resources (as shown by
> > top) during the entire write operation. Also, iostat clearly shows that
> the
> > max disk bandwidth is not reached anytime during the write operation,
> every
> > now and then the i/o activity on "commitlog" disk and the data disk spike
> > but it is never consistently maintained by cassandra close to their
> peak. I
> > would imagine that the CPU is probably the bottleneck here. Does anyone
> have
> > any idea why Cassandra beats the heck out of the CPU here? Any
> suggestions
> > on how to go about finding the exact bottleneck here?
> > Some more information about the writes: I have 2 column families, the
> data
> > though is mostly written in one column family with column sizes of around
> > 32k and each row having around 256 or 512 columns. I would really
> appreciate
> > any help here.
> > Thanks,
> > Rishi
> >
> >
>
Re: Cassandra Write Performance, CPU usage
Posted by Jonathan Shook <js...@gmail.com>.
You are testing Cassandra in a way that it was not designed to be used.
Bandwidth to disk is not a meaningful example for nearly anything
except for filesystem benchmarking and things very nearly the same as
filesystem benchmarking.
Unless the usage patterns of your application match your test data,
there is not a good reason to expect a strong correlation between this
test and actual performance.
Cassandra is not simply shuffling data through IO when you write.
There are calculations that have to be done as writes filter their way
through various stages of processing. The point of this is to minimize
the overall effort Cassandra has to make in order to retrieve the data
again. One example would be bloom filters. Each column that is written
requires bloom filter processing and potentially auxiliary IO. Some of
these steps are allowed to happen in the background, but if you try,
you can cause them to stack up on top of the available CPU and memory
resources.
In such a case (continuous bulk writes), you are causing all of these
costs to be taken in more of a synchronous (not delayed) fashion. You
are not allowing the background processing that helps reduce client
blocking (by deferring some processing) to do its magic.
On Thu, Jun 10, 2010 at 7:42 PM, Rishi Bhardwaj <kh...@yahoo.com> wrote:
> Hi
> I am investigating Cassandra write performance and see very heavy CPU usage
> from Cassandra. I have a single node Cassandra instance running on a dual
> core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are being
> generated from the same server using BatchMutate(). The client makes exactly
> one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2 MB of
> data and once it is acknowledged by Cassandra, the next RPC is done.
> Cassandra has two separate disks, one for commitlog with a sequential b/w of
> 130MBps and the other a solid state disk for data with b/w of 90MBps. Tuning
> various parameters, I observe that I am able to attain a maximum write
> performance of about 45 to 50 MBps from Cassandra. I see that the Cassandra
> java process consistently uses 100% to 150% of CPU resources (as shown by
> top) during the entire write operation. Also, iostat clearly shows that the
> max disk bandwidth is not reached anytime during the write operation, every
> now and then the i/o activity on "commitlog" disk and the data disk spike
> but it is never consistently maintained by cassandra close to their peak. I
> would imagine that the CPU is probably the bottleneck here. Does anyone have
> any idea why Cassandra beats the heck out of the CPU here? Any suggestions
> on how to go about finding the exact bottleneck here?
> Some more information about the writes: I have 2 column families, the data
> though is mostly written in one column family with column sizes of around
> 32k and each row having around 256 or 512 columns. I would really appreciate
> any help here.
> Thanks,
> Rishi
>
>
Re: Cassandra Write Performance, CPU usage
Posted by vd <vi...@gmail.com>.
Hi Rishi
The writes in Cassandra are not directly written to the Disk, they are
stored in memory and later on flushed to the disk. May be thats why you are
not getting much out of iostat. Cant say about high cpu usage.
_______________________________________
Vineet Daniel
_______________________________________
Let your email find you....
On Fri, Jun 11, 2010 at 6:12 AM, Rishi Bhardwaj <kh...@yahoo.com>wrote:
> Hi
>
> I am investigating Cassandra write performance and see very heavy CPU usage
> from Cassandra. I have a single node Cassandra instance running on a dual
> core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are being
> generated from the same server using BatchMutate(). The client makes exactly
> one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2 MB of
> data and once it is acknowledged by Cassandra, the next RPC is done.
> Cassandra has two separate disks, one for commitlog with a sequential b/w of
> 130MBps and the other a solid state disk for data with b/w of 90MBps. Tuning
> various parameters, I observe that I am able to attain a maximum write
> performance of about 45 to 50 MBps from Cassandra. I see that the
> Cassandra java process consistently uses 100% to 150% of CPU resources (as
> shown by top) during the entire write operation. Also, iostat clearly shows
> that the max disk bandwidth is not reached anytime during the write
> operation, every now and then the i/o activity on "commitlog" disk and the
> data disk spike but it is never consistently maintained by cassandra close
> to their peak. I would imagine that the CPU is probably the bottleneck
> here. Does anyone have any idea why Cassandra beats the heck out of the CPU
> here? Any suggestions on how to go about finding the exact bottleneck here?
>
> Some more information about the writes: I have 2 column families, the data
> though is mostly written in one column family with column sizes of around
> 32k and each row having around 256 or 512 columns. I would really appreciate
> any help here.
>
> Thanks,
> Rishi
>
>
>