You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Arindam Barua <ab...@247-inc.com> on 2013/11/07 00:35:37 UTC
Config changes to leverage new hardware
We want to upgrade our Cassandra cluster to have newer hardware, and were wondering if anyone has suggestions on Cassandra or linux config changes that will prove to be beneficial.
As of now, our performance tests (our application specific as well as cassandra-stress) are not showing any significant difference in the hardwares, which is a little disheartening, since the new hardware has a lot more RAM and CPU.
Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk used for commitlog and 3 disks RAID 0 for data)
New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks ( 1 disk used for commitlog and 7 disks RAID 0 for data)
Most of the cassandra config currently is the default, and we are using LeveledCompaction strategy. Default key cache, row cache turned off.
The config we tried modifying so far was concurrent_reads to (16 * number of drives) and concurrent_writes to (8 * number of cores) as per recommendation in cassandra.yaml, but that didn't make much difference.
We were hoping that at least the extra RAM in the new hardware will be used for Linux file caching and hence an improvement in performance will be observed.
Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
Thanks,
Arindam
RE: Config changes to leverage new hardware
Posted by Arindam Barua <ab...@247-inc.com>.
Here are some calculated 'latency' results reported by cassandra-stress when asked to write 10M rows, i.e.
cassandra-stress -d <ip1>,<ip2> -n 10000000
(we actually had cassandra-stress running in deamon mode for the below tests)
avg_latency
(percentile)
90
99
99.9
99.99
Write: 8 cores, 32 GB, 3-disk RAID 0
0.002982182
0.003963931
0.004692996
0.004792326
Write: 32 cores, 128 GB, 7-disk RAID 0
0.003157515
0.003763181
0.005184429
0.005441946
Read: 8 cores, 32 GB, 3-disk RAID 0
0.002289879
0.057178021
0.173753058
0.24386912
Read: 32 cores, 128 GB, 7-disk RAID 0
0.002317525
0.010937648
0.013205977
0.014270511
The client was another node on the same network with the 8 core, 32 GB RAM specs. I wouldn't expect it to bottleneck, but I can monitor it while generating the load. In general, what would you expect it to bottleneck at?
>> Another interesting thing is that the linux disk cache doesn't seem to be growing in spite of a lot of free memory available.
>Things will only get paged in when they are accessed.
Hmm, interesting. I did a test where I just wrote large files to disk, eg.
dd if=/dev/zero of=bigfile18 bs=1M count=10000
and checked the disk cache, and it increased by exactly the same size of the file written (no reads were done in this case)
-----Original Message-----
From: Aaron Morton [mailto:aaron@thelastpickle.com]
Sent: Monday, November 25, 2013 11:55 AM
To: Cassandra User
Subject: Re: Config changes to leverage new hardware
> However, for both writes and reads there was virtually no difference in the latencies.
What sort of latency were you getting ?
> I'm still not very sure where the current *write* bottleneck is though.
What numbers are you getting ?
Could the bottle neck be the client ? Can it send writes fast enough to saturate the nodes ?
As a rule of thumb you should get 3,000 to 4,000 (non counter) writes per second per core.
> Sample iostat data (captured every 10s) for the dedicated disk where commit logs are written is below. Does this seem like a bottle neck?
Does not look too bad.
> Another interesting thing is that the linux disk cache doesn't seem to be growing in spite of a lot of free memory available.
Things will only get paged in when they are accessed.
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 21/11/2013, at 12:42 pm, Arindam Barua <ab...@247-inc.com>> wrote:
>
> Thanks for the suggestions Aaron.
>
> As a follow up, we ran a bunch of tests with different combinations of these changes on a 2-node ring. The load was generated using cassandra-stress, run with default values to write 30 million rows, and read them back.
> However, for both writes and reads there was virtually no difference in the latencies.
>
> The different combinations attempted:
> 1. Baseline test with none of the below changes.
> 2. Grabbing the TLAB setting from 1.2
> 3. Moving the commit logs too to the 7 disk RAID 0.
> 4. Increasing the concurrent_read to 32, and concurrent_write to 64
> 5. (3) + (4), i.e. moving commit logs to the RAID + increasing concurrent_read and concurrent_write config to 32 and 64.
>
> The write latencies were very similar, except them being ~3x worse for the 99.9th percentile and above for scenario (5) above.
> The read latencies were also similar, with (3) and (5) being a little worse for the 99.99th percentile.
>
> Overall, not making any changes, i.e. (1) performed as well or slightly better than any of the other changes.
>
> Running cassandra-stress on both the old and new hardware without making any config changes, the write performance was very similar, but the new hardware did show ~10x improvement in the read for the 99.9th percentile and higher. After thinking about this, the reason why we were not seeing any difference with our test framework was perhaps the nature of the test where we write the rows, and then do a bunch of reads to read the rows that were just written immediately following. The data is read back from the memtables, and never from the disk/sstables. Hence the new hardware's increased RAM and size of the disk cache or higher number of disks never helps.
>
> I'm still not very sure where the current *write* bottleneck is though. The new hardware has 32 cores vs 8 cores of the old hardware. Moving the commit log from a dedicated disk to a 7 RAID-0 disk system (where it would be shared by other data though) didn't make a difference too. (unless the extra contention on the RAID nullified the positive effects of the RAID).
>
> Sample iostat data (captured every 10s) for the dedicated disk where commit logs are written is below. Does this seem like a bottle neck? When the commit logs are written the await/svctm ratio is high.
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> 0.00 8.09 0.04 8.85 0.00 0.07 15.74 0.00 0.12 0.03 0.02
> 0.00 768.03 0.00 9.49 0.00 3.04 655.41 0.04 4.52 0.33 0.31
> 0.00 8.10 0.04 8.85 0.00 0.07 15.75 0.00 0.12 0.03 0.02
> 0.00 752.65 0.00 10.09 0.00 2.98 604.75 0.03 3.00 0.26 0.26
>
> Another interesting thing is that the linux disk cache doesn't seem to be growing in spite of a lot of free memory available. The total disk cache used reported by 'free' is less than the size of the sstables written with over 100 GB unused RAM.
> Even in production, where we have the older hardware running with 32 GB RAM for a long time now, looking at 5 hosts in 1 DC, only 2.5 GB to 8 GB was used for the disk cache. The Cassandra java process uses the 8 GB allocated to it, and at least 10-15 GB on all the hosts is not used at all.
>
> Thanks,
> Arindam
>
> From: Aaron Morton [mailto:aaron@thelastpickle.com]
> Sent: Wednesday, November 06, 2013 8:34 PM
> To: Cassandra User
> Subject: Re: Config changes to leverage new hardware
>
> Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
> You will make more use of the extra memory moving to 1.2 as it moves bloom filters and compression data off heap.
>
> Also grab the TLAB setting from cassandra-env.sh in v1.2
>
> As of now, our performance tests (our application specific as well as cassandra-stress) are not showing any significant difference in the hardwares, which is a little disheartening, since the new hardware has a lot more RAM and CPU.
> For reads or writes or both ?
>
> Writes tend to scale with cores as long as the commit log can keep up.
> Reads improve with disk IO and page cache size when the hot set is in memory.
>
> Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1
> disk used for commitlog and 3 disks RAID 0 for data) New Hardware: 32
> cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks ( 1 disk used for commitlog and 7 disks RAID 0 for data) Is the disk IO on the commit log volume keeping up ?
> You cranked up the concurrent writers and the commit log may not keep up. You could put the commit log on the same RAID volume to see if that improves writes.
>
> The config we tried modifying so far was concurrent_reads to (16 *
> number of drives) and concurrent_writes to (8 * number of cores) as
> per
> 256 write threads is a lot. Make sure the commit log can keep up, I would put it back to 32, maybe try 64. Not sure the concurrent list for the commit log will work well with that many threads.
>
> May want to put the reads down as well.
>
> It's easier to tune the system if you can provide some info on the workload.
>
> Cheers
>
> -----------------
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 7/11/2013, at 12:35 pm, Arindam Barua <ab...@247-inc.com>> wrote:
>
>
>
> We want to upgrade our Cassandra cluster to have newer hardware, and were wondering if anyone has suggestions on Cassandra or linux config changes that will prove to be beneficial.
> As of now, our performance tests (our application specific as well as cassandra-stress) are not showing any significant difference in the hardwares, which is a little disheartening, since the new hardware has a lot more RAM and CPU.
>
> Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1
> disk used for commitlog and 3 disks RAID 0 for data) New Hardware: 32
> cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks ( 1
> disk used for commitlog and 7 disks RAID 0 for data)
>
> Most of the cassandra config currently is the default, and we are using LeveledCompaction strategy. Default key cache, row cache turned off.
> The config we tried modifying so far was concurrent_reads to (16 * number of drives) and concurrent_writes to (8 * number of cores) as per recommendation in cassandra.yaml, but that didn't make much difference.
> We were hoping that at least the extra RAM in the new hardware will be used for Linux file caching and hence an improvement in performance will be observed.
>
> Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
>
> Thanks,
> Arindam
Re: Config changes to leverage new hardware
Posted by Aaron Morton <aa...@thelastpickle.com>.
> However, for both writes and reads there was virtually no difference in the latencies.
What sort of latency were you getting ?
> I’m still not very sure where the current *write* bottleneck is though.
What numbers are you getting ?
Could the bottle neck be the client ? Can it send writes fast enough to saturate the nodes ?
As a rule of thumb you should get 3,000 to 4,000 (non counter) writes per second per core.
> Sample iostat data (captured every 10s) for the dedicated disk where commit logs are written is below. Does this seem like a bottle neck?
Does not look too bad.
> Another interesting thing is that the linux disk cache doesn’t seem to be growing in spite of a lot of free memory available.
Things will only get paged in when they are accessed.
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 21/11/2013, at 12:42 pm, Arindam Barua <ab...@247-inc.com> wrote:
>
> Thanks for the suggestions Aaron.
>
> As a follow up, we ran a bunch of tests with different combinations of these changes on a 2-node ring. The load was generated using cassandra-stress, run with default values to write 30 million rows, and read them back.
> However, for both writes and reads there was virtually no difference in the latencies.
>
> The different combinations attempted:
> 1. Baseline test with none of the below changes.
> 2. Grabbing the TLAB setting from 1.2
> 3. Moving the commit logs too to the 7 disk RAID 0.
> 4. Increasing the concurrent_read to 32, and concurrent_write to 64
> 5. (3) + (4), i.e. moving commit logs to the RAID + increasing concurrent_read and concurrent_write config to 32 and 64.
>
> The write latencies were very similar, except them being ~3x worse for the 99.9th percentile and above for scenario (5) above.
> The read latencies were also similar, with (3) and (5) being a little worse for the 99.99th percentile.
>
> Overall, not making any changes, i.e. (1) performed as well or slightly better than any of the other changes.
>
> Running cassandra-stress on both the old and new hardware without making any config changes, the write performance was very similar, but the new hardware did show ~10x improvement in the read for the 99.9th percentile and higher. After thinking about this, the reason why we were not seeing any difference with our test framework was perhaps the nature of the test where we write the rows, and then do a bunch of reads to read the rows that were just written immediately following. The data is read back from the memtables, and never from the disk/sstables. Hence the new hardware’s increased RAM and size of the disk cache or higher number of disks never helps.
>
> I’m still not very sure where the current *write* bottleneck is though. The new hardware has 32 cores vs 8 cores of the old hardware. Moving the commit log from a dedicated disk to a 7 RAID-0 disk system (where it would be shared by other data though) didn’t make a difference too. (unless the extra contention on the RAID nullified the positive effects of the RAID).
>
> Sample iostat data (captured every 10s) for the dedicated disk where commit logs are written is below. Does this seem like a bottle neck? When the commit logs are written the await/svctm ratio is high.
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> 0.00 8.09 0.04 8.85 0.00 0.07 15.74 0.00 0.12 0.03 0.02
> 0.00 768.03 0.00 9.49 0.00 3.04 655.41 0.04 4.52 0.33 0.31
> 0.00 8.10 0.04 8.85 0.00 0.07 15.75 0.00 0.12 0.03 0.02
> 0.00 752.65 0.00 10.09 0.00 2.98 604.75 0.03 3.00 0.26 0.26
>
> Another interesting thing is that the linux disk cache doesn’t seem to be growing in spite of a lot of free memory available. The total disk cache used reported by ‘free’ is less than the size of the sstables written with over 100 GB unused RAM.
> Even in production, where we have the older hardware running with 32 GB RAM for a long time now, looking at 5 hosts in 1 DC, only 2.5 GB to 8 GB was used for the disk cache. The Cassandra java process uses the 8 GB allocated to it, and at least 10-15 GB on all the hosts is not used at all.
>
> Thanks,
> Arindam
>
> From: Aaron Morton [mailto:aaron@thelastpickle.com]
> Sent: Wednesday, November 06, 2013 8:34 PM
> To: Cassandra User
> Subject: Re: Config changes to leverage new hardware
>
> Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
> You will make more use of the extra memory moving to 1.2 as it moves bloom filters and compression data off heap.
>
> Also grab the TLAB setting from cassandra-env.sh in v1.2
>
> As of now, our performance tests (our application specific as well as cassandra-stress) are not showing any significant difference in the hardwares, which is a little disheartening, since the new hardware has a lot more RAM and CPU.
> For reads or writes or both ?
>
> Writes tend to scale with cores as long as the commit log can keep up.
> Reads improve with disk IO and page cache size when the hot set is in memory.
>
> Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk used for commitlog and 3 disks RAID 0 for data)
> New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks ( 1 disk used for commitlog and 7 disks RAID 0 for data)
> Is the disk IO on the commit log volume keeping up ?
> You cranked up the concurrent writers and the commit log may not keep up. You could put the commit log on the same RAID volume to see if that improves writes.
>
> The config we tried modifying so far was concurrent_reads to (16 * number of drives) and concurrent_writes to (8 * number of cores) as per
> 256 write threads is a lot. Make sure the commit log can keep up, I would put it back to 32, maybe try 64. Not sure the concurrent list for the commit log will work well with that many threads.
>
> May want to put the reads down as well.
>
> It’s easier to tune the system if you can provide some info on the workload.
>
> Cheers
>
> -----------------
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 7/11/2013, at 12:35 pm, Arindam Barua <ab...@247-inc.com> wrote:
>
>
>
> We want to upgrade our Cassandra cluster to have newer hardware, and were wondering if anyone has suggestions on Cassandra or linux config changes that will prove to be beneficial.
> As of now, our performance tests (our application specific as well as cassandra-stress) are not showing any significant difference in the hardwares, which is a little disheartening, since the new hardware has a lot more RAM and CPU.
>
> Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk used for commitlog and 3 disks RAID 0 for data)
> New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks ( 1 disk used for commitlog and 7 disks RAID 0 for data)
>
> Most of the cassandra config currently is the default, and we are using LeveledCompaction strategy. Default key cache, row cache turned off.
> The config we tried modifying so far was concurrent_reads to (16 * number of drives) and concurrent_writes to (8 * number of cores) as per recommendation in cassandra.yaml, but that didn’t make much difference.
> We were hoping that at least the extra RAM in the new hardware will be used for Linux file caching and hence an improvement in performance will be observed.
>
> Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
>
> Thanks,
> Arindam
RE: Config changes to leverage new hardware
Posted by Arindam Barua <ab...@247-inc.com>.
Thanks for the suggestions Aaron.
As a follow up, we ran a bunch of tests with different combinations of these changes on a 2-node ring. The load was generated using cassandra-stress, run with default values to write 30 million rows, and read them back.
However, for both writes and reads there was virtually no difference in the latencies.
The different combinations attempted:
1. Baseline test with none of the below changes.
2. Grabbing the TLAB setting from 1.2
3. Moving the commit logs too to the 7 disk RAID 0.
4. Increasing the concurrent_read to 32, and concurrent_write to 64
5. (3) + (4), i.e. moving commit logs to the RAID + increasing concurrent_read and concurrent_write config to 32 and 64.
The write latencies were very similar, except them being ~3x worse for the 99.9th percentile and above for scenario (5) above.
The read latencies were also similar, with (3) and (5) being a little worse for the 99.99th percentile.
Overall, not making any changes, i.e. (1) performed as well or slightly better than any of the other changes.
Running cassandra-stress on both the old and new hardware without making any config changes, the write performance was very similar, but the new hardware did show ~10x improvement in the read for the 99.9th percentile and higher. After thinking about this, the reason why we were not seeing any difference with our test framework was perhaps the nature of the test where we write the rows, and then do a bunch of reads to read the rows that were just written immediately following. The data is read back from the memtables, and never from the disk/sstables. Hence the new hardware's increased RAM and size of the disk cache or higher number of disks never helps.
I'm still not very sure where the current *write* bottleneck is though. The new hardware has 32 cores vs 8 cores of the old hardware. Moving the commit log from a dedicated disk to a 7 RAID-0 disk system (where it would be shared by other data though) didn't make a difference too. (unless the extra contention on the RAID nullified the positive effects of the RAID).
Sample iostat data (captured every 10s) for the dedicated disk where commit logs are written is below. Does this seem like a bottle neck? When the commit logs are written the await/svctm ratio is high.
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
0.00 8.09 0.04 8.85 0.00 0.07 15.74 0.00 0.12 0.03 0.02
0.00 768.03 0.00 9.49 0.00 3.04 655.41 0.04 4.52 0.33 0.31
0.00 8.10 0.04 8.85 0.00 0.07 15.75 0.00 0.12 0.03 0.02
0.00 752.65 0.00 10.09 0.00 2.98 604.75 0.03 3.00 0.26 0.26
Another interesting thing is that the linux disk cache doesn't seem to be growing in spite of a lot of free memory available. The total disk cache used reported by 'free' is less than the size of the sstables written with over 100 GB unused RAM.
Even in production, where we have the older hardware running with 32 GB RAM for a long time now, looking at 5 hosts in 1 DC, only 2.5 GB to 8 GB was used for the disk cache. The Cassandra java process uses the 8 GB allocated to it, and at least 10-15 GB on all the hosts is not used at all.
Thanks,
Arindam
From: Aaron Morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, November 06, 2013 8:34 PM
To: Cassandra User
Subject: Re: Config changes to leverage new hardware
Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
You will make more use of the extra memory moving to 1.2 as it moves bloom filters and compression data off heap.
Also grab the TLAB setting from cassandra-env.sh in v1.2
As of now, our performance tests (our application specific as well as cassandra-stress) are not showing any significant difference in the hardwares, which is a little disheartening, since the new hardware has a lot more RAM and CPU.
For reads or writes or both ?
Writes tend to scale with cores as long as the commit log can keep up.
Reads improve with disk IO and page cache size when the hot set is in memory.
Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk used for commitlog and 3 disks RAID 0 for data)
New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks ( 1 disk used for commitlog and 7 disks RAID 0 for data)
Is the disk IO on the commit log volume keeping up ?
You cranked up the concurrent writers and the commit log may not keep up. You could put the commit log on the same RAID volume to see if that improves writes.
The config we tried modifying so far was concurrent_reads to (16 * number of drives) and concurrent_writes to (8 * number of cores) as per
256 write threads is a lot. Make sure the commit log can keep up, I would put it back to 32, maybe try 64. Not sure the concurrent list for the commit log will work well with that many threads.
May want to put the reads down as well.
It's easier to tune the system if you can provide some info on the workload.
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 7/11/2013, at 12:35 pm, Arindam Barua <ab...@247-inc.com>> wrote:
We want to upgrade our Cassandra cluster to have newer hardware, and were wondering if anyone has suggestions on Cassandra or linux config changes that will prove to be beneficial.
As of now, our performance tests (our application specific as well as cassandra-stress) are not showing any significant difference in the hardwares, which is a little disheartening, since the new hardware has a lot more RAM and CPU.
Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk used for commitlog and 3 disks RAID 0 for data)
New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks ( 1 disk used for commitlog and 7 disks RAID 0 for data)
Most of the cassandra config currently is the default, and we are using LeveledCompaction strategy. Default key cache, row cache turned off.
The config we tried modifying so far was concurrent_reads to (16 * number of drives) and concurrent_writes to (8 * number of cores) as per recommendation in cassandra.yaml, but that didn't make much difference.
We were hoping that at least the extra RAM in the new hardware will be used for Linux file caching and hence an improvement in performance will be observed.
Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
Thanks,
Arindam
Re: Config changes to leverage new hardware
Posted by Aaron Morton <aa...@thelastpickle.com>.
> Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
You will make more use of the extra memory moving to 1.2 as it moves bloom filters and compression data off heap.
Also grab the TLAB setting from cassandra-env.sh in v1.2
> As of now, our performance tests (our application specific as well as cassandra-stress) are not showing any significant difference in the hardwares, which is a little disheartening, since the new hardware has a lot more RAM and CPU.
For reads or writes or both ?
Writes tend to scale with cores as long as the commit log can keep up.
Reads improve with disk IO and page cache size when the hot set is in memory.
> Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk used for commitlog and 3 disks RAID 0 for data)
> New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks ( 1 disk used for commitlog and 7 disks RAID 0 for data)
Is the disk IO on the commit log volume keeping up ?
You cranked up the concurrent writers and the commit log may not keep up. You could put the commit log on the same RAID volume to see if that improves writes.
> The config we tried modifying so far was concurrent_reads to (16 * number of drives) and concurrent_writes to (8 * number of cores) as per
256 write threads is a lot. Make sure the commit log can keep up, I would put it back to 32, maybe try 64. Not sure the concurrent list for the commit log will work well with that many threads.
May want to put the reads down as well.
It’s easier to tune the system if you can provide some info on the workload.
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 7/11/2013, at 12:35 pm, Arindam Barua <ab...@247-inc.com> wrote:
>
> We want to upgrade our Cassandra cluster to have newer hardware, and were wondering if anyone has suggestions on Cassandra or linux config changes that will prove to be beneficial.
> As of now, our performance tests (our application specific as well as cassandra-stress) are not showing any significant difference in the hardwares, which is a little disheartening, since the new hardware has a lot more RAM and CPU.
>
> Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk used for commitlog and 3 disks RAID 0 for data)
> New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks ( 1 disk used for commitlog and 7 disks RAID 0 for data)
>
> Most of the cassandra config currently is the default, and we are using LeveledCompaction strategy. Default key cache, row cache turned off.
> The config we tried modifying so far was concurrent_reads to (16 * number of drives) and concurrent_writes to (8 * number of cores) as per recommendation in cassandra.yaml, but that didn’t make much difference.
> We were hoping that at least the extra RAM in the new hardware will be used for Linux file caching and hence an improvement in performance will be observed.
>
> Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
>
> Thanks,
> Arindam