You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Soerian Lieve <sl...@liveramp.com> on 2015/07/23 23:51:53 UTC
Reduced write performance when reading
Hi,
I am currently performing benchmarks on Cassandra. Independently from each
other I am seeing ~100k writes/sec and ~50k reads/sec. When I read and
write at the same time, writing drops down to ~1000 writes/sec and reading
stays roughly the same.
The heap used is the same as when only reading, as is the disk utilization.
Replication factor is 3, consistency level on both reads and writes is ONE.
Using Cassandra 2.1.6. All cassandra.yaml settings set up according to the
Datastax guide. All nodes are running on SSDs.
Any ideas what could cause this?
Thanks,
Soerian
Re: Reduced write performance when reading
Posted by Soerian Lieve <sl...@liveramp.com>.
Found the problem, it turns out that what Bharatendra suggested was
correct.
I had set the memtable_flush_writers to equal the number of cores but
hadn't restarted the Cassandra process, so they didn't take the
configuration.
On Wed, Jul 29, 2015 at 12:59 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Tue, Jul 28, 2015 at 4:49 PM, Soerian Lieve <sl...@liveramp.com>
> wrote:
>
>> I did already set that to the number of cores of the machines (24), but
>> it made no difference.
>>
>
> I continue to suggest that you file a JIRA ticket... I feel you have done
> sufficient community based due dilligence to question whether this
> performance is expected on the server side. You have a repro path to
> illustrate the behavior. Why not file a JIRA?
>
> (and respond on thread with its url...)
>
> =Rob
>
>
Re: Reduced write performance when reading
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Jul 28, 2015 at 4:49 PM, Soerian Lieve <sl...@liveramp.com> wrote:
> I did already set that to the number of cores of the machines (24), but it
> made no difference.
>
I continue to suggest that you file a JIRA ticket... I feel you have done
sufficient community based due dilligence to question whether this
performance is expected on the server side. You have a repro path to
illustrate the behavior. Why not file a JIRA?
(and respond on thread with its url...)
=Rob
Re: Reduced write performance when reading
Posted by Soerian Lieve <sl...@liveramp.com>.
I did already set that to the number of cores of the machines (24), but it
made no difference.
On Tue, Jul 28, 2015 at 4:44 PM, Bharatendra Boddu <bh...@gmail.com>
wrote:
> Increase memtable_flush_writers. In cassandra.yaml, it was recommended to
> increase this setting when SSDs used for storing data.
>
> On Fri, Jul 24, 2015 at 1:55 PM, Soerian Lieve <sl...@liveramp.com>
> wrote:
>
>> I was on CFQ so I changed it to noop. The problem still persisted
>> however. Do you have any other ideas?
>>
>> On Thu, Jul 23, 2015 at 5:00 PM, Jeff Ferland <jb...@tubularlabs.com>
>> wrote:
>>
>>> Imbalanced disk use is ok in itself. It’s only saturated throughput
>>> that’s harmful. RAID 0 does give more consistent throughput and balancing,
>>> but that’s another story.
>>>
>>> As for your situation with SSD drive, you can probably tweak this by
>>> changing the scheduler is set to noop, or read up on
>>> https://www.kernel.org/doc/Documentation/block/deadline-iosched.txt for
>>> the deadline scheduler (lower writes_starved value). If you’re one CFQ,
>>> definitely ditch it.
>>>
>>> -Jeff
>>>
>>> On Jul 23, 2015, at 4:17 PM, Soerian Lieve <sl...@liveramp.com> wrote:
>>>
>>> I set up RAID0 after experiencing highly imbalanced disk usage with a
>>> JBOD setup so my transaction logs are indeed on the same media as the
>>> sstables.
>>> Is there any alternative to setting up RAID0 that doesn't have this
>>> issue?
>>>
>>> On Thu, Jul 23, 2015 at 4:03 PM, Jeff Ferland <jb...@tubularlabs.com>
>>> wrote:
>>>
>>>> My immediate guess: your transaction logs are on the same media as your
>>>> sstables and your OS prioritizes read requests.
>>>>
>>>> -Jeff
>>>>
>>>> > On Jul 23, 2015, at 2:51 PM, Soerian Lieve <sl...@liveramp.com>
>>>> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > I am currently performing benchmarks on Cassandra. Independently from
>>>> each other I am seeing ~100k writes/sec and ~50k reads/sec. When I read and
>>>> write at the same time, writing drops down to ~1000 writes/sec and reading
>>>> stays roughly the same.
>>>> >
>>>> > The heap used is the same as when only reading, as is the disk
>>>> utilization. Replication factor is 3, consistency level on both reads and
>>>> writes is ONE. Using Cassandra 2.1.6. All cassandra.yaml settings set up
>>>> according to the Datastax guide. All nodes are running on SSDs.
>>>> >
>>>> > Any ideas what could cause this?
>>>> >
>>>> > Thanks,
>>>> > Soerian
>>>>
>>>>
>>>
>>>
>>
>
Re: Reduced write performance when reading
Posted by Bharatendra Boddu <bh...@gmail.com>.
Increase memtable_flush_writers. In cassandra.yaml, it was recommended to
increase this setting when SSDs used for storing data.
On Fri, Jul 24, 2015 at 1:55 PM, Soerian Lieve <sl...@liveramp.com> wrote:
> I was on CFQ so I changed it to noop. The problem still persisted however.
> Do you have any other ideas?
>
> On Thu, Jul 23, 2015 at 5:00 PM, Jeff Ferland <jb...@tubularlabs.com> wrote:
>
>> Imbalanced disk use is ok in itself. It’s only saturated throughput
>> that’s harmful. RAID 0 does give more consistent throughput and balancing,
>> but that’s another story.
>>
>> As for your situation with SSD drive, you can probably tweak this by
>> changing the scheduler is set to noop, or read up on
>> https://www.kernel.org/doc/Documentation/block/deadline-iosched.txt for
>> the deadline scheduler (lower writes_starved value). If you’re one CFQ,
>> definitely ditch it.
>>
>> -Jeff
>>
>> On Jul 23, 2015, at 4:17 PM, Soerian Lieve <sl...@liveramp.com> wrote:
>>
>> I set up RAID0 after experiencing highly imbalanced disk usage with a
>> JBOD setup so my transaction logs are indeed on the same media as the
>> sstables.
>> Is there any alternative to setting up RAID0 that doesn't have this issue?
>>
>> On Thu, Jul 23, 2015 at 4:03 PM, Jeff Ferland <jb...@tubularlabs.com>
>> wrote:
>>
>>> My immediate guess: your transaction logs are on the same media as your
>>> sstables and your OS prioritizes read requests.
>>>
>>> -Jeff
>>>
>>> > On Jul 23, 2015, at 2:51 PM, Soerian Lieve <sl...@liveramp.com>
>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I am currently performing benchmarks on Cassandra. Independently from
>>> each other I am seeing ~100k writes/sec and ~50k reads/sec. When I read and
>>> write at the same time, writing drops down to ~1000 writes/sec and reading
>>> stays roughly the same.
>>> >
>>> > The heap used is the same as when only reading, as is the disk
>>> utilization. Replication factor is 3, consistency level on both reads and
>>> writes is ONE. Using Cassandra 2.1.6. All cassandra.yaml settings set up
>>> according to the Datastax guide. All nodes are running on SSDs.
>>> >
>>> > Any ideas what could cause this?
>>> >
>>> > Thanks,
>>> > Soerian
>>>
>>>
>>
>>
>
Re: Reduced write performance when reading
Posted by Soerian Lieve <sl...@liveramp.com>.
I was on CFQ so I changed it to noop. The problem still persisted however.
Do you have any other ideas?
On Thu, Jul 23, 2015 at 5:00 PM, Jeff Ferland <jb...@tubularlabs.com> wrote:
> Imbalanced disk use is ok in itself. It’s only saturated throughput that’s
> harmful. RAID 0 does give more consistent throughput and balancing, but
> that’s another story.
>
> As for your situation with SSD drive, you can probably tweak this by
> changing the scheduler is set to noop, or read up on
> https://www.kernel.org/doc/Documentation/block/deadline-iosched.txt for
> the deadline scheduler (lower writes_starved value). If you’re one CFQ,
> definitely ditch it.
>
> -Jeff
>
> On Jul 23, 2015, at 4:17 PM, Soerian Lieve <sl...@liveramp.com> wrote:
>
> I set up RAID0 after experiencing highly imbalanced disk usage with a JBOD
> setup so my transaction logs are indeed on the same media as the sstables.
> Is there any alternative to setting up RAID0 that doesn't have this issue?
>
> On Thu, Jul 23, 2015 at 4:03 PM, Jeff Ferland <jb...@tubularlabs.com> wrote:
>
>> My immediate guess: your transaction logs are on the same media as your
>> sstables and your OS prioritizes read requests.
>>
>> -Jeff
>>
>> > On Jul 23, 2015, at 2:51 PM, Soerian Lieve <sl...@liveramp.com> wrote:
>> >
>> > Hi,
>> >
>> > I am currently performing benchmarks on Cassandra. Independently from
>> each other I am seeing ~100k writes/sec and ~50k reads/sec. When I read and
>> write at the same time, writing drops down to ~1000 writes/sec and reading
>> stays roughly the same.
>> >
>> > The heap used is the same as when only reading, as is the disk
>> utilization. Replication factor is 3, consistency level on both reads and
>> writes is ONE. Using Cassandra 2.1.6. All cassandra.yaml settings set up
>> according to the Datastax guide. All nodes are running on SSDs.
>> >
>> > Any ideas what could cause this?
>> >
>> > Thanks,
>> > Soerian
>>
>>
>
>
Re: Reduced write performance when reading
Posted by Jeff Ferland <jb...@tubularlabs.com>.
Imbalanced disk use is ok in itself. It’s only saturated throughput that’s harmful. RAID 0 does give more consistent throughput and balancing, but that’s another story.
As for your situation with SSD drive, you can probably tweak this by changing the scheduler is set to noop, or read up on https://www.kernel.org/doc/Documentation/block/deadline-iosched.txt <https://www.kernel.org/doc/Documentation/block/deadline-iosched.txt> for the deadline scheduler (lower writes_starved value). If you’re one CFQ, definitely ditch it.
-Jeff
> On Jul 23, 2015, at 4:17 PM, Soerian Lieve <sl...@liveramp.com> wrote:
>
> I set up RAID0 after experiencing highly imbalanced disk usage with a JBOD setup so my transaction logs are indeed on the same media as the sstables.
> Is there any alternative to setting up RAID0 that doesn't have this issue?
>
> On Thu, Jul 23, 2015 at 4:03 PM, Jeff Ferland <jbf@tubularlabs.com <ma...@tubularlabs.com>> wrote:
> My immediate guess: your transaction logs are on the same media as your sstables and your OS prioritizes read requests.
>
> -Jeff
>
> > On Jul 23, 2015, at 2:51 PM, Soerian Lieve <slieve@liveramp.com <ma...@liveramp.com>> wrote:
> >
> > Hi,
> >
> > I am currently performing benchmarks on Cassandra. Independently from each other I am seeing ~100k writes/sec and ~50k reads/sec. When I read and write at the same time, writing drops down to ~1000 writes/sec and reading stays roughly the same.
> >
> > The heap used is the same as when only reading, as is the disk utilization. Replication factor is 3, consistency level on both reads and writes is ONE. Using Cassandra 2.1.6. All cassandra.yaml settings set up according to the Datastax guide. All nodes are running on SSDs.
> >
> > Any ideas what could cause this?
> >
> > Thanks,
> > Soerian
>
>
Re: Reduced write performance when reading
Posted by Soerian Lieve <sl...@liveramp.com>.
I set up RAID0 after experiencing highly imbalanced disk usage with a JBOD
setup so my transaction logs are indeed on the same media as the sstables.
Is there any alternative to setting up RAID0 that doesn't have this issue?
On Thu, Jul 23, 2015 at 4:03 PM, Jeff Ferland <jb...@tubularlabs.com> wrote:
> My immediate guess: your transaction logs are on the same media as your
> sstables and your OS prioritizes read requests.
>
> -Jeff
>
> > On Jul 23, 2015, at 2:51 PM, Soerian Lieve <sl...@liveramp.com> wrote:
> >
> > Hi,
> >
> > I am currently performing benchmarks on Cassandra. Independently from
> each other I am seeing ~100k writes/sec and ~50k reads/sec. When I read and
> write at the same time, writing drops down to ~1000 writes/sec and reading
> stays roughly the same.
> >
> > The heap used is the same as when only reading, as is the disk
> utilization. Replication factor is 3, consistency level on both reads and
> writes is ONE. Using Cassandra 2.1.6. All cassandra.yaml settings set up
> according to the Datastax guide. All nodes are running on SSDs.
> >
> > Any ideas what could cause this?
> >
> > Thanks,
> > Soerian
>
>
Re: Reduced write performance when reading
Posted by Jeff Ferland <jb...@tubularlabs.com>.
My immediate guess: your transaction logs are on the same media as your sstables and your OS prioritizes read requests.
-Jeff
> On Jul 23, 2015, at 2:51 PM, Soerian Lieve <sl...@liveramp.com> wrote:
>
> Hi,
>
> I am currently performing benchmarks on Cassandra. Independently from each other I am seeing ~100k writes/sec and ~50k reads/sec. When I read and write at the same time, writing drops down to ~1000 writes/sec and reading stays roughly the same.
>
> The heap used is the same as when only reading, as is the disk utilization. Replication factor is 3, consistency level on both reads and writes is ONE. Using Cassandra 2.1.6. All cassandra.yaml settings set up according to the Datastax guide. All nodes are running on SSDs.
>
> Any ideas what could cause this?
>
> Thanks,
> Soerian