You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Pranay Agarwal <ag...@gmail.com> on 2015/03/20 18:57:19 UTC
Cassandra cluster Too high DISK IOs
Hi All.
I am using 15 nodes cassandra cluster(m3.2xlarge) with provisioned IOPS
disks (4000). I can see around 12k reads/sec ops on the cassandra cluster.
But I see around *~3500 read IOPS* on each of the cassandra nodes. Is that
normal?
I am using LevelledCompaction and I can see in the histograms that most
read requests are coming from 1/2 sstables only. Why are there so many disk
IOPS or is the normal for cassandra?
-Pranay
Re: Cassandra cluster Too high DISK IOs
Posted by Pranay Agarwal <ag...@gmail.com>.
No. as shown in the histograms, 99% of reads are using 2 or less number of
tables. What's typical usually? Can anyone share from experience?
On Fri, Mar 20, 2015 at 1:12 PM, Duncan Sands <du...@gmail.com>
wrote:
> On 20/03/15 19:34, Pranay Agarwal wrote:
>
>> The cluster is processing something like 12k reads and 2k writes/seconds.
>> The
>> disks are locally attached and latency is just fine. It's the number of
>> disk
>> iops that's too high.
>>
>
> Maybe each read is accessing many sstables.
>
> Ciao, Duncan.
>
Re: Cassandra cluster Too high DISK IOs
Posted by Duncan Sands <du...@gmail.com>.
On 20/03/15 19:34, Pranay Agarwal wrote:
> The cluster is processing something like 12k reads and 2k writes/seconds. The
> disks are locally attached and latency is just fine. It's the number of disk
> iops that's too high.
Maybe each read is accessing many sstables.
Ciao, Duncan.
Re: Cassandra cluster Too high DISK IOs
Posted by Pranay Agarwal <ag...@gmail.com>.
The cluster is processing something like 12k reads and 2k writes/seconds.
The disks are locally attached and latency is just fine. It's the number of
disk iops that's too high.
On Fri, Mar 20, 2015 at 11:05 AM, Ali Akhtar <al...@gmail.com> wrote:
> That probably depends on how many read / write queries your cluster is
> processing?
>
> Also, since you mentioned provisoned IOPS, are you using EBS for storing
> the data? If so, you probably want to switch to the ephemeral storage since
> its locally attached to the instance and doesn't require a network call for
> each lookup.
>
> On Fri, Mar 20, 2015 at 10:57 PM, Pranay Agarwal <agarwalpranaya@gmail.com
> > wrote:
>
>> Hi All.
>>
>>
>> I am using 15 nodes cassandra cluster(m3.2xlarge) with provisioned IOPS
>> disks (4000). I can see around 12k reads/sec ops on the cassandra cluster.
>>
>> But I see around *~3500 read IOPS* on each of the cassandra nodes. Is
>> that normal?
>>
>> I am using LevelledCompaction and I can see in the histograms that most
>> read requests are coming from 1/2 sstables only. Why are there so many disk
>> IOPS or is the normal for cassandra?
>>
>> -Pranay
>>
>
>
Re: Cassandra cluster Too high DISK IOs
Posted by Ali Akhtar <al...@gmail.com>.
That probably depends on how many read / write queries your cluster is
processing?
Also, since you mentioned provisoned IOPS, are you using EBS for storing
the data? If so, you probably want to switch to the ephemeral storage since
its locally attached to the instance and doesn't require a network call for
each lookup.
On Fri, Mar 20, 2015 at 10:57 PM, Pranay Agarwal <ag...@gmail.com>
wrote:
> Hi All.
>
>
> I am using 15 nodes cassandra cluster(m3.2xlarge) with provisioned IOPS
> disks (4000). I can see around 12k reads/sec ops on the cassandra cluster.
>
> But I see around *~3500 read IOPS* on each of the cassandra nodes. Is
> that normal?
>
> I am using LevelledCompaction and I can see in the histograms that most
> read requests are coming from 1/2 sstables only. Why are there so many disk
> IOPS or is the normal for cassandra?
>
> -Pranay
>