How can OpsCenter show me Read Request Latency where there are no read requests??

[Brian Tarbox <ta...@cabotresearch.com>]

I am making heavy use of DataStax OpsCenter to help tune my system and its
great.

And yet puzzling.  I see my clients do a burst of Reads causing the
OpsCenter Read Requests chart to go up and stay up until the clients finish
doing their reads.  The read request latency chart also goes up....but it
stays up even after all the reads are done.  At last glance I've had next
to zero reads for 10 minutes but still have a read request latency thats
basically unchanged from when there were actual reads.

How am I to interpret this?

Thanks.

Brian Tarbox

Re: How can OpsCenter show me Read Request Latency where there are no read requests??

[Tyler Hobbs <ty...@datastax.com>]

A few milliseconds (or a few thousand usecs) isn't terribly high,
considering that number includes at least one round trip between nodes.
I'm not sure about the tracking behavior that you're describing -- could
you provide some more details or perhaps screenshots?


On Wed, Jan 16, 2013 at 12:16 PM, Brian Tarbox <ta...@cabotresearch.com>wrote:

> Hmm, that's sense but then why is the latency for the reads that get the
> metric often so high (several thousand uSecs) and why does it so closely
> track the latency of my normal reads?
>
>
> On Wed, Jan 16, 2013 at 12:14 PM, Tyler Hobbs <ty...@datastax.com> wrote:
>
>> When you view OpsCenter metrics, you're generating a small number of
>> reads to fetch the metric data, which is why your read count is near zero
>> instead of actually being zero.  Since reads are still occurring, Cassandra
>> will continue to show a read latency.  Basically, you're just viewing the
>> latency on the reads to fetch metric data.
>>
>> Normally the number of reads required to view metrics are small enough
>> that they only make a minor difference in your overall read latency
>> average, but when you have no other reads occurring, they're the only reads
>> that are included in the average.
>>
>>
>> On Tue, Jan 15, 2013 at 9:28 PM, Brian Tarbox <ta...@cabotresearch.com>wrote:
>>
>>> I am making heavy use of DataStax OpsCenter to help tune my system and
>>> its great.
>>>
>>> And yet puzzling.  I see my clients do a burst of Reads causing the
>>> OpsCenter Read Requests chart to go up and stay up until the clients finish
>>> doing their reads.  The read request latency chart also goes up....but it
>>> stays up even after all the reads are done.  At last glance I've had next
>>> to zero reads for 10 minutes but still have a read request latency thats
>>> basically unchanged from when there were actual reads.
>>>
>>> How am I to interpret this?
>>>
>>> Thanks.
>>>
>>> Brian Tarbox
>>>
>>
>>
>>
>> --
>> Tyler Hobbs
>> DataStax <http://datastax.com/>
>>
>
>


-- 
Tyler Hobbs
DataStax <http://datastax.com/>

Re: How can OpsCenter show me Read Request Latency where there are no read requests??

[Brian Tarbox <ta...@cabotresearch.com>]

Hmm, that's sense but then why is the latency for the reads that get the
metric often so high (several thousand uSecs) and why does it so closely
track the latency of my normal reads?


On Wed, Jan 16, 2013 at 12:14 PM, Tyler Hobbs <ty...@datastax.com> wrote:

> When you view OpsCenter metrics, you're generating a small number of reads
> to fetch the metric data, which is why your read count is near zero instead
> of actually being zero.  Since reads are still occurring, Cassandra will
> continue to show a read latency.  Basically, you're just viewing the
> latency on the reads to fetch metric data.
>
> Normally the number of reads required to view metrics are small enough
> that they only make a minor difference in your overall read latency
> average, but when you have no other reads occurring, they're the only reads
> that are included in the average.
>
>
> On Tue, Jan 15, 2013 at 9:28 PM, Brian Tarbox <ta...@cabotresearch.com>wrote:
>
>> I am making heavy use of DataStax OpsCenter to help tune my system and
>> its great.
>>
>> And yet puzzling.  I see my clients do a burst of Reads causing the
>> OpsCenter Read Requests chart to go up and stay up until the clients finish
>> doing their reads.  The read request latency chart also goes up....but it
>> stays up even after all the reads are done.  At last glance I've had next
>> to zero reads for 10 minutes but still have a read request latency thats
>> basically unchanged from when there were actual reads.
>>
>> How am I to interpret this?
>>
>> Thanks.
>>
>> Brian Tarbox
>>
>
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>
>

Re: How can OpsCenter show me Read Request Latency where there are no read requests??

[Tyler Hobbs <ty...@datastax.com>]

When you view OpsCenter metrics, you're generating a small number of reads
to fetch the metric data, which is why your read count is near zero instead
of actually being zero.  Since reads are still occurring, Cassandra will
continue to show a read latency.  Basically, you're just viewing the
latency on the reads to fetch metric data.

Normally the number of reads required to view metrics are small enough that
they only make a minor difference in your overall read latency average, but
when you have no other reads occurring, they're the only reads that are
included in the average.


On Tue, Jan 15, 2013 at 9:28 PM, Brian Tarbox <ta...@cabotresearch.com>wrote:

> I am making heavy use of DataStax OpsCenter to help tune my system and its
> great.
>
> And yet puzzling.  I see my clients do a burst of Reads causing the
> OpsCenter Read Requests chart to go up and stay up until the clients finish
> doing their reads.  The read request latency chart also goes up....but it
> stays up even after all the reads are done.  At last glance I've had next
> to zero reads for 10 minutes but still have a read request latency thats
> basically unchanged from when there were actual reads.
>
> How am I to interpret this?
>
> Thanks.
>
> Brian Tarbox
>



-- 
Tyler Hobbs
DataStax <http://datastax.com/>

Re: How can OpsCenter show me Read Request Latency where there are no read requests??

[Mikhail Panchenko <m...@mihasya.com>]

I haven't used OpsCenter specifically, so this is a guess: often latency
metrics are based on the last N samples and what is graphed are percentiles
(as opposed to a sliding time window). That means that the graph will
remain the same until more requests occur, as the data that the percentiles
are calculated from is not changing (i.e. there are no new samples of
latency being added). If that's not it, then I don't know :D


On Tue, Jan 15, 2013 at 7:28 PM, Brian Tarbox <ta...@cabotresearch.com>wrote:

> I am making heavy use of DataStax OpsCenter to help tune my system and its
> great.
>
> And yet puzzling.  I see my clients do a burst of Reads causing the
> OpsCenter Read Requests chart to go up and stay up until the clients finish
> doing their reads.  The read request latency chart also goes up....but it
> stays up even after all the reads are done.  At last glance I've had next
> to zero reads for 10 minutes but still have a read request latency thats
> basically unchanged from when there were actual reads.
>
> How am I to interpret this?
>
> Thanks.
>
> Brian Tarbox
>