You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nitan Kainth <ni...@bamlabs.com> on 2017/05/16 15:53:12 UTC
Read timeouts
Hi,
We see read timeouts intermittently. Mostly after they have occurred. Timeouts are not consistent and does not occur in 100s at a moment.
1. Does read timeout considered as Dropped Mutation?
2. What is best way to nail down exact issue of scattered timeouts?
Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Read timeouts
Posted by Nitan Kainth <ni...@bamlabs.com>.
Thank you Jeff.
We are at Cassandra 3.0.10
Will look forward to upgrade or enable driver logging.
> On May 16, 2017, at 11:44 AM, Jeff Jirsa <jj...@apache.org> wrote:
>
>
>
> On 2017-05-16 08:53 (-0700), Nitan Kainth <ni...@bamlabs.com> wrote:
>> Hi,
>>
>> We see read timeouts intermittently. Mostly after they have occurred. Timeouts are not consistent and does not occur in 100s at a moment.
>>
>> 1. Does read timeout considered as Dropped Mutation?
>
> No, a dropped mutation is a failed write, not a failed read.
>
>> 2. What is best way to nail down exact issue of scattered timeouts?
>>
>
> First, be aware that tombstone overwhelming exceptions also get propagated as read timeouts - you should check your logs for warnings about tombstone problems.
>
> Second, you need to identify the slow queries somehow. You have a few options:
>
> 1) If you happen to be running 3.10 or newer , turn on the slow query log ( https://issues.apache.org/jira/browse/CASSANDRA-12403 ) . 3.10 is the newest release, and may not be fully stable, so you probably don't want to upgrade to 3.10 JUST to get this feature. But if you're already on that version, definitely use that tool.
>
> 2) Some drivers have a log-slow-queries feature. Consider turning that on, and let the application side log the slow queries. It's possible that you have a bad partition or two, and you may see patterns there.
>
> 3) Probabilistic tracing - you can tell cassandra to trace 1% of your queries, and hope you catch a timeout. It'll be unpleasant to track alone - this is really a last-resort type option, because you'll need to dig through that trace table to find the outliers after the fact.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
Re: Read timeouts
Posted by Jeff Jirsa <jj...@apache.org>.
On 2017-05-16 08:53 (-0700), Nitan Kainth <ni...@bamlabs.com> wrote:
> Hi,
>
> We see read timeouts intermittently. Mostly after they have occurred. Timeouts are not consistent and does not occur in 100s at a moment.
>
> 1. Does read timeout considered as Dropped Mutation?
No, a dropped mutation is a failed write, not a failed read.
> 2. What is best way to nail down exact issue of scattered timeouts?
>
First, be aware that tombstone overwhelming exceptions also get propagated as read timeouts - you should check your logs for warnings about tombstone problems.
Second, you need to identify the slow queries somehow. You have a few options:
1) If you happen to be running 3.10 or newer , turn on the slow query log ( https://issues.apache.org/jira/browse/CASSANDRA-12403 ) . 3.10 is the newest release, and may not be fully stable, so you probably don't want to upgrade to 3.10 JUST to get this feature. But if you're already on that version, definitely use that tool.
2) Some drivers have a log-slow-queries feature. Consider turning that on, and let the application side log the slow queries. It's possible that you have a bad partition or two, and you may see patterns there.
3) Probabilistic tracing - you can tell cassandra to trace 1% of your queries, and hope you catch a timeout. It'll be unpleasant to track alone - this is really a last-resort type option, because you'll need to dig through that trace table to find the outliers after the fact.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org