You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by S G <sg...@gmail.com> on 2021/10/12 16:52:33 UTC
Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Hello,
Is there a way to stop long running queries in Cassandra (versions 3.11.x
or 4.x) ?
The use-case is to have some kind of a circuit breaker based on query-time
that has exceeded the client's SLAs.
Example: If server response is useless to the client after 10 ms, then we
could
have a *query_killing_timeout* set to 15 ms (where additional 5ms allows
for some buffer).
And when that much time has elapsed, Cassandra will kill the query
execution automatically.
If this is not possible in Cassandra currently, any chance we can do it
outside of Cassandra, like
a shell script that monitors such long running queries (through users table
etc) and kills the
OS-thread responsible for that query (Looks unsafe though as that might
leave the DB in an inconsistent state) ?
We are trying this as a proactive measure to safeguard our clusters from
any rogue queries fired accidentally or maliciously.
Thanks !
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by S G <sg...@gmail.com>.
I see. Thanks Jeff !
On Wed, Oct 13, 2021 at 2:25 PM Jeff Jirsa <jj...@gmail.com> wrote:
> Convention in the yaml is default being visible commented out.
>
>
> On Wed, Oct 13, 2021 at 2:17 PM S G <sg...@gmail.com> wrote:
>
>> ok, the link given has the value commented, so I was a bit confused.
>> But then https://github.com/apache/cassandra/search?q=cross_node_timeout
>> shows that default value is indeed true.
>> Thanks for the help,
>>
>> On Wed, Oct 13, 2021 at 11:26 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>> The default is true:
>>>
>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1000
>>>
>>> There is no equivalent to `alter system kill session`, because it is
>>> assumed that any query has a short, finite life in the order of seconds.
>>>
>>>
>>>
>>> On Wed, Oct 13, 2021 at 11:10 AM S G <sg...@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Does anyone know about the default being turned off for this setting?
>>>> It seems like a good one to be turned on - why have replicas process
>>>> something for which coordinator has already sent the timeout to client?
>>>>
>>>> Thanks
>>>>
>>>> On Tue, Oct 12, 2021 at 11:06 AM S G <sg...@gmail.com> wrote:
>>>>
>>>>> Thanks Bowen.
>>>>> Any idea why is cross_node_timeout commented out by default? That
>>>>> seems like a good option to enable even as per the documentation:
>>>>> # If disabled, replicas will assume that requests
>>>>> # were forwarded to them instantly by the coordinator, which means that
>>>>> # under overload conditions we will waste that much extra time
>>>>> processing
>>>>> # already-timed-out requests.
>>>>>
>>>>> Also, taking an example from Oracle kind of RDBMS systems, is there a
>>>>> command like the following that can be fired from an external script to
>>>>> kill a long running query on each node:
>>>>>
>>>>> alter system kill session
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 12, 2021 at 10:49 AM Bowen Song <bo...@bso.ng> wrote:
>>>>>
>>>>>> That will depend on whether you have cross_node_timeout enabled.
>>>>>> However, I have to point out that set timeout to 15ms is perhaps not a good
>>>>>> idea, the JVM GC can easily cause a lots of timeouts.
>>>>>> On 12/10/2021 18:20, S G wrote:
>>>>>>
>>>>>> ok, when a coordinator node sends timeout to the client, does it mean
>>>>>> all the replica nodes have stopped processing that specific query too?
>>>>>> Or is it just the coordinator node that has stopped waiting for the
>>>>>> replicas to return response?
>>>>>>
>>>>>> On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>>>>>
>>>>>>> It sends an exception to the client, it doesnt sever the connection.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 12, 2021 at 10:06 AM S G <sg...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Do the timeout values only kill the connection with the client or
>>>>>>>> send error to the client?
>>>>>>>> Or do they also kill the corresponding query execution happening on
>>>>>>>> the Cassandra servers (co-ordinator, replicas etc) ?
>>>>>>>>
>>>>>>>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa <jj...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The read and write timeout values do this today.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Oct 12, 2021 at 9:53 AM S G <sg...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> Is there a way to stop long running queries in Cassandra
>>>>>>>>>> (versions 3.11.x or 4.x) ?
>>>>>>>>>> The use-case is to have some kind of a circuit breaker based on
>>>>>>>>>> query-time that has exceeded the client's SLAs.
>>>>>>>>>> Example: If server response is useless to the client after 10 ms,
>>>>>>>>>> then we could
>>>>>>>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms
>>>>>>>>>> allows for some buffer).
>>>>>>>>>> And when that much time has elapsed, Cassandra will kill the
>>>>>>>>>> query execution automatically.
>>>>>>>>>>
>>>>>>>>>> If this is not possible in Cassandra currently, any chance we can
>>>>>>>>>> do it outside of Cassandra, like
>>>>>>>>>> a shell script that monitors such long running queries (through
>>>>>>>>>> users table etc) and kills the
>>>>>>>>>> OS-thread responsible for that query (Looks unsafe though as that
>>>>>>>>>> might leave the DB in an inconsistent state) ?
>>>>>>>>>>
>>>>>>>>>> We are trying this as a proactive measure to safeguard our
>>>>>>>>>> clusters from any rogue queries fired accidentally or maliciously.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by Jeff Jirsa <jj...@gmail.com>.
Convention in the yaml is default being visible commented out.
On Wed, Oct 13, 2021 at 2:17 PM S G <sg...@gmail.com> wrote:
> ok, the link given has the value commented, so I was a bit confused.
> But then https://github.com/apache/cassandra/search?q=cross_node_timeout
> shows that default value is indeed true.
> Thanks for the help,
>
> On Wed, Oct 13, 2021 at 11:26 AM Jeff Jirsa <jj...@gmail.com> wrote:
>
>> The default is true:
>>
>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1000
>>
>> There is no equivalent to `alter system kill session`, because it is
>> assumed that any query has a short, finite life in the order of seconds.
>>
>>
>>
>> On Wed, Oct 13, 2021 at 11:10 AM S G <sg...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Does anyone know about the default being turned off for this setting?
>>> It seems like a good one to be turned on - why have replicas process
>>> something for which coordinator has already sent the timeout to client?
>>>
>>> Thanks
>>>
>>> On Tue, Oct 12, 2021 at 11:06 AM S G <sg...@gmail.com> wrote:
>>>
>>>> Thanks Bowen.
>>>> Any idea why is cross_node_timeout commented out by default? That seems
>>>> like a good option to enable even as per the documentation:
>>>> # If disabled, replicas will assume that requests
>>>> # were forwarded to them instantly by the coordinator, which means that
>>>> # under overload conditions we will waste that much extra time
>>>> processing
>>>> # already-timed-out requests.
>>>>
>>>> Also, taking an example from Oracle kind of RDBMS systems, is there a
>>>> command like the following that can be fired from an external script to
>>>> kill a long running query on each node:
>>>>
>>>> alter system kill session
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Oct 12, 2021 at 10:49 AM Bowen Song <bo...@bso.ng> wrote:
>>>>
>>>>> That will depend on whether you have cross_node_timeout enabled.
>>>>> However, I have to point out that set timeout to 15ms is perhaps not a good
>>>>> idea, the JVM GC can easily cause a lots of timeouts.
>>>>> On 12/10/2021 18:20, S G wrote:
>>>>>
>>>>> ok, when a coordinator node sends timeout to the client, does it mean
>>>>> all the replica nodes have stopped processing that specific query too?
>>>>> Or is it just the coordinator node that has stopped waiting for the
>>>>> replicas to return response?
>>>>>
>>>>> On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>>>>
>>>>>> It sends an exception to the client, it doesnt sever the connection.
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 12, 2021 at 10:06 AM S G <sg...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Do the timeout values only kill the connection with the client or
>>>>>>> send error to the client?
>>>>>>> Or do they also kill the corresponding query execution happening on
>>>>>>> the Cassandra servers (co-ordinator, replicas etc) ?
>>>>>>>
>>>>>>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa <jj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The read and write timeout values do this today.
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 12, 2021 at 9:53 AM S G <sg...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> Is there a way to stop long running queries in Cassandra (versions
>>>>>>>>> 3.11.x or 4.x) ?
>>>>>>>>> The use-case is to have some kind of a circuit breaker based on
>>>>>>>>> query-time that has exceeded the client's SLAs.
>>>>>>>>> Example: If server response is useless to the client after 10 ms,
>>>>>>>>> then we could
>>>>>>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms
>>>>>>>>> allows for some buffer).
>>>>>>>>> And when that much time has elapsed, Cassandra will kill the query
>>>>>>>>> execution automatically.
>>>>>>>>>
>>>>>>>>> If this is not possible in Cassandra currently, any chance we can
>>>>>>>>> do it outside of Cassandra, like
>>>>>>>>> a shell script that monitors such long running queries (through
>>>>>>>>> users table etc) and kills the
>>>>>>>>> OS-thread responsible for that query (Looks unsafe though as that
>>>>>>>>> might leave the DB in an inconsistent state) ?
>>>>>>>>>
>>>>>>>>> We are trying this as a proactive measure to safeguard our
>>>>>>>>> clusters from any rogue queries fired accidentally or maliciously.
>>>>>>>>>
>>>>>>>>> Thanks !
>>>>>>>>>
>>>>>>>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by S G <sg...@gmail.com>.
ok, the link given has the value commented, so I was a bit confused.
But then https://github.com/apache/cassandra/search?q=cross_node_timeout
shows that default value is indeed true.
Thanks for the help,
On Wed, Oct 13, 2021 at 11:26 AM Jeff Jirsa <jj...@gmail.com> wrote:
> The default is true:
>
> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1000
>
> There is no equivalent to `alter system kill session`, because it is
> assumed that any query has a short, finite life in the order of seconds.
>
>
>
> On Wed, Oct 13, 2021 at 11:10 AM S G <sg...@gmail.com> wrote:
>
>> Hello,
>>
>> Does anyone know about the default being turned off for this setting?
>> It seems like a good one to be turned on - why have replicas process
>> something for which coordinator has already sent the timeout to client?
>>
>> Thanks
>>
>> On Tue, Oct 12, 2021 at 11:06 AM S G <sg...@gmail.com> wrote:
>>
>>> Thanks Bowen.
>>> Any idea why is cross_node_timeout commented out by default? That seems
>>> like a good option to enable even as per the documentation:
>>> # If disabled, replicas will assume that requests
>>> # were forwarded to them instantly by the coordinator, which means that
>>> # under overload conditions we will waste that much extra time processing
>>> # already-timed-out requests.
>>>
>>> Also, taking an example from Oracle kind of RDBMS systems, is there a
>>> command like the following that can be fired from an external script to
>>> kill a long running query on each node:
>>>
>>> alter system kill session
>>>
>>>
>>>
>>>
>>> On Tue, Oct 12, 2021 at 10:49 AM Bowen Song <bo...@bso.ng> wrote:
>>>
>>>> That will depend on whether you have cross_node_timeout enabled.
>>>> However, I have to point out that set timeout to 15ms is perhaps not a good
>>>> idea, the JVM GC can easily cause a lots of timeouts.
>>>> On 12/10/2021 18:20, S G wrote:
>>>>
>>>> ok, when a coordinator node sends timeout to the client, does it mean
>>>> all the replica nodes have stopped processing that specific query too?
>>>> Or is it just the coordinator node that has stopped waiting for the
>>>> replicas to return response?
>>>>
>>>> On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>>>
>>>>> It sends an exception to the client, it doesnt sever the connection.
>>>>>
>>>>>
>>>>> On Tue, Oct 12, 2021 at 10:06 AM S G <sg...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Do the timeout values only kill the connection with the client or
>>>>>> send error to the client?
>>>>>> Or do they also kill the corresponding query execution happening on
>>>>>> the Cassandra servers (co-ordinator, replicas etc) ?
>>>>>>
>>>>>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>>>>>
>>>>>>> The read and write timeout values do this today.
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 12, 2021 at 9:53 AM S G <sg...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Is there a way to stop long running queries in Cassandra (versions
>>>>>>>> 3.11.x or 4.x) ?
>>>>>>>> The use-case is to have some kind of a circuit breaker based on
>>>>>>>> query-time that has exceeded the client's SLAs.
>>>>>>>> Example: If server response is useless to the client after 10 ms,
>>>>>>>> then we could
>>>>>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms
>>>>>>>> allows for some buffer).
>>>>>>>> And when that much time has elapsed, Cassandra will kill the query
>>>>>>>> execution automatically.
>>>>>>>>
>>>>>>>> If this is not possible in Cassandra currently, any chance we can
>>>>>>>> do it outside of Cassandra, like
>>>>>>>> a shell script that monitors such long running queries (through
>>>>>>>> users table etc) and kills the
>>>>>>>> OS-thread responsible for that query (Looks unsafe though as that
>>>>>>>> might leave the DB in an inconsistent state) ?
>>>>>>>>
>>>>>>>> We are trying this as a proactive measure to safeguard our clusters
>>>>>>>> from any rogue queries fired accidentally or maliciously.
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>>
>>>>>>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by Jeff Jirsa <jj...@gmail.com>.
The default is true:
https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1000
There is no equivalent to `alter system kill session`, because it is
assumed that any query has a short, finite life in the order of seconds.
On Wed, Oct 13, 2021 at 11:10 AM S G <sg...@gmail.com> wrote:
> Hello,
>
> Does anyone know about the default being turned off for this setting?
> It seems like a good one to be turned on - why have replicas process
> something for which coordinator has already sent the timeout to client?
>
> Thanks
>
> On Tue, Oct 12, 2021 at 11:06 AM S G <sg...@gmail.com> wrote:
>
>> Thanks Bowen.
>> Any idea why is cross_node_timeout commented out by default? That seems
>> like a good option to enable even as per the documentation:
>> # If disabled, replicas will assume that requests
>> # were forwarded to them instantly by the coordinator, which means that
>> # under overload conditions we will waste that much extra time processing
>> # already-timed-out requests.
>>
>> Also, taking an example from Oracle kind of RDBMS systems, is there a
>> command like the following that can be fired from an external script to
>> kill a long running query on each node:
>>
>> alter system kill session
>>
>>
>>
>>
>> On Tue, Oct 12, 2021 at 10:49 AM Bowen Song <bo...@bso.ng> wrote:
>>
>>> That will depend on whether you have cross_node_timeout enabled.
>>> However, I have to point out that set timeout to 15ms is perhaps not a good
>>> idea, the JVM GC can easily cause a lots of timeouts.
>>> On 12/10/2021 18:20, S G wrote:
>>>
>>> ok, when a coordinator node sends timeout to the client, does it mean
>>> all the replica nodes have stopped processing that specific query too?
>>> Or is it just the coordinator node that has stopped waiting for the
>>> replicas to return response?
>>>
>>> On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>>
>>>> It sends an exception to the client, it doesnt sever the connection.
>>>>
>>>>
>>>> On Tue, Oct 12, 2021 at 10:06 AM S G <sg...@gmail.com> wrote:
>>>>
>>>>> Do the timeout values only kill the connection with the client or send
>>>>> error to the client?
>>>>> Or do they also kill the corresponding query execution happening on
>>>>> the Cassandra servers (co-ordinator, replicas etc) ?
>>>>>
>>>>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>>>>
>>>>>> The read and write timeout values do this today.
>>>>>>
>>>>>>
>>>>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 12, 2021 at 9:53 AM S G <sg...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Is there a way to stop long running queries in Cassandra (versions
>>>>>>> 3.11.x or 4.x) ?
>>>>>>> The use-case is to have some kind of a circuit breaker based on
>>>>>>> query-time that has exceeded the client's SLAs.
>>>>>>> Example: If server response is useless to the client after 10 ms,
>>>>>>> then we could
>>>>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms
>>>>>>> allows for some buffer).
>>>>>>> And when that much time has elapsed, Cassandra will kill the query
>>>>>>> execution automatically.
>>>>>>>
>>>>>>> If this is not possible in Cassandra currently, any chance we can do
>>>>>>> it outside of Cassandra, like
>>>>>>> a shell script that monitors such long running queries (through
>>>>>>> users table etc) and kills the
>>>>>>> OS-thread responsible for that query (Looks unsafe though as that
>>>>>>> might leave the DB in an inconsistent state) ?
>>>>>>>
>>>>>>> We are trying this as a proactive measure to safeguard our clusters
>>>>>>> from any rogue queries fired accidentally or maliciously.
>>>>>>>
>>>>>>> Thanks !
>>>>>>>
>>>>>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by S G <sg...@gmail.com>.
Hello,
Does anyone know about the default being turned off for this setting?
It seems like a good one to be turned on - why have replicas process
something for which coordinator has already sent the timeout to client?
Thanks
On Tue, Oct 12, 2021 at 11:06 AM S G <sg...@gmail.com> wrote:
> Thanks Bowen.
> Any idea why is cross_node_timeout commented out by default? That seems
> like a good option to enable even as per the documentation:
> # If disabled, replicas will assume that requests
> # were forwarded to them instantly by the coordinator, which means that
> # under overload conditions we will waste that much extra time processing
> # already-timed-out requests.
>
> Also, taking an example from Oracle kind of RDBMS systems, is there a
> command like the following that can be fired from an external script to
> kill a long running query on each node:
>
> alter system kill session
>
>
>
>
> On Tue, Oct 12, 2021 at 10:49 AM Bowen Song <bo...@bso.ng> wrote:
>
>> That will depend on whether you have cross_node_timeout enabled. However,
>> I have to point out that set timeout to 15ms is perhaps not a good idea,
>> the JVM GC can easily cause a lots of timeouts.
>> On 12/10/2021 18:20, S G wrote:
>>
>> ok, when a coordinator node sends timeout to the client, does it mean all
>> the replica nodes have stopped processing that specific query too?
>> Or is it just the coordinator node that has stopped waiting for the
>> replicas to return response?
>>
>> On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>> It sends an exception to the client, it doesnt sever the connection.
>>>
>>>
>>> On Tue, Oct 12, 2021 at 10:06 AM S G <sg...@gmail.com> wrote:
>>>
>>>> Do the timeout values only kill the connection with the client or send
>>>> error to the client?
>>>> Or do they also kill the corresponding query execution happening on the
>>>> Cassandra servers (co-ordinator, replicas etc) ?
>>>>
>>>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>>>
>>>>> The read and write timeout values do this today.
>>>>>
>>>>>
>>>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
>>>>>
>>>>>
>>>>> On Tue, Oct 12, 2021 at 9:53 AM S G <sg...@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Is there a way to stop long running queries in Cassandra (versions
>>>>>> 3.11.x or 4.x) ?
>>>>>> The use-case is to have some kind of a circuit breaker based on
>>>>>> query-time that has exceeded the client's SLAs.
>>>>>> Example: If server response is useless to the client after 10 ms,
>>>>>> then we could
>>>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms
>>>>>> allows for some buffer).
>>>>>> And when that much time has elapsed, Cassandra will kill the query
>>>>>> execution automatically.
>>>>>>
>>>>>> If this is not possible in Cassandra currently, any chance we can do
>>>>>> it outside of Cassandra, like
>>>>>> a shell script that monitors such long running queries (through users
>>>>>> table etc) and kills the
>>>>>> OS-thread responsible for that query (Looks unsafe though as that
>>>>>> might leave the DB in an inconsistent state) ?
>>>>>>
>>>>>> We are trying this as a proactive measure to safeguard our clusters
>>>>>> from any rogue queries fired accidentally or maliciously.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by S G <sg...@gmail.com>.
Thanks Bowen.
Any idea why is cross_node_timeout commented out by default? That seems
like a good option to enable even as per the documentation:
# If disabled, replicas will assume that requests
# were forwarded to them instantly by the coordinator, which means that
# under overload conditions we will waste that much extra time processing
# already-timed-out requests.
Also, taking an example from Oracle kind of RDBMS systems, is there a
command like the following that can be fired from an external script to
kill a long running query on each node:
alter system kill session
On Tue, Oct 12, 2021 at 10:49 AM Bowen Song <bo...@bso.ng> wrote:
> That will depend on whether you have cross_node_timeout enabled. However,
> I have to point out that set timeout to 15ms is perhaps not a good idea,
> the JVM GC can easily cause a lots of timeouts.
> On 12/10/2021 18:20, S G wrote:
>
> ok, when a coordinator node sends timeout to the client, does it mean all
> the replica nodes have stopped processing that specific query too?
> Or is it just the coordinator node that has stopped waiting for the
> replicas to return response?
>
> On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
>
>> It sends an exception to the client, it doesnt sever the connection.
>>
>>
>> On Tue, Oct 12, 2021 at 10:06 AM S G <sg...@gmail.com> wrote:
>>
>>> Do the timeout values only kill the connection with the client or send
>>> error to the client?
>>> Or do they also kill the corresponding query execution happening on the
>>> Cassandra servers (co-ordinator, replicas etc) ?
>>>
>>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>>
>>>> The read and write timeout values do this today.
>>>>
>>>>
>>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
>>>>
>>>>
>>>> On Tue, Oct 12, 2021 at 9:53 AM S G <sg...@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Is there a way to stop long running queries in Cassandra (versions
>>>>> 3.11.x or 4.x) ?
>>>>> The use-case is to have some kind of a circuit breaker based on
>>>>> query-time that has exceeded the client's SLAs.
>>>>> Example: If server response is useless to the client after 10 ms, then
>>>>> we could
>>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms
>>>>> allows for some buffer).
>>>>> And when that much time has elapsed, Cassandra will kill the query
>>>>> execution automatically.
>>>>>
>>>>> If this is not possible in Cassandra currently, any chance we can do
>>>>> it outside of Cassandra, like
>>>>> a shell script that monitors such long running queries (through users
>>>>> table etc) and kills the
>>>>> OS-thread responsible for that query (Looks unsafe though as that
>>>>> might leave the DB in an inconsistent state) ?
>>>>>
>>>>> We are trying this as a proactive measure to safeguard our clusters
>>>>> from any rogue queries fired accidentally or maliciously.
>>>>>
>>>>> Thanks !
>>>>>
>>>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by Bowen Song <bo...@bso.ng>.
That will depend on whether you have cross_node_timeout enabled.
However, I have to point out that set timeout to 15ms is perhaps not a
good idea, the JVM GC can easily cause a lots of timeouts.
On 12/10/2021 18:20, S G wrote:
> ok, when a coordinator node sends timeout to the client, does it mean
> all the replica nodes have stopped processing that specific query too?
> Or is it just the coordinator node that has stopped waiting for the
> replicas to return response?
>
> On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
>
> It sends an exception to the client, it doesnt sever the connection.
>
>
> On Tue, Oct 12, 2021 at 10:06 AM S G <sg...@gmail.com>
> wrote:
>
> Do the timeout values only kill the connection with the client
> or send error to the client?
> Or do they also kill the corresponding query execution
> happening on the Cassandra servers (co-ordinator, replicas etc) ?
>
> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa <jj...@gmail.com>
> wrote:
>
> The read and write timeout values do this today.
>
> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
>
>
> On Tue, Oct 12, 2021 at 9:53 AM S G
> <sg...@gmail.com> wrote:
>
> Hello,
>
> Is there a way to stop long running queries in
> Cassandra (versions 3.11.x or 4.x) ?
> The use-case is to have some kind of a circuit breaker
> based on query-time that has exceeded the client's SLAs.
> Example: If server response is useless to the client
> after 10 ms, then we could
> have a *query_killing_timeout* set to 15 ms (where
> additional 5ms allows for some buffer).
> And when that much time has elapsed, Cassandra will
> kill the query execution automatically.
>
> If this is not possible in Cassandra currently, any
> chance we can do it outside of Cassandra, like
> a shell script that monitors such long running queries
> (through users table etc) and kills the
> OS-thread responsible for that query (Looks unsafe
> though as that might leave the DB in an inconsistent
> state) ?
>
> We are trying this as a proactive measure to safeguard
> our clusters from any rogue queries fired accidentally
> or maliciously.
>
> Thanks !
>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by S G <sg...@gmail.com>.
ok, when a coordinator node sends timeout to the client, does it mean all
the replica nodes have stopped processing that specific query too?
Or is it just the coordinator node that has stopped waiting for the
replicas to return response?
On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
> It sends an exception to the client, it doesnt sever the connection.
>
>
> On Tue, Oct 12, 2021 at 10:06 AM S G <sg...@gmail.com> wrote:
>
>> Do the timeout values only kill the connection with the client or send
>> error to the client?
>> Or do they also kill the corresponding query execution happening on the
>> Cassandra servers (co-ordinator, replicas etc) ?
>>
>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>> The read and write timeout values do this today.
>>>
>>>
>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
>>>
>>>
>>> On Tue, Oct 12, 2021 at 9:53 AM S G <sg...@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Is there a way to stop long running queries in Cassandra (versions
>>>> 3.11.x or 4.x) ?
>>>> The use-case is to have some kind of a circuit breaker based on
>>>> query-time that has exceeded the client's SLAs.
>>>> Example: If server response is useless to the client after 10 ms, then
>>>> we could
>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms
>>>> allows for some buffer).
>>>> And when that much time has elapsed, Cassandra will kill the query
>>>> execution automatically.
>>>>
>>>> If this is not possible in Cassandra currently, any chance we can do it
>>>> outside of Cassandra, like
>>>> a shell script that monitors such long running queries (through users
>>>> table etc) and kills the
>>>> OS-thread responsible for that query (Looks unsafe though as that might
>>>> leave the DB in an inconsistent state) ?
>>>>
>>>> We are trying this as a proactive measure to safeguard our clusters
>>>> from any rogue queries fired accidentally or maliciously.
>>>>
>>>> Thanks !
>>>>
>>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by Jeff Jirsa <jj...@gmail.com>.
It sends an exception to the client, it doesnt sever the connection.
On Tue, Oct 12, 2021 at 10:06 AM S G <sg...@gmail.com> wrote:
> Do the timeout values only kill the connection with the client or send
> error to the client?
> Or do they also kill the corresponding query execution happening on the
> Cassandra servers (co-ordinator, replicas etc) ?
>
> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa <jj...@gmail.com> wrote:
>
>> The read and write timeout values do this today.
>>
>>
>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
>>
>>
>> On Tue, Oct 12, 2021 at 9:53 AM S G <sg...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Is there a way to stop long running queries in Cassandra (versions
>>> 3.11.x or 4.x) ?
>>> The use-case is to have some kind of a circuit breaker based on
>>> query-time that has exceeded the client's SLAs.
>>> Example: If server response is useless to the client after 10 ms, then
>>> we could
>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms allows
>>> for some buffer).
>>> And when that much time has elapsed, Cassandra will kill the query
>>> execution automatically.
>>>
>>> If this is not possible in Cassandra currently, any chance we can do it
>>> outside of Cassandra, like
>>> a shell script that monitors such long running queries (through users
>>> table etc) and kills the
>>> OS-thread responsible for that query (Looks unsafe though as that might
>>> leave the DB in an inconsistent state) ?
>>>
>>> We are trying this as a proactive measure to safeguard our clusters from
>>> any rogue queries fired accidentally or maliciously.
>>>
>>> Thanks !
>>>
>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by S G <sg...@gmail.com>.
Do the timeout values only kill the connection with the client or send
error to the client?
Or do they also kill the corresponding query execution happening on the
Cassandra servers (co-ordinator, replicas etc) ?
On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa <jj...@gmail.com> wrote:
> The read and write timeout values do this today.
>
>
> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
>
>
> On Tue, Oct 12, 2021 at 9:53 AM S G <sg...@gmail.com> wrote:
>
>> Hello,
>>
>> Is there a way to stop long running queries in Cassandra (versions 3.11.x
>> or 4.x) ?
>> The use-case is to have some kind of a circuit breaker based on
>> query-time that has exceeded the client's SLAs.
>> Example: If server response is useless to the client after 10 ms, then we
>> could
>> have a *query_killing_timeout* set to 15 ms (where additional 5ms allows
>> for some buffer).
>> And when that much time has elapsed, Cassandra will kill the query
>> execution automatically.
>>
>> If this is not possible in Cassandra currently, any chance we can do it
>> outside of Cassandra, like
>> a shell script that monitors such long running queries (through users
>> table etc) and kills the
>> OS-thread responsible for that query (Looks unsafe though as that might
>> leave the DB in an inconsistent state) ?
>>
>> We are trying this as a proactive measure to safeguard our clusters from
>> any rogue queries fired accidentally or maliciously.
>>
>> Thanks !
>>
>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Posted by Jeff Jirsa <jj...@gmail.com>.
The read and write timeout values do this today.
https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943
On Tue, Oct 12, 2021 at 9:53 AM S G <sg...@gmail.com> wrote:
> Hello,
>
> Is there a way to stop long running queries in Cassandra (versions 3.11.x
> or 4.x) ?
> The use-case is to have some kind of a circuit breaker based on query-time
> that has exceeded the client's SLAs.
> Example: If server response is useless to the client after 10 ms, then we
> could
> have a *query_killing_timeout* set to 15 ms (where additional 5ms allows
> for some buffer).
> And when that much time has elapsed, Cassandra will kill the query
> execution automatically.
>
> If this is not possible in Cassandra currently, any chance we can do it
> outside of Cassandra, like
> a shell script that monitors such long running queries (through users
> table etc) and kills the
> OS-thread responsible for that query (Looks unsafe though as that might
> leave the DB in an inconsistent state) ?
>
> We are trying this as a proactive measure to safeguard our clusters from
> any rogue queries fired accidentally or maliciously.
>
> Thanks !
>
>