You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Bryce Godfrey <Br...@azaleos.com> on 2012/09/24 22:33:21 UTC
Prevent queries from OOM nodes
Is there anything I can do on the configuration side to prevent nodes from going OOM due to queries that will read large amounts of data and exceed the heap available?
For the past few days of we had some nodes consistently freezing/crashing with OOM. We got a heap dump into MAT and figured out the nodes were dying due to some queries for a few extremely large data sets. Tracked it back to an app that just didn't prevent users from doing these large queries, but it seems like Cassandra could be smart enough to guard against this type of thing?
Basically some kind of setting like "if the data to satisfy query > available heap then throw an error to the caller and about query". I would much rather return errors to clients then crash a node, as the error is easier to track down that way and resolve.
Thanks.
Re: Prevent queries from OOM nodes
Posted by Sylvain Lebresne <sy...@datastax.com>.
> Could you create one ?
> https://issues.apache.org/jira/browse/CASSANDRA
There's one already. See
https://issues.apache.org/jira/browse/CASSANDRA-3702 that redirect to
https://issues.apache.org/jira/browse/CASSANDRA-4415.
--
Sylvain
Re: Prevent queries from OOM nodes
Posted by aaron morton <aa...@thelastpickle.com>.
> It's not about columns, it's about rows, see example statement.
My bad, misread the CQL.
My jira search fu is poor, but I could not find an open ticket for paging row counts. Could you create one ?
https://issues.apache.org/jira/browse/CASSANDRA
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 2/10/2012, at 12:17 AM, Віталій Тимчишин <ti...@gmail.com> wrote:
> It's not about columns, it's about rows, see example statement.
> In QueryProcessor#processStatement it reads rows into list, then does list.size()
>
> 2012/10/1 aaron morton <aa...@thelastpickle.com>
>> CQL will read everything into List to make latter a count.
>
>
> From 1.0 onwards count paginated reading the columns. What version are you on ?
>
> https://issues.apache.org/jira/browse/CASSANDRA-2894
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 26/09/2012, at 8:26 PM, Віталій Тимчишин <ti...@gmail.com> wrote:
>
>> Actually an easy way to put cassandra down is
>> select count(*) from A limit 10000000
>> CQL will read everything into List to make latter a count.
>>
>> 2012/9/26 aaron morton <aa...@thelastpickle.com>
>> Can you provide some information on the queries and the size of the data they traversed ?
>>
>> The default maximum size for a single thrift message is 16MB, was it larger than that ? https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L375
>>
>> Cheers
>>
>>
>> On 25/09/2012, at 8:33 AM, Bryce Godfrey <Br...@azaleos.com> wrote:
>>
>>> Is there anything I can do on the configuration side to prevent nodes from going OOM due to queries that will read large amounts of data and exceed the heap available?
>>>
>>> For the past few days of we had some nodes consistently freezing/crashing with OOM. We got a heap dump into MAT and figured out the nodes were dying due to some queries for a few extremely large data sets. Tracked it back to an app that just didn’t prevent users from doing these large queries, but it seems like Cassandra could be smart enough to guard against this type of thing?
>>>
>>> Basically some kind of setting like “if the data to satisfy query > available heap then throw an error to the caller and about query”. I would much rather return errors to clients then crash a node, as the error is easier to track down that way and resolve.
>>>
>>> Thanks.
>>
>>
>>
>>
>> --
>> Best regards,
>> Vitalii Tymchyshyn
>
>
>
>
> --
> Best regards,
> Vitalii Tymchyshyn
Re: Prevent queries from OOM nodes
Posted by Віталій Тимчишин <ti...@gmail.com>.
It's not about columns, it's about rows, see example statement.
In QueryProcessor#processStatement it reads rows into list, then does
list.size()
2012/10/1 aaron morton <aa...@thelastpickle.com>
> CQL will read everything into List to make latter a count.
>
>
> From 1.0 onwards count paginated reading the columns. What version are you
> on ?
>
> https://issues.apache.org/jira/browse/CASSANDRA-2894
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 26/09/2012, at 8:26 PM, Віталій Тимчишин <ti...@gmail.com> wrote:
>
> Actually an easy way to put cassandra down is
> select count(*) from A limit 10000000
> CQL will read everything into List to make latter a count.
>
> 2012/9/26 aaron morton <aa...@thelastpickle.com>
>
>> Can you provide some information on the queries and the size of the data
>> they traversed ?
>>
>> The default maximum size for a single thrift message is 16MB, was it
>> larger than that ?
>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L375
>>
>> Cheers
>>
>>
>> On 25/09/2012, at 8:33 AM, Bryce Godfrey <Br...@azaleos.com>
>> wrote:
>>
>> Is there anything I can do on the configuration side to prevent nodes
>> from going OOM due to queries that will read large amounts of data and
>> exceed the heap available? ****
>> ** **
>> For the past few days of we had some nodes consistently freezing/crashing
>> with OOM. We got a heap dump into MAT and figured out the nodes were dying
>> due to some queries for a few extremely large data sets. Tracked it back
>> to an app that just didn't prevent users from doing these large queries,
>> but it seems like Cassandra could be smart enough to guard against this
>> type of thing?****
>> ** **
>> Basically some kind of setting like "if the data to satisfy query >
>> available heap then throw an error to the caller and about query". I would
>> much rather return errors to clients then crash a node, as the error is
>> easier to track down that way and resolve.****
>> ** **
>> Thanks.****
>>
>>
>>
>
>
> --
> Best regards,
> Vitalii Tymchyshyn
>
>
>
--
Best regards,
Vitalii Tymchyshyn
Re: Prevent queries from OOM nodes
Posted by aaron morton <aa...@thelastpickle.com>.
> CQL will read everything into List to make latter a count.
From 1.0 onwards count paginated reading the columns. What version are you on ?
https://issues.apache.org/jira/browse/CASSANDRA-2894
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 26/09/2012, at 8:26 PM, Віталій Тимчишин <ti...@gmail.com> wrote:
> Actually an easy way to put cassandra down is
> select count(*) from A limit 10000000
> CQL will read everything into List to make latter a count.
>
> 2012/9/26 aaron morton <aa...@thelastpickle.com>
> Can you provide some information on the queries and the size of the data they traversed ?
>
> The default maximum size for a single thrift message is 16MB, was it larger than that ? https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L375
>
> Cheers
>
>
> On 25/09/2012, at 8:33 AM, Bryce Godfrey <Br...@azaleos.com> wrote:
>
>> Is there anything I can do on the configuration side to prevent nodes from going OOM due to queries that will read large amounts of data and exceed the heap available?
>>
>> For the past few days of we had some nodes consistently freezing/crashing with OOM. We got a heap dump into MAT and figured out the nodes were dying due to some queries for a few extremely large data sets. Tracked it back to an app that just didn’t prevent users from doing these large queries, but it seems like Cassandra could be smart enough to guard against this type of thing?
>>
>> Basically some kind of setting like “if the data to satisfy query > available heap then throw an error to the caller and about query”. I would much rather return errors to clients then crash a node, as the error is easier to track down that way and resolve.
>>
>> Thanks.
>
>
>
>
> --
> Best regards,
> Vitalii Tymchyshyn
Re: Prevent queries from OOM nodes
Posted by Віталій Тимчишин <ti...@gmail.com>.
Actually an easy way to put cassandra down is
select count(*) from A limit 10000000
CQL will read everything into List to make latter a count.
2012/9/26 aaron morton <aa...@thelastpickle.com>
> Can you provide some information on the queries and the size of the data
> they traversed ?
>
> The default maximum size for a single thrift message is 16MB, was it
> larger than that ?
> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L375
>
> Cheers
>
>
> On 25/09/2012, at 8:33 AM, Bryce Godfrey <Br...@azaleos.com>
> wrote:
>
> Is there anything I can do on the configuration side to prevent nodes from
> going OOM due to queries that will read large amounts of data and exceed
> the heap available? ****
> ** **
> For the past few days of we had some nodes consistently freezing/crashing
> with OOM. We got a heap dump into MAT and figured out the nodes were dying
> due to some queries for a few extremely large data sets. Tracked it back
> to an app that just didn’t prevent users from doing these large queries,
> but it seems like Cassandra could be smart enough to guard against this
> type of thing?****
> ** **
> Basically some kind of setting like “if the data to satisfy query >
> available heap then throw an error to the caller and about query”. I would
> much rather return errors to clients then crash a node, as the error is
> easier to track down that way and resolve.****
> ** **
> Thanks.****
>
>
>
--
Best regards,
Vitalii Tymchyshyn
Re: Prevent queries from OOM nodes
Posted by aaron morton <aa...@thelastpickle.com>.
Can you provide some information on the queries and the size of the data they traversed ?
The default maximum size for a single thrift message is 16MB, was it larger than that ? https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L375
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 25/09/2012, at 8:33 AM, Bryce Godfrey <Br...@azaleos.com> wrote:
> Is there anything I can do on the configuration side to prevent nodes from going OOM due to queries that will read large amounts of data and exceed the heap available?
>
> For the past few days of we had some nodes consistently freezing/crashing with OOM. We got a heap dump into MAT and figured out the nodes were dying due to some queries for a few extremely large data sets. Tracked it back to an app that just didn’t prevent users from doing these large queries, but it seems like Cassandra could be smart enough to guard against this type of thing?
>
> Basically some kind of setting like “if the data to satisfy query > available heap then throw an error to the caller and about query”. I would much rather return errors to clients then crash a node, as the error is easier to track down that way and resolve.
>
> Thanks.