You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kevin Burton <bu...@spinn3r.com> on 2016/08/03 19:58:34 UTC
Mutation of X bytes is too large for the maximum size of Y
It seems these are basically impossible to track down.
https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y-
has some information but their work around is to increase the transaction
log. There's no way to find out WHAT client or what CQL is causing the
large mutation.
Any thoughts on how to mitigate this?
Kevin
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Mutation of X bytes is too large for the maximum size of Y
Posted by Ryan Svihla <rs...@foundev.pro>.
On a related note, I still need to file a Jira just to make it easier to find large cells in general, I've had 2 customers now with a bunch of 10mb+ writes (single cell) they weren't expecting and tracking that down is equally challenging (Spark in both cases made it doable, but slow to find).
Regards,
Ryan Svihla
> On Aug 3, 2016, at 4:21 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
> I haven't verified, so i'm not 100% certain, but I believe you'd get back an exception to the client. Yes, this belongs in the DB, but I don't think you're totally blind to what went wrong.
>
> My guess is this exception in the Python driver (but other drivers should have a similar exception): https://github.com/datastax/python-driver/blob/master/cassandra/protocol.py#L288
>
>> On Wed, Aug 3, 2016 at 1:59 PM Ryan Svihla <rs...@foundev.pro> wrote:
>> Made a Jira about it already https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-12231
>>
>> Regards,
>>
>> Ryan Svihla
>>
>>> On Aug 3, 2016, at 2:58 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>>>
>>> It seems these are basically impossible to track down.
>>>
>>> https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y-
>>>
>>> has some information but their work around is to increase the transaction log. There's no way to find out WHAT client or what CQL is causing the large mutation.
>>>
>>> Any thoughts on how to mitigate this?
>>>
>>> Kevin
>>>
>>> --
>>> We’re hiring if you know of any awesome Java Devops or Linux Operations Engineers!
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: San Francisco, CA
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>>
Re: Mutation of X bytes is too large for the maximum size of Y
Posted by Kevin Burton <bu...@spinn3r.com>.
Yes.. Logging it is far far far far better.
I think a lot of devs don't have experience working in actual production
environments. YES the client should probably handle it, but WHICH client.
This is why you log things. Log the statement that was aborted (at least
the first 100 bytes),
On Wed, Aug 3, 2016 at 2:30 PM, Ryan Svihla <rs...@foundev.pro> wrote:
> Where I see this a lot is:
>
> 1. DBA notices it in logs
> 2. Everyone says code works fine no errors
> 3. Weeks of combing all apps find out 3 teams are doing fire and forget
> futures...
> 4. Convince each team they really need to handle futures
> 5. Couple months before you figure out who was the culprit by the time he
> deploys hit production.
>
> Would save everyone a ton of brain cells if we just logged it.
>
> Regards,
>
> Ryan Svihla
>
> On Aug 3, 2016, at 4:21 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
> I haven't verified, so i'm not 100% certain, but I believe you'd get back
> an exception to the client. Yes, this belongs in the DB, but I don't think
> you're totally blind to what went wrong.
>
> My guess is this exception in the Python driver (but other drivers should
> have a similar exception):
> https://github.com/datastax/python-driver/blob/master/cassandra/protocol.py#L288
>
> On Wed, Aug 3, 2016 at 1:59 PM Ryan Svihla <rs...@foundev.pro> wrote:
>
>> Made a Jira about it already
>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-12231
>>
>> Regards,
>>
>> Ryan Svihla
>>
>> On Aug 3, 2016, at 2:58 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>>
>> It seems these are basically impossible to track down.
>>
>>
>> https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y-
>>
>> has some information but their work around is to increase the transaction
>> log. There's no way to find out WHAT client or what CQL is causing the
>> large mutation.
>>
>> Any thoughts on how to mitigate this?
>>
>> Kevin
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>>
>>
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Mutation of X bytes is too large for the maximum size of Y
Posted by Ryan Svihla <rs...@foundev.pro>.
Where I see this a lot is:
1. DBA notices it in logs
2. Everyone says code works fine no errors
3. Weeks of combing all apps find out 3 teams are doing fire and forget futures...
4. Convince each team they really need to handle futures
5. Couple months before you figure out who was the culprit by the time he deploys hit production.
Would save everyone a ton of brain cells if we just logged it.
Regards,
Ryan Svihla
> On Aug 3, 2016, at 4:21 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
> I haven't verified, so i'm not 100% certain, but I believe you'd get back an exception to the client. Yes, this belongs in the DB, but I don't think you're totally blind to what went wrong.
>
> My guess is this exception in the Python driver (but other drivers should have a similar exception): https://github.com/datastax/python-driver/blob/master/cassandra/protocol.py#L288
>
>> On Wed, Aug 3, 2016 at 1:59 PM Ryan Svihla <rs...@foundev.pro> wrote:
>> Made a Jira about it already https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-12231
>>
>> Regards,
>>
>> Ryan Svihla
>>
>>> On Aug 3, 2016, at 2:58 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>>>
>>> It seems these are basically impossible to track down.
>>>
>>> https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y-
>>>
>>> has some information but their work around is to increase the transaction log. There's no way to find out WHAT client or what CQL is causing the large mutation.
>>>
>>> Any thoughts on how to mitigate this?
>>>
>>> Kevin
>>>
>>> --
>>> We’re hiring if you know of any awesome Java Devops or Linux Operations Engineers!
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: San Francisco, CA
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>>
Re: Mutation of X bytes is too large for the maximum size of Y
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
I haven't verified, so i'm not 100% certain, but I believe you'd get back
an exception to the client. Yes, this belongs in the DB, but I don't think
you're totally blind to what went wrong.
My guess is this exception in the Python driver (but other drivers should
have a similar exception):
https://github.com/datastax/python-driver/blob/master/cassandra/protocol.py#L288
On Wed, Aug 3, 2016 at 1:59 PM Ryan Svihla <rs...@foundev.pro> wrote:
> Made a Jira about it already
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-12231
>
> Regards,
>
> Ryan Svihla
>
> On Aug 3, 2016, at 2:58 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
> It seems these are basically impossible to track down.
>
>
> https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y-
>
> has some information but their work around is to increase the transaction
> log. There's no way to find out WHAT client or what CQL is causing the
> large mutation.
>
> Any thoughts on how to mitigate this?
>
> Kevin
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
>
Re: Mutation of X bytes is too large for the maximum size of Y
Posted by Ryan Svihla <rs...@foundev.pro>.
Made a Jira about it already https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-12231
Regards,
Ryan Svihla
> On Aug 3, 2016, at 2:58 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
> It seems these are basically impossible to track down.
>
> https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y-
>
> has some information but their work around is to increase the transaction log. There's no way to find out WHAT client or what CQL is causing the large mutation.
>
> Any thoughts on how to mitigate this?
>
> Kevin
>
> --
> We’re hiring if you know of any awesome Java Devops or Linux Operations Engineers!
>
> Founder/CEO Spinn3r.com
> Location: San Francisco, CA
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
>