You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dipan Shah <di...@hotmail.com> on 2019/02/28 10:08:16 UTC
MV's stuck in build state
Hello All,
I have a few MV's that are stuck in build state because of a bad schema design and thus getting a lot of messages like this "Mutation xxx is too large for maximum size of 16.000MiB".
[cid:cde7867a-aa71-4046-97f8-7a16b1a4f3c9]
I have dropped those MV's and I can no longer see their schema in the keyspace. But they are visible under "system.views_build_in_progress" and "nodetool viewbuildstatus".
I have tried "nodetool stop VIEW_BUILD" as suggested here: https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build and have also reboot a few nodes in the cluster. This has also not helped.
Is there anything else that can be done over here?
[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build>
Stop Cassandra Materialized View Build - Stack Overflow<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build>
Its not documented, but nodetool stop actually takes any compaction type, not just the ones listed (which the view build is one of). So you can simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction operation.. All thats really gonna do is set a flag for the view builder to stop on its next loop.
stackoverflow.com
Thanks,
Dipan Shah
RE: MV's stuck in build state
Posted by Kenneth Brotman <ke...@yahoo.com.INVALID>.
Hi Dipan,
What is the state of the dev and production clusters now?
How big is the production cluster?
How many nodes on each cluster spin out of control?
On the production cluster, the data is 67 MB, you'd have use a value at
least twice that size as the commitlog_segment_size. Of course you don't
want to leave it really high if you do change it.
On the dev cluster the data is 18 MB, you used a value way over twice size
when you bumped the commitlog_sement_size to 128 MB, but in doing so you are
wasting a lot of memory capacity of course, but you say it didn't fix the
problem on that cluster, so. is the message you are getting reflecting this
change by showing <y> to be 128 MB now or is it still a different value?
Is there another problem too perhaps you were running low on capacity on the
node? Could low capacity be a problem on either cluster?
From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Monday, March 04, 2019 12:52 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state
Hello Kenneth,
Apologies for the late reply.
1) On production the value of x was 67 MB and y was 16 MV as value of
commitlog_segment_size_in_mb is 32.
2) On Dev the value of x was 18 MB and y was 16 MV as value of
commitlog_segment_size_in_mb was 32 initially. I had bumped up the value of
commitlog_segment_size_in_mb to 128 when the node eventually crashed.
3) No I did not try org.apache.cassandra.db:type=CompactionManager but I did
try "nodetool stop" and "nodetool stop VIEW_BUILD".
Thanks,
Dipan Shah
_____
From: Kenneth Brotman <ke...@yahoo.com.INVALID>
Sent: Friday, March 1, 2019 8:19 PM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state
Dipan,
On your production cluster, when you were first getting the "Mutation of <x>
bytes ." message, what was the value of x and y?
How about when you got the message on the Dev Cluster, what was the value of
x and y in that message?
On the Dev cluster, did you try going into JMX and directly hitting the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation?
From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Friday, March 01, 2019 12:56 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state
Hello Kenneth,
Thanks for replying.
I had actually tried this on a Dev environment earlier and it caused the
node to spin out of control. I'll explain what I did over there:
1) Found "Mutation of <x> bytes is too large for the maxiumum size of <y>"
and thus increased the value of "commitlog_segment_size_in_mb" to 64
2) This worked for a few minutes and again the view started failing when it
hit the new limits and the messages now were "Mutation of <x> bytes is too
large for the maxiumum size of 2*<y>"
3) So just to try I increased the value to 128
4) Now after this change the node started crashing as soon as I brought the
service online. I was not able to recover even after restoring the value of
"commitlog_segment_size_in_mb" to 32
Now there is a key differences to that issue and what I am facing currently:
The views were not dropped on the earlier environment whereas I have already
dropped the view on the current environment (and cant experiment much as the
current environment is in production).
I know this is a bit tricky but I'm pretty much stuck over here and thinking
of finding a non-problem creating solution over here.
Thanks,
Dipan Shah
_____
From: Kenneth Brotman <ke...@yahoo.com.INVALID>
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state
Hi Dipan,
Did you try following the advice in the referenced DataStax article called
Mutation of
<https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-byte
s-is-too-large-for-the-maxiumum-size-of-y-> <x> bytes is too large for the
maximum size of <y> as suggested in the stackoverflow.com post you cited?
Kenneth Brotman
From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state
Forgot to add version info. This is on 37.
[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4]
Thanks,
Dipan Shah
_____
From: Dipan Shah <di...@hotmail.com>
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state
Hello All,
I have a few MV's that are stuck in build state because of a bad schema
design and thus getting a lot of messages like this "Mutation xxx is too
large for maximum size of 16.000MiB".
I have dropped those MV's and I can no longer see their schema in the
keyspace. But they are visible under "system.views_build_in_progress" and
"nodetool viewbuildstatus"
I have tried "nodetool stop VIEW_BUILD" as suggested here:
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vie
w-build and have also reboot a few nodes in the cluster. This has also not
helped.
Is there anything else that can be done over here?
<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vi
ew-build>
<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-vie
w-build> Stop Cassandra Materialized View Build - Stack Overflow
Its not documented, but nodetool stop actually takes any compaction type,
not just the ones listed (which the view build is one of). So you can
simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation.. All thats really gonna do is set a flag for the view builder to
stop on its next loop.
stackoverflow.com
Thanks,
Dipan Shah
Re: MV's stuck in build state
Posted by Dipan Shah <di...@hotmail.com>.
Hello Kenneth,
Apologies for the late reply.
1) On production the value of x was 67 MB and y was 16 MV as value of commitlog_segment_size_in_mb is 32.
2) On Dev the value of x was 18 MB and y was 16 MV as value of commitlog_segment_size_in_mb was 32 initially. I had bumped up the value of commitlog_segment_size_in_mb to 128 when the node eventually crashed.
3) No I did not try org.apache.cassandra.db:type=CompactionManager but I did try "nodetool stop" and "nodetool stop VIEW_BUILD".
Thanks,
Dipan Shah
________________________________
From: Kenneth Brotman <ke...@yahoo.com.INVALID>
Sent: Friday, March 1, 2019 8:19 PM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state
Dipan,
On your production cluster, when you were first getting the “Mutation of <x> bytes …” message, what was the value of x and y?
How about when you got the message on the Dev Cluster, what was the value of x and y in that message?
On the Dev cluster, did you try going into JMX and directly hitting the org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction operation?
From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Friday, March 01, 2019 12:56 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state
Hello Kenneth,
Thanks for replying.
I had actually tried this on a Dev environment earlier and it caused the node to spin out of control. I'll explain what I did over there:
1) Found "Mutation of <x> bytes is too large for the maxiumum size of <y>" and thus increased the value of "commitlog_segment_size_in_mb" to 64
2) This worked for a few minutes and again the view started failing when it hit the new limits and the messages now were "Mutation of <x> bytes is too large for the maxiumum size of 2*<y>"
3) So just to try I increased the value to 128
4) Now after this change the node started crashing as soon as I brought the service online. I was not able to recover even after restoring the value of "commitlog_segment_size_in_mb" to 32
Now there is a key differences to that issue and what I am facing currently:
The views were not dropped on the earlier environment whereas I have already dropped the view on the current environment (and cant experiment much as the current environment is in production).
I know this is a bit tricky but I'm pretty much stuck over here and thinking of finding a non-problem creating solution over here.
Thanks,
Dipan Shah
________________________________
From: Kenneth Brotman <ke...@yahoo.com.INVALID>
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state
Hi Dipan,
Did you try following the advice in the referenced DataStax article called Mutation of <x> bytes is too large for the maximum size of <y><https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y-> as suggested in the stackoverflow.com post you cited?
Kenneth Brotman
From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state
Forgot to add version info. This is on 3.7.
[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4]
Thanks,
Dipan Shah
________________________________
From: Dipan Shah <di...@hotmail.com>
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state
Hello All,
I have a few MV's that are stuck in build state because of a bad schema design and thus getting a lot of messages like this "Mutation xxx is too large for maximum size of 16.000MiB".
[cid:image001.png@01D4CFFA.539C41B0]
I have dropped those MV's and I can no longer see their schema in the keyspace. But they are visible under "system.views_build_in_progress" and "nodetool viewbuildstatus".
I have tried "nodetool stop VIEW_BUILD" as suggested here: https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build and have also reboot a few nodes in the cluster. This has also not helped.
Is there anything else that can be done over here?
[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build>
Stop Cassandra Materialized View Build - Stack Overflow<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-view-build>
Its not documented, but nodetool stop actually takes any compaction type, not just the ones listed (which the view build is one of). So you can simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction operation.. All thats really gonna do is set a flag for the view builder to stop on its next loop.
stackoverflow.com
Thanks,
Dipan Shah
RE: MV's stuck in build state
Posted by Kenneth Brotman <ke...@yahoo.com.INVALID>.
Dipan,
On your production cluster, when you were first getting the "Mutation of <x>
bytes ." message, what was the value of x and y?
How about when you got the message on the Dev Cluster, what was the value of
x and y in that message?
On the Dev cluster, did you try going into JMX and directly hitting the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation?
From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Friday, March 01, 2019 12:56 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state
Hello Kenneth,
Thanks for replying.
I had actually tried this on a Dev environment earlier and it caused the
node to spin out of control. I'll explain what I did over there:
1) Found "Mutation of <x> bytes is too large for the maxiumum size of <y>"
and thus increased the value of "commitlog_segment_size_in_mb" to 64
2) This worked for a few minutes and again the view started failing when it
hit the new limits and the messages now were "Mutation of <x> bytes is too
large for the maxiumum size of 2*<y>"
3) So just to try I increased the value to 128
4) Now after this change the node started crashing as soon as I brought the
service online. I was not able to recover even after restoring the value of
"commitlog_segment_size_in_mb" to 32
Now there is a key differences to that issue and what I am facing currently:
The views were not dropped on the earlier environment whereas I have already
dropped the view on the current environment (and cant experiment much as the
current environment is in production).
I know this is a bit tricky but I'm pretty much stuck over here and thinking
of finding a non-problem creating solution over here.
Thanks,
Dipan Shah
_____
From: Kenneth Brotman <ke...@yahoo.com.INVALID>
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state
Hi Dipan,
Did you try following the advice in the referenced DataStax article called
Mutation of
<https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-byte
s-is-too-large-for-the-maxiumum-size-of-y-> <x> bytes is too large for the
maximum size of <y> as suggested in the stackoverflow.com post you cited?
Kenneth Brotman
From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state
Forgot to add version info. This is on 3.7.
[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4]
Thanks,
Dipan Shah
_____
From: Dipan Shah <di...@hotmail.com>
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state
Hello All,
I have a few MV's that are stuck in build state because of a bad schema
design and thus getting a lot of messages like this "Mutation xxx is too
large for maximum size of 16.000MiB".
I have dropped those MV's and I can no longer see their schema in the
keyspace. But they are visible under "system.views_build_in_progress" and
"nodetool viewbuildstatus".
I have tried "nodetool stop VIEW_BUILD" as suggested here:
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vie
w-build and have also reboot a few nodes in the cluster. This has also not
helped.
Is there anything else that can be done over here?
<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vi
ew-build>
<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-vie
w-build> Stop Cassandra Materialized View Build - Stack Overflow
Its not documented, but nodetool stop actually takes any compaction type,
not just the ones listed (which the view build is one of). So you can
simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation.. All thats really gonna do is set a flag for the view builder to
stop on its next loop.
stackoverflow.com
Thanks,
Dipan Shah
Re: MV's stuck in build state
Posted by Dipan Shah <di...@hotmail.com>.
Hello Kenneth,
Thanks for replying.
I had actually tried this on a Dev environment earlier and it caused the node to spin out of control. I'll explain what I did over there:
1) Found "Mutation of <x> bytes is too large for the maxiumum size of <y>" and thus increased the value of "commitlog_segment_size_in_mb" to 64
2) This worked for a few minutes and again the view started failing when it hit the new limits and the messages now were "Mutation of <x> bytes is too large for the maxiumum size of 2*<y>"
3) So just to try I increased the value to 128
4) Now after this change the node started crashing as soon as I brought the service online. I was not able to recover even after restoring the value of "commitlog_segment_size_in_mb" to 32
Now there is a key differences to that issue and what I am facing currently:
The views were not dropped on the earlier environment whereas I have already dropped the view on the current environment (and cant experiment much as the current environment is in production).
I know this is a bit tricky but I'm pretty much stuck over here and thinking of finding a non-problem creating solution over here.
Thanks,
Dipan Shah
________________________________
From: Kenneth Brotman <ke...@yahoo.com.INVALID>
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state
Hi Dipan,
Did you try following the advice in the referenced DataStax article called Mutation of <x> bytes is too large for the maximum size of <y><https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y-> as suggested in the stackoverflow.com post you cited?
Kenneth Brotman
From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state
Forgot to add version info. This is on 3.7.
[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]
Thanks,
Dipan Shah
________________________________
From: Dipan Shah <di...@hotmail.com>
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state
Hello All,
I have a few MV's that are stuck in build state because of a bad schema design and thus getting a lot of messages like this "Mutation xxx is too large for maximum size of 16.000MiB".
[cid:image001.png@01D4CF54.4F7AFE10]
I have dropped those MV's and I can no longer see their schema in the keyspace. But they are visible under "system.views_build_in_progress" and "nodetool viewbuildstatus".
I have tried "nodetool stop VIEW_BUILD" as suggested here: https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build and have also reboot a few nodes in the cluster. This has also not helped.
Is there anything else that can be done over here?
[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build>
Stop Cassandra Materialized View Build - Stack Overflow<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-view-build>
Its not documented, but nodetool stop actually takes any compaction type, not just the ones listed (which the view build is one of). So you can simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction operation.. All thats really gonna do is set a flag for the view builder to stop on its next loop.
stackoverflow.com
Thanks,
Dipan Shah
RE: MV's stuck in build state
Posted by Kenneth Brotman <ke...@yahoo.com.INVALID>.
Hi Dipan,
Did you try following the advice in the referenced DataStax article called
Mutation of
<https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-byte
s-is-too-large-for-the-maxiumum-size-of-y-> <x> bytes is too large for the
maximum size of <y> as suggested in the stackoverflow.com post you cited?
Kenneth Brotman
From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state
Forgot to add version info. This is on 3.7.
[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]
Thanks,
Dipan Shah
_____
From: Dipan Shah <di...@hotmail.com>
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state
Hello All,
I have a few MV's that are stuck in build state because of a bad schema
design and thus getting a lot of messages like this "Mutation xxx is too
large for maximum size of 16.000MiB".
I have dropped those MV's and I can no longer see their schema in the
keyspace. But they are visible under "system.views_build_in_progress" and
"nodetool viewbuildstatus".
I have tried "nodetool stop VIEW_BUILD" as suggested here:
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vie
w-build and have also reboot a few nodes in the cluster. This has also not
helped.
Is there anything else that can be done over here?
<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vi
ew-build>
<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-vie
w-build> Stop Cassandra Materialized View Build - Stack Overflow
Its not documented, but nodetool stop actually takes any compaction type,
not just the ones listed (which the view build is one of). So you can
simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation.. All thats really gonna do is set a flag for the view builder to
stop on its next loop.
stackoverflow.com
Thanks,
Dipan Shah
Re: MV's stuck in build state
Posted by Dipan Shah <di...@hotmail.com>.
Forgot to add version info. This is on 3.7.
[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]
Thanks,
Dipan Shah
________________________________
From: Dipan Shah <di...@hotmail.com>
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state
Hello All,
I have a few MV's that are stuck in build state because of a bad schema design and thus getting a lot of messages like this "Mutation xxx is too large for maximum size of 16.000MiB".
[cid:cde7867a-aa71-4046-97f8-7a16b1a4f3c9]
I have dropped those MV's and I can no longer see their schema in the keyspace. But they are visible under "system.views_build_in_progress" and "nodetool viewbuildstatus".
I have tried "nodetool stop VIEW_BUILD" as suggested here: https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build and have also reboot a few nodes in the cluster. This has also not helped.
Is there anything else that can be done over here?
[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build>
Stop Cassandra Materialized View Build - Stack Overflow<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build>
Its not documented, but nodetool stop actually takes any compaction type, not just the ones listed (which the view build is one of). So you can simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction operation.. All thats really gonna do is set a flag for the view builder to stop on its next loop.
stackoverflow.com
Thanks,
Dipan Shah