You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Surbhi Gupta <su...@gmail.com> on 2023/03/01 22:43:00 UTC
Bootstrapping new node throwing error - Mutation too large
Hi Cassandra Community,
We have to expand our cluster and I tried to add the first node to the
cluster and when the new node was bootstrapping , I noticed the error like
below in the system.log, but the bootstrap process was successful .
We are on 3.11.5 .
ERROR [MutationStage-7] 2023-03-01 07:01:40,026
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[MutationStage-7,5,main]
java.lang.IllegalArgumentException: Mutation of 24.510MiB is too large for
the maximum size of 16.000MiB
I tried to add another node after 5 days of adding 1st node and the 2nd
node also gave an error but this time the mutation size was bigger .
ERROR [BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228 -
Exception in thread Thread[BatchlogTasks:1,5,main]
java.lang.IllegalArgumentException: Mutation of 40.717MiB is too large for
the maximum size of 16.000MiB
And on the first node( which was added 5 days ago), we saw messages like
below till the 2nd node bootstrap was completed.
"Mutation of 24.510MiB is too large for the maximum size of 16.000MiB"
but after the 2nd node bootstarp process was completed, we started seeing
the message about hint byte is too large but there are no hint files in any
of the nodes on hints directory . And the message about mutation too large
has stopped popping in system.log on the first added node. Now we are
seeing hints error as below on the first newly added node.
ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091 CassandraDaemon.java:228 -
Exception in thread Thread[BatchlogTasks:1,5,main]
java.lang.IllegalArgumentException: Hint of 25700336 bytes is too large -
the maximum size is 16777216
As per the application team , nothing changed .
As per the workaround commitlog_segment_size_in_mb can be increased to
accommodate the increased size but that doesnt seem to be a concrete
solution and it can have performance impact , because by design intent the
maximum allowed segment size is 50% of the configured
commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments
with large amounts of empty space.
Looks like i am hitting
https://issues.apache.org/jira/browse/CASSANDRA-15152
<https://issues.apache.org/jira/browse/CASSANDRA-15152>
Anyone have any suggestions?
Thanks
Surbhi
Re: Bootstrapping new node throwing error - Mutation too large
Posted by Surbhi Gupta <su...@gmail.com>.
Thanks Scott
On Wed, Mar 1, 2023 at 4:00 PM C. Scott Andreas <sc...@paradoxica.net>
wrote:
> The performance implications would primarily be due to the challenge of
> handling mutations this large themselves rather than the commitlog segment
> size. These would occupy large, contiguous areas of heap and increase
> memory pressure in the process.
>
> Increasing commit_log_segment_size_in_mb is likely the best / only
> approach, along with addressing mutation size for future writes from the
> application.
>
> I'd also strongly recommend upgrading from 3.11.5 to 3.11.15. 3.11.5 was
> released about 3.5 years ago, with a large number of bugfixes available in
> 3.11.15. That release is also drop-in, so you can upgrade simply by rev'ing
> the version and performing a rolling restart of the instances.
>
>
> – Scott
>
> On Mar 1, 2023, at 2:43 PM, Surbhi Gupta <su...@gmail.com> wrote:
>
>
> Hi Cassandra Community,
>
> We have to expand our cluster and I tried to add the first node to the
> cluster and when the new node was bootstrapping , I noticed the error like
> below in the system.log, but the bootstrap process was successful .
>
> We are on 3.11.5 .
>
> ERROR [MutationStage-7] 2023-03-01 07:01:40,026
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[MutationStage-7,5,main]
>
> java.lang.IllegalArgumentException: Mutation of 24.510MiB is too large for
> the maximum size of 16.000MiB
>
> I tried to add another node after 5 days of adding 1st node and the 2nd
> node also gave an error but this time the mutation size was bigger .
>
> ERROR [BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228
> - Exception in thread Thread[BatchlogTasks:1,5,main]
>
> java.lang.IllegalArgumentException: Mutation of 40.717MiB is too large for
> the maximum size of 16.000MiB
>
> And on the first node( which was added 5 days ago), we saw messages like
> below till the 2nd node bootstrap was completed.
> "Mutation of 24.510MiB is too large for the maximum size of 16.000MiB"
> but after the 2nd node bootstarp process was completed, we started seeing
> the message about hint byte is too large but there are no hint files in any
> of the nodes on hints directory . And the message about mutation too large
> has stopped popping in system.log on the first added node. Now we are
> seeing hints error as below on the first newly added node.
>
> ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091 CassandraDaemon.java:228
> - Exception in thread Thread[BatchlogTasks:1,5,main]
>
> java.lang.IllegalArgumentException: Hint of 25700336 bytes is too large -
> the maximum size is 16777216
>
> As per the application team , nothing changed .
>
> As per the workaround commitlog_segment_size_in_mb can be increased to
> accommodate the increased size but that doesnt seem to be a concrete
> solution and it can have performance impact , because by design intent
> the maximum allowed segment size is 50% of the configured
> commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments
> with large amounts of empty space.
>
> Looks like i am hitting
> https://issues.apache.org/jira/browse/CASSANDRA-15152
> <https://issues.apache.org/jira/browse/CASSANDRA-15152>
>
> Anyone have any suggestions?
>
> Thanks
> Surbhi
>
>
>
>
>
>
>
>
>
Re: Bootstrapping new node throwing error - Mutation too large
Posted by "C. Scott Andreas" <sc...@paradoxica.net>.
The performance implications would primarily be due to the challenge of handling mutations this large themselves rather than the commitlog segment size. These would occupy large, contiguous areas of heap and increase memory pressure in the process.Increasing commit_log_segment_size_in_mb is likely the best / only approach, along with addressing mutation size for future writes from the application.I'd also strongly recommend upgrading from 3.11.5 to 3.11.15. 3.11.5 was released about 3.5 years ago, with a large number of bugfixes available in 3.11.15. That release is also drop-in, so you can upgrade simply by rev'ing the version and performing a rolling restart of the instances.– ScottOn Mar 1, 2023, at 2:43 PM, Surbhi Gupta <su...@gmail.com> wrote:Hi Cassandra Community,We have to expand our cluster and I tried to add the first node to the cluster and when the new node was bootstrapping , I noticed the error like below in the system.log, but the bootstrap process was successful . We are on 3.11.5 .ERROR [MutationStage-7] 2023-03-01 07:01:40,026 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[MutationStage-7,5,main]java.lang.IllegalArgumentException: Mutation of 24.510MiB is too large for the maximum size of 16.000MiBI tried to add another node after 5 days of adding 1st node and the 2nd node also gave an error but this time the mutation size was bigger .ERROR [BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228 - Exception in thread Thread[BatchlogTasks:1,5,main]java.lang.IllegalArgumentException: Mutation of 40.717MiB is too large for the maximum size of 16.000MiBAnd on the first node( which was added 5 days ago), we saw messages like below till the 2nd node bootstrap was completed. "Mutation of 24.510MiB is too large for the maximum size of 16.000MiB"but after the 2nd node bootstarp process was completed, we started seeing the message about hint byte is too large but there are no hint files in any of the nodes on hints directory . And the message about mutation too large has stopped popping in system.log on the first added node. Now we are seeing hints error as below on the first newly added node. ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091 CassandraDaemon.java:228 - Exception in thread Thread[BatchlogTasks:1,5,main]java.lang.IllegalArgumentException: Hint of 25700336 bytes is too large - the maximum size is 16777216As per the application team , nothing changed . As per the workaround commitlog_segment_size_in_mb can be increased to accommodate the increased size but that doesnt seem to be a concrete solution and it can have performance impact , because by design intent the maximum allowed segment size is 50% of the configured commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments with large amounts of empty space. Looks like i am hitting https://issues.apache.org/jira/browse/CASSANDRA-15152 Anyone have any suggestions?ThanksSurbhi