You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Andrés Beck-Ruiz <an...@gmail.com> on 2026/04/08 21:57:22 UTC
[DISCUSS] Stale FastBuilder objects can cause ClassCastException on read and write requests (CASSANDRA-21216)
Hi all,
I’d like to discuss my investigation into CASSANDRA-21216
<https://issues.apache.org/jira/browse/CASSANDRA-21216>, a bug that my team
encountered during schema modification on a very wide table (~4200 columns)
in a 4.1 Cassandra cluster that was also serving reads and writes.
*Events required to reproduce:*
We have found that this bug triggers in the following scenario:
- There is a schema disagreement between a coordinator node and a replica
node when a new column is added:
- The coordinator node is performing a read request for the newly
added column
- The replica node does not have the new column yet
- The size of the internode READ_REQ is greater than ~ 65 KB
- The thread that fails to deserialize the READ_REQ is later reused by the
MutationStage
- The internode READ_REQ contains more than 31 columns (populates
savedBuffer and savedNextKey during deserialization)
- There are no more than 31 rows in-memory for a partition being mutated
(Row BTree has one leaf node)
- No more than 31 rows are being added in the mutation (PartitionUpdate
BTree has one leaf node)
*Why this bug surfaces: *
This bug stems from the schema disagreement described above while adding
new columns to a table, which causes internode message deserialization to
fail. When a replica node does not find a column included in the READ_REQ
message, it throws this exception
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/Columns.java#L489>,
causing the original request to fail. As a result, the BTree FastBuilder
build function
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/Columns.java#L493>
is never reached. On the happy path, the build function calls completeBuild
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/utils/btree/BTree.java#L2420>,
which consumes savedBuffer and savedNextKey. These FastBuilder objects are
used to create multi-level BTrees and are populated when a tree under
construction exceeds 31 elements.
When an exception is thrown, the build method is not called. As a result,
these fields retain their stale ColumnMetadata values. Although the
FastBuilder does call a reset
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/utils/btree/BTree.java#L3325>
function when it closes the object, the function does not clear out the
savedBuffer or savedNextKey, unlike the AbstractUpdater reset function
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/utils/btree/BTree.java#L3371-L3372>.
It is important that these objects are cleared out after use given that
they are shared locally per thread
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/utils/btree/BTree.java#L2313>.
When Cassandra deserializes large messages, deserialization is dispatched
to an SEPWorker thread pool
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/net/InboundMessageHandler.java#L200>instead
of being handled in the Netty event loop. For our case, we observed that
internode READ_REQ messages after expanding SELECT * queries on the 4200
column table were passing the 65 KB threshold. If the same thread that
handled a failed message deserialization then handles a mutation request,
it might use the same FastBuilder with stale ColumnMetadata objects to merge
the existing BTree of Rows in a partition with the partition update
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java#L148>.
If the existing BTree of Rows and partition update are leaf nodes, the
BTree update function uses the FastBuilder instead of the Updater
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/utils/btree/BTree.java#L391-L393>
to rebuild the tree. The FastBuilder then empties out the ColumnMetadata
objects in the savedBuffer and savedNextKey into the BTree that contains
Row objects. The ClassCastException surfaces when we read or write from the
partition in memory with the corrupted BTree.
I found that the same bug could resurface in trunk, given that the
deserialization of large messages is still handled on the SEPWorker thread
pool, and the FastBuilder reset function has not been updated to clear
savedBuffer and savedNextKey. However, I’m not sure whether TCM will reduce
or eliminate the possibility of schema mismatches across a cluster.
More information about how we verified this theory, as well as the full
stack traces, can be found on the bug ticket.
*Proposed fixes: *
A possible fix to this issue that I’ve added here
<https://github.com/andresbeckruiz/cassandra/commit/14dbac67bee3917ce71cd18dc48ef19f5f0cf649>
and verified builds successfully on Cassandra 4.1 is to clear the
savedBuffer and savedNextKey in the reset function, as is done in the
AbstractUpdater; I also noticed that AbstractUpdater does not set
savedNextKey to null, so this could be worth adding as well.
Another possible fix could be to handle both small and large messages in
the Netty event loop. If the FastBuilder objects are not cleared out, this
will guarantee that the FastBuilder is not reused by SEPWorker threads,
which could potentially corrupt Row BTrees during a MutationStage. However,
this could possibly block the event loop thread.
Given that this patch affects critical paths in Cassandra, I would
appreciate any feedback on any potential side-effects I have not
considered. If the approach to fixing this bug seems reasonable, I can also
add a DTest to mimic this edge case.
Best,
Andrés
Re: [DISCUSS] Stale FastBuilder objects can cause ClassCastException on read and write requests (CASSANDRA-21216)
Posted by Andrés Beck-Ruiz <an...@gmail.com>.
Hi all,
I have a patch ready fixing this bug on Cassandra 4.1 (
https://github.com/apache/cassandra/pull/4746). CI can be found here (
https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-before-5/2734/);
I've confirmed that all the unit tests pass locally, and that the dtests
that failed were flaky and unrelated to the changes in this patch.
I would appreciate a review on this before I make a similar patch for 4.0,
5.0, and trunk. Thank you in advance for your time!
Best,
Andrés
On Fri, Apr 10, 2026 at 3:52 AM Štefan Miklošovič <sm...@apache.org>
wrote:
> On Fri, Apr 10, 2026 at 9:37 AM Runtian Liu <cu...@gmail.com> wrote:
> >
> > Hi Andrés, Isaac,
> >
> > Thank you for the detailed write-up, Andrés. Your investigation into the
> FastBuilder.reset() bug was the starting point for our own analysis, which
> led us to identify an additional impact beyond the ClassCastException.
> >
> > Isaac — yes, we believe CASSANDRA-21260 and CASSANDRA-21216 are directly
> related. CASSANDRA-21260 was filed by our team to track the SSTable header
> contamination we've been seeing. Based on Andrés' findings about the stale
> savedBuffer/savedNextKey in FastBuilder.reset(), we investigated whether
> the same bug could explain our corrupted SSTable headers — and we believe
> it does.
> >
> > What we observed (CASSANDRA-21260)
> >
> > We have been seeing corrupted SSTable headers where an SSTable for one
> table contains column metadata belonging to a completely different table.
> When we deserialize the on-disk SerializationHeader.Component and compare
> it against the table's TableMetadata, we find column names that are not
> part of the table's schema — they belong to another table in the same
> keyspace. In one case, a table with ~2000 columns had 29 foreign columns
> from a ~150-column table embedded in its SSTable header.
> >
> > These corrupted SSTables are otherwise structurally valid — they are
> accepted into the live set and only detected by explicit header validation
> we added. The foreign columns do not correspond to dropped columns or any
> prior schema version of the affected table. As noted in CASSANDRA-21260,
> once a corrupted SSTable exists, compaction merges headers blindly, so the
> contamination propagates to new SSTables indefinitely.
>
> I do not want to be a scope creep here, but there is also
> CASSANDRA-21000 which will keep deleted columns in there forever. I do
> not see any issue with fixing it (details in the ticket), but at the
> same time I can not say 100% that it will not have any side-effects we
> did not count on.
>
> However, if we change the logic / do some fixes around
> SerializationHeader, I think it would be great to think about the
> inclusion of this ticket as well / to have it in mind.
>
> > How the FastBuilder bug (CASSANDRA-21216) causes this
> >
> > Building on Andrés' analysis of the FastBuilder state leakage, we traced
> a path from the stale savedBuffer/savedNextKey all the way to on-disk
> SSTable header contamination:
> >
> > 1. A schema disagreement (e.g. during column addition) causes an
> internode READ_REQ deserialization to fail on a replica.
> Columns.Serializer.deserialize() uses a thread-local pooled FastBuilder,
> and if the table has more than 31 columns, the overflow populates
> savedBuffer and savedNextKey before the exception. Since reset() does not
> clear these fields, the FastBuilder is returned to the pool with stale
> ColumnMetadata from the source table.
> >
> > 2. When a deletion-only mutation (partition delete or range tombstone)
> for a different table is later deserialized on the same thread,
> Columns.Serializer.deserialize() acquires the poisoned FastBuilder. The
> stale ColumnMetadata from the source table are drained into the victim
> table's Columns via propagateOverflow(). Because the mutation contains only
> a deletion — no rows, no static row — no per-row column-subset
> deserialization occurs, so the contaminated Columns survives without error.
> (Mutations with actual row data would fail due to subset encoding
> mismatches, which is why only deletion-only mutations propagate the
> contamination silently.)
> >
> > When the contaminated PartitionUpdate is applied to the memtable,
> ColumnsCollector.update() records the foreign ColumnMetadata. At flush,
> BigTableWriter.openFinal() writes the SSTable using the in-memory
> SerializationHeader directly, bypassing toHeader() validation. The result
> is an on-disk SSTable whose header contains columns from the wrong table.
> >
> > This also affects small messages on the Netty event loop
> >
> > Andrés, your investigation focused on wide tables where messages exceed
> the ~64KB large-message threshold and are deserialized on SEPWorker
> threads. We found that the same contamination also occurs with small
> messages deserialized on the Netty event loop.
> >
> > For messages under 64KB, processSmallMessage() deserializes the payload
> inline on the event loop thread, which has its own
> TinyThreadLocalPool<FastBuilder>. Since Netty binds each channel to a
> single EventLoop, messages from the same peer are handled by the same
> thread — making thread reuse virtually guaranteed rather than probabilistic.
> >
> > This lowers the trigger threshold significantly: the source table only
> needs more than 31 columns (for FastBuilder overflow) rather than the ~4200
> needed to exceed the large-message threshold. In our case, a 150-column
> table was the contamination source. The 29 foreign columns we observed are
> consistent with the 31 + 1 items retained in savedBuffer/savedNextKey,
> minus a few consumed as internal BTree node keys during build().
> >
> > Summary
> >
> > We strongly support the proposed fix to clear savedBuffer and
> savedNextKey in FastBuilder.reset(). Beyond the ClassCastException that
> Andrés identified, the same bug can cause the silent SSTable header
> contamination tracked in CASSANDRA-21260. We have written JVM dtests
> reproducing both the large-message and small-message contamination paths
> and are happy to share them.
> >
> > Best regards
> > Runtian
>
Re: [DISCUSS] Stale FastBuilder objects can cause ClassCastException on read and write requests (CASSANDRA-21216)
Posted by Štefan Miklošovič <sm...@apache.org>.
On Fri, Apr 10, 2026 at 9:37 AM Runtian Liu <cu...@gmail.com> wrote:
>
> Hi Andrés, Isaac,
>
> Thank you for the detailed write-up, Andrés. Your investigation into the FastBuilder.reset() bug was the starting point for our own analysis, which led us to identify an additional impact beyond the ClassCastException.
>
> Isaac — yes, we believe CASSANDRA-21260 and CASSANDRA-21216 are directly related. CASSANDRA-21260 was filed by our team to track the SSTable header contamination we've been seeing. Based on Andrés' findings about the stale savedBuffer/savedNextKey in FastBuilder.reset(), we investigated whether the same bug could explain our corrupted SSTable headers — and we believe it does.
>
> What we observed (CASSANDRA-21260)
>
> We have been seeing corrupted SSTable headers where an SSTable for one table contains column metadata belonging to a completely different table. When we deserialize the on-disk SerializationHeader.Component and compare it against the table's TableMetadata, we find column names that are not part of the table's schema — they belong to another table in the same keyspace. In one case, a table with ~2000 columns had 29 foreign columns from a ~150-column table embedded in its SSTable header.
>
> These corrupted SSTables are otherwise structurally valid — they are accepted into the live set and only detected by explicit header validation we added. The foreign columns do not correspond to dropped columns or any prior schema version of the affected table. As noted in CASSANDRA-21260, once a corrupted SSTable exists, compaction merges headers blindly, so the contamination propagates to new SSTables indefinitely.
I do not want to be a scope creep here, but there is also
CASSANDRA-21000 which will keep deleted columns in there forever. I do
not see any issue with fixing it (details in the ticket), but at the
same time I can not say 100% that it will not have any side-effects we
did not count on.
However, if we change the logic / do some fixes around
SerializationHeader, I think it would be great to think about the
inclusion of this ticket as well / to have it in mind.
> How the FastBuilder bug (CASSANDRA-21216) causes this
>
> Building on Andrés' analysis of the FastBuilder state leakage, we traced a path from the stale savedBuffer/savedNextKey all the way to on-disk SSTable header contamination:
>
> 1. A schema disagreement (e.g. during column addition) causes an internode READ_REQ deserialization to fail on a replica. Columns.Serializer.deserialize() uses a thread-local pooled FastBuilder, and if the table has more than 31 columns, the overflow populates savedBuffer and savedNextKey before the exception. Since reset() does not clear these fields, the FastBuilder is returned to the pool with stale ColumnMetadata from the source table.
>
> 2. When a deletion-only mutation (partition delete or range tombstone) for a different table is later deserialized on the same thread, Columns.Serializer.deserialize() acquires the poisoned FastBuilder. The stale ColumnMetadata from the source table are drained into the victim table's Columns via propagateOverflow(). Because the mutation contains only a deletion — no rows, no static row — no per-row column-subset deserialization occurs, so the contaminated Columns survives without error. (Mutations with actual row data would fail due to subset encoding mismatches, which is why only deletion-only mutations propagate the contamination silently.)
>
> When the contaminated PartitionUpdate is applied to the memtable, ColumnsCollector.update() records the foreign ColumnMetadata. At flush, BigTableWriter.openFinal() writes the SSTable using the in-memory SerializationHeader directly, bypassing toHeader() validation. The result is an on-disk SSTable whose header contains columns from the wrong table.
>
> This also affects small messages on the Netty event loop
>
> Andrés, your investigation focused on wide tables where messages exceed the ~64KB large-message threshold and are deserialized on SEPWorker threads. We found that the same contamination also occurs with small messages deserialized on the Netty event loop.
>
> For messages under 64KB, processSmallMessage() deserializes the payload inline on the event loop thread, which has its own TinyThreadLocalPool<FastBuilder>. Since Netty binds each channel to a single EventLoop, messages from the same peer are handled by the same thread — making thread reuse virtually guaranteed rather than probabilistic.
>
> This lowers the trigger threshold significantly: the source table only needs more than 31 columns (for FastBuilder overflow) rather than the ~4200 needed to exceed the large-message threshold. In our case, a 150-column table was the contamination source. The 29 foreign columns we observed are consistent with the 31 + 1 items retained in savedBuffer/savedNextKey, minus a few consumed as internal BTree node keys during build().
>
> Summary
>
> We strongly support the proposed fix to clear savedBuffer and savedNextKey in FastBuilder.reset(). Beyond the ClassCastException that Andrés identified, the same bug can cause the silent SSTable header contamination tracked in CASSANDRA-21260. We have written JVM dtests reproducing both the large-message and small-message contamination paths and are happy to share them.
>
> Best regards
> Runtian
Re: [DISCUSS] Stale FastBuilder objects can cause ClassCastException on read and write requests (CASSANDRA-21216)
Posted by Runtian Liu <cu...@gmail.com>.
Hi Andrés, Isaac,
Thank you for the detailed write-up, Andrés. Your investigation into the
FastBuilder.reset() bug was the starting point for our own analysis, which
led us to identify an additional impact beyond the ClassCastException.
Isaac — yes, we believe CASSANDRA-21260 and CASSANDRA-21216 are directly
related. CASSANDRA-21260 was filed by our team to track the SSTable header
contamination we've been seeing. Based on Andrés' findings about the stale
savedBuffer/savedNextKey in FastBuilder.reset(), we investigated whether
the same bug could explain our corrupted SSTable headers — and we believe
it does.
What we observed (CASSANDRA-21260)
We have been seeing corrupted SSTable headers where an SSTable for one
table contains column metadata belonging to a completely different table.
When we deserialize the on-disk SerializationHeader.Component and compare
it against the table's TableMetadata, we find column names that are not
part of the table's schema — they belong to another table in the same
keyspace. In one case, a table with ~2000 columns had 29 foreign columns
from a ~150-column table embedded in its SSTable header.
These corrupted SSTables are otherwise structurally valid — they are
accepted into the live set and only detected by explicit header validation
we added. The foreign columns do not correspond to dropped columns or any
prior schema version of the affected table. As noted in CASSANDRA-21260,
once a corrupted SSTable exists, compaction merges headers blindly, so the
contamination propagates to new SSTables indefinitely.
How the FastBuilder bug (CASSANDRA-21216) causes this
Building on Andrés' analysis of the FastBuilder state leakage, we traced a
path from the stale savedBuffer/savedNextKey all the way to on-disk SSTable
header contamination:
1. A schema disagreement (e.g. during column addition) causes an internode
READ_REQ deserialization to fail on a replica.
Columns.Serializer.deserialize() uses a thread-local pooled FastBuilder,
and if the table has more than 31 columns, the overflow populates
savedBuffer and savedNextKey before the exception. Since reset() does not
clear these fields, the FastBuilder is returned to the pool with stale
ColumnMetadata from the source table.
2. When a deletion-only mutation (partition delete or range tombstone) for
a different table is later deserialized on the same thread,
Columns.Serializer.deserialize() acquires the poisoned FastBuilder. The
stale ColumnMetadata from the source table are drained into the victim
table's Columns via propagateOverflow(). Because the mutation contains only
a deletion — no rows, no static row — no per-row column-subset
deserialization occurs, so the contaminated Columns survives without error.
(Mutations with actual row data would fail due to subset encoding
mismatches, which is why only deletion-only mutations propagate the
contamination silently.)
When the contaminated PartitionUpdate is applied to the memtable,
ColumnsCollector.update() records the foreign ColumnMetadata. At flush,
BigTableWriter.openFinal() writes the SSTable using the in-memory
SerializationHeader directly, bypassing toHeader() validation. The result
is an on-disk SSTable whose header contains columns from the wrong table.
This also affects small messages on the Netty event loop
Andrés, your investigation focused on wide tables where messages exceed the
~64KB large-message threshold and are deserialized on SEPWorker threads. We
found that the same contamination also occurs with small messages
deserialized on the Netty event loop.
For messages under 64KB, processSmallMessage() deserializes the payload
inline on the event loop thread, which has its own
TinyThreadLocalPool<FastBuilder>. Since Netty binds each channel to a
single EventLoop, messages from the same peer are handled by the same
thread — making thread reuse virtually guaranteed rather than probabilistic.
This lowers the trigger threshold significantly: the source table only
needs more than 31 columns (for FastBuilder overflow) rather than the ~4200
needed to exceed the large-message threshold. In our case, a 150-column
table was the contamination source. The 29 foreign columns we observed are
consistent with the 31 + 1 items retained in savedBuffer/savedNextKey,
minus a few consumed as internal BTree node keys during build().
Summary
We strongly support the proposed fix to clear savedBuffer and savedNextKey
in FastBuilder.reset(). Beyond the ClassCastException that Andrés
identified, the same bug can cause the silent SSTable header contamination
tracked in CASSANDRA-21260. We have written JVM dtests reproducing both the
large-message and small-message contamination paths and are happy to share
them.
Best regards
Runtian
>
Re: [DISCUSS] Stale FastBuilder objects can cause ClassCastException on read and write requests (CASSANDRA-21216)
Posted by Isaac Reath <is...@gmail.com>.
I also wonder if this could be related to
https://issues.apache.org/jira/browse/CASSANDRA-21260 as well? It is
produced in a similar way, a schema disagreement happens, then, later on,
resulting SSTables have bad metadata in them. If the FastBuilder is reused
without being cleaned up, I could see how this bad SSTable metadata could
end up being created.
On Wed, Apr 8, 2026 at 5:57 PM Andrés Beck-Ruiz <an...@gmail.com>
wrote:
> Hi all,
>
> I’d like to discuss my investigation into CASSANDRA-21216
> <https://issues.apache.org/jira/browse/CASSANDRA-21216>, a bug that my
> team encountered during schema modification on a very wide table (~4200
> columns) in a 4.1 Cassandra cluster that was also serving reads and writes.
>
> *Events required to reproduce:*
>
> We have found that this bug triggers in the following scenario:
>
> - There is a schema disagreement between a coordinator node and a replica
> node when a new column is added:
> - The coordinator node is performing a read request for the newly
> added column
> - The replica node does not have the new column yet
> - The size of the internode READ_REQ is greater than ~ 65 KB
> - The thread that fails to deserialize the READ_REQ is later reused by the
> MutationStage
> - The internode READ_REQ contains more than 31 columns (populates
> savedBuffer and savedNextKey during deserialization)
> - There are no more than 31 rows in-memory for a partition being mutated
> (Row BTree has one leaf node)
> - No more than 31 rows are being added in the mutation (PartitionUpdate
> BTree has one leaf node)
>
> *Why this bug surfaces: *
>
> This bug stems from the schema disagreement described above while adding
> new columns to a table, which causes internode message deserialization to
> fail. When a replica node does not find a column included in the READ_REQ
> message, it throws this exception
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/Columns.java#L489>,
> causing the original request to fail. As a result, the BTree FastBuilder
> build function
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/Columns.java#L493>
> is never reached. On the happy path, the build function calls
> completeBuild
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/utils/btree/BTree.java#L2420>,
> which consumes savedBuffer and savedNextKey. These FastBuilder objects are
> used to create multi-level BTrees and are populated when a tree under
> construction exceeds 31 elements.
>
> When an exception is thrown, the build method is not called. As a result,
> these fields retain their stale ColumnMetadata values. Although the
> FastBuilder does call a reset
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/utils/btree/BTree.java#L3325>
> function when it closes the object, the function does not clear out the
> savedBuffer or savedNextKey, unlike the AbstractUpdater reset function
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/utils/btree/BTree.java#L3371-L3372>.
> It is important that these objects are cleared out after use given that
> they are shared locally per thread
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/utils/btree/BTree.java#L2313>.
>
>
> When Cassandra deserializes large messages, deserialization is dispatched
> to an SEPWorker thread pool
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/net/InboundMessageHandler.java#L200>instead
> of being handled in the Netty event loop. For our case, we observed that
> internode READ_REQ messages after expanding SELECT * queries on the 4200
> column table were passing the 65 KB threshold. If the same thread that
> handled a failed message deserialization then handles a mutation request,
> it might use the same FastBuilder with stale ColumnMetadata objects to merge
> the existing BTree of Rows in a partition with the partition update
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java#L148>.
>
>
> If the existing BTree of Rows and partition update are leaf nodes, the
> BTree update function uses the FastBuilder instead of the Updater
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/utils/btree/BTree.java#L391-L393>
> to rebuild the tree. The FastBuilder then empties out the ColumnMetadata
> objects in the savedBuffer and savedNextKey into the BTree that contains
> Row objects. The ClassCastException surfaces when we read or write from the
> partition in memory with the corrupted BTree.
>
> I found that the same bug could resurface in trunk, given that the
> deserialization of large messages is still handled on the SEPWorker thread
> pool, and the FastBuilder reset function has not been updated to clear
> savedBuffer and savedNextKey. However, I’m not sure whether TCM will reduce
> or eliminate the possibility of schema mismatches across a cluster.
>
> More information about how we verified this theory, as well as the full
> stack traces, can be found on the bug ticket.
>
> *Proposed fixes: *
>
> A possible fix to this issue that I’ve added here
> <https://github.com/andresbeckruiz/cassandra/commit/14dbac67bee3917ce71cd18dc48ef19f5f0cf649>
> and verified builds successfully on Cassandra 4.1 is to clear the
> savedBuffer and savedNextKey in the reset function, as is done in the
> AbstractUpdater; I also noticed that AbstractUpdater does not set
> savedNextKey to null, so this could be worth adding as well.
>
> Another possible fix could be to handle both small and large messages in
> the Netty event loop. If the FastBuilder objects are not cleared out, this
> will guarantee that the FastBuilder is not reused by SEPWorker threads,
> which could potentially corrupt Row BTrees during a MutationStage. However,
> this could possibly block the event loop thread.
>
> Given that this patch affects critical paths in Cassandra, I would
> appreciate any feedback on any potential side-effects I have not
> considered. If the approach to fixing this bug seems reasonable, I can also
> add a DTest to mimic this edge case.
>
> Best,
> Andrés
>