You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Dmitry Konstantinov (Jira)" <ji...@apache.org> on 2025/01/30 12:34:00 UTC
[jira] [Comment Edited] (CASSANDRA-20052) Size of CQL messages is not limited in V5 protocol logic
[ https://issues.apache.org/jira/browse/CASSANDRA-20052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922359#comment-17922359 ]
Dmitry Konstantinov edited comment on CASSANDRA-20052 at 1/30/25 12:33 PM:
---------------------------------------------------------------------------
Hi [~m.sherstyuk], JFI, we have an on-going dev group discussion now about Config.java parameters documenting in general: [https://lists.apache.org/thread/ror3ynqthssf4yghq3o2hlbd9jyx8y0l]
As a part of the current ticket the following single parameter was added to cassandra.yaml: _native_transport_max_message_size_
the default value for it:
{code:java}
min(
max_mutation_size,
native_transport_max_request_data_in_flight,
native_transport_max_request_data_in_flight_per_ip
)
{code}
where
{code:java}
max_mutation_size default = commitlog_segment_size / 2 = 32MiB / 2 = 16 MiB
native_transport_max_request_data_in_flight default = Xmx/10
native_transport_max_request_data_in_flight_per_ip default = Xmx/40{code}
so, if Xmx >= 1024M then native_transport_max_message_size by default will be 16 MiB. Taking in account what in any case such big messages will be rejected during a mutation creation - the default is not more restrictive than the older logic implicitly had, we just start to fail earlier.
Regarding impact on existing highload clusters, I did not notice an impact from this change in synthetic stress test cases (like the test mentioned here CASSANDRA-20165) but if a processing of messages takes more time - it may be an impact from fixing of rate limiting logic introduced in CASSANDRA-15013. Before the current ticket the actual rate limiting was not applied in cases when messages are more than a frame size = 131071 bytes and the amount of in-flight messages could be higher than expected and defined by native_transport_max_request_data_in_flight/native_transport_max_request_data_in_flight_per_ip.
If you face such case you can mitigate it by increasing native_transport_max_request_data_in_flight/native_transport_max_request_data_in_flight_per_ip values if you feel that it is ok to give more memory for in-flight messages (be careful, it can cause an OOM if you relax the thresholds too much).
was (Author: dnk):
Hi [~m.sherstyuk], JFI, we have an on-going dev group discussion now about Config.java parameters documenting in general: [https://lists.apache.org/thread/ror3ynqthssf4yghq3o2hlbd9jyx8y0l]
As a part of the current ticket the following single parameter was added to cassandra.yaml: _native_transport_max_message_size_
the default value for it:
{code:java}
min(
max_mutation_size,
native_transport_max_request_data_in_flight,
native_transport_max_request_data_in_flight_per_ip
)
{code}
where
{code:java}
max_mutation_size default = commitlog_segment_size / 2 = 32MiB / 2 = 16 MiB
native_transport_max_request_data_in_flight default = Xmx/10
native_transport_max_request_data_in_flight_per_ip default = Xmx/40{code}
so, if Xmx >= 1024M then native_transport_max_message_size by default will be 16 MiB. Taking in account what in any case such big messages will be rejected during a mutation creation - the default is not more restrictive than the older logic implicitly had, we just start to fail earlier.
Regarding impact on existing highload clusters, I did not notice an impact from this change in synthetic stress test cases (like the test mentioned here CASSANDRA-20165) but if a processing of messages takes more time - it may be an impact from fixing of rate limiting logic introduced in CASSANDRA-15013. Before the current ticket the actual rate limiting was not applied in cases when messages are more than a frame size = 131071 bytes and the amount of in-flight messages could be higher than expected and defined by native_transport_max_request_data_in_flight/native_transport_max_request_data_in_flight_per_ip.
If you face such case you can mitigate by increasing native_transport_max_request_data_in_flight/native_transport_max_request_data_in_flight_per_ip values if you feel that it is ok to give more memory for in-flight messages (be careful, it can cause an OOM if you relax the thresholds too much).
> Size of CQL messages is not limited in V5 protocol logic
> --------------------------------------------------------
>
> Key: CASSANDRA-20052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20052
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Messaging/Client
> Reporter: Dmitry Konstantinov
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Fix For: 4.1.8, 5.0.3, 5.1
>
> Attachments: cassandra_rate_limit.svg, ci_summary-4.1.html, ci_summary-5.0.html, ci_summary-trunk.html, ci_summary.html
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Size of CQL messages is not limited in V5 protocol logic
> - After introducing of v5 frames we do not have any CQL message limit anymore, native_transport_max_frame_size_in_mb which had such limit in pre-V5 epoch is applicable now only to pre-V5 protocol sessions, otherwise it is applied only to the initial STARTUP/OPTIONS messages handling, it is not checked in any v5 logic. So, currently a v5 CQL message of any size can be sent to Cassandra server.
> - The overload logic just allows to process huge messages for free to avoid starvation, so it does not provide any protection against the most dangerous requests from a memory pressure point of view.
> - The situation even more dangerous: the v5 framing logic is enabled just after AUTH response, so we do not limit message size even for AUTH_RESPONSE messages from a client. It can be used as a DoS attack: a non-authenticated client can send a huge username/password to Cassandra server to cause troubles with GC or even kill it.
> An easy example:
> {code:java}
> public class TestBigAuthRequest {
> public static void main(String[] args) {
> String password = getString(500_000_000, '-');
> try (CqlSession session = CqlSession.builder()
> .addContactEndPoint(new DefaultEndPoint(new InetSocketAddress("localhost", 9042)))
> .withAuthCredentials("cassandra", password)
> .withLocalDatacenter("datacenter1")
> .build()) {
> session.execute("select * from system.local");
> }
> }
> private static String getString(int length, char charToFill) {
> if (length > 0) {
> char[] array = new char[length];
> Arrays.fill(array, charToFill);
> return new String(array);
> }
> return "";
> }
> }
> {code}
> A thread stack of such invocation (captured to show the execution flow):
> {code:java}
> "nioEventLoopGroup-5-21@9164" prio=10 tid=0x86 nid=NA runnable
> java.lang.Thread.State: RUNNABLE
> at org.apache.cassandra.transport.messages.AuthResponse$1.decode(AuthResponse.java:45)
> at org.apache.cassandra.transport.messages.AuthResponse$1.decode(AuthResponse.java:39)
> at org.apache.cassandra.transport.Message$Decoder.decodeMessage(Message.java:432)
> at org.apache.cassandra.transport.Message$Decoder$RequestDecoder.decode(Message.java:467)
> at org.apache.cassandra.transport.Message$Decoder$RequestDecoder.decode(Message.java:459)
> at org.apache.cassandra.transport.CQLMessageHandler.processRequest(CQLMessageHandler.java:377)
> at org.apache.cassandra.transport.CQLMessageHandler$LargeMessage.onComplete(CQLMessageHandler.java:755)
> at org.apache.cassandra.net.AbstractMessageHandler$LargeMessage.supply(AbstractMessageHandler.java:561)
> at org.apache.cassandra.net.AbstractMessageHandler.processSubsequentFrameOfLargeMessage(AbstractMessageHandler.java:257)
> at org.apache.cassandra.net.AbstractMessageHandler.processIntactFrame(AbstractMessageHandler.java:229)
> at org.apache.cassandra.net.AbstractMessageHandler.process(AbstractMessageHandler.java:216)
> at org.apache.cassandra.transport.CQLMessageHandler.process(CQLMessageHandler.java:147)
> at org.apache.cassandra.net.FrameDecoder.deliver(FrameDecoder.java:330)
> at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:294)
> at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:277)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:829)
> {code}
> The provided MR (https://github.com/apache/cassandra/pull/3655) contains a fix for the issue which introduces 2 new parameters:
> native_transport_max_message_size - to limit any CQL message size
> native_transport_max_auth_message_size (default = 128KiB) - to limit auth response message size more strictly and add an extra protection against a possible DoS attack.
> Design questions:
> * The current implementation closes a CQL connection if a message is bigger than the limits. A skip message body logic can be implemented to continue the connection usage but it is more complicated and error prone.
> * The tricky question is the default value for native_transport_max_message_size,
> from one side - we want to have it not more than min(native_transport_max_request_data_in_flight_per_ip, native_transport_max_request_data_in_flight) to reduce chances to invoke the branch of logic when a error handling does not work
> from another size - min(native_transport_max_request_data_in_flight_per_ip, native_transport_max_request_data_in_flight) can be too small and there is a chance to break a backward compatibility for existing deployments where people use large messages and small heaps (while it is not a good idea).
> Related observations:
> 1) https://issues.apache.org/jira/browse/CASSANDRA-16886 - Reduce native_transport_max_frame_size_in_mb (from 256M to 16M)
> 2) A correspondent logic for Cassandra server internode protocol a message limit exists and rate limiting parameters are validated to be smaller than a single message max size:
> internode_max_message_size = min(internode_application_receive_queue_reserve_endpoint_capacity, internode_application_send_queue_reserve_endpoint_capacity)
> internode_application_receive_queue_reserve_endpoint_capacity = 128MiB
> internode_application_send_queue_reserve_endpoint_capacity = 128MiB
> internode_max_message_size <= internode_application_receive_queue_reserve_endpoint_capacity
> internode_max_message_size <= internode_application_receive_queue_reserve_global_capacity
> internode_max_message_size <= internode_application_send_queue_reserve_endpoint_capacity
> internode_max_message_size <= internode_application_send_queue_reserve_global_capacity
> 3) Request types according to CQL specification:
> 4.1.1. STARTUP, in normal cases should be small
> 4.1.2. AUTH_RESPONSE, in normal cases should be small
> 4.1.3. OPTIONS, in normal cases should be small
> 4.1.4. QUERY, in normal cases should be small
> 4.1.5. PREPARE, in normal cases should be small
> 4.1.6. EXECUTE <-- potentially large in case of inserts, max_mutation_size = commitlog_segment_size / 2; where commitlog_segment_size_in_mb = 32MiB
> 4.1.7. BATCH <-- potentially large, max_mutation_size = commitlog_segment_size / 2; where commitlog_segment_size_in_mb = 32MiB
> 4.1.8. REGISTER, in normal cases should be small
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org