You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by GitBox <gi...@apache.org> on 2020/12/01 22:17:21 UTC

[GitHub] [cassandra] aholmberg commented on a change in pull request #824: CASSANDRA-16143: Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout

aholmberg commented on a change in pull request #824:
URL: https://github.com/apache/cassandra/pull/824#discussion_r533608554



##########
File path: src/java/org/apache/cassandra/config/Config.java
##########
@@ -178,6 +178,8 @@
     // (it takes much longer than 30s) as of Linux 4.12. If you want something that high set this to 0
     // (which picks up the OS default) and configure the net.ipv4.tcp_retries2 sysctl to be ~8.
     public int internode_tcp_user_timeout_in_ms = 30000;
+    // Similar to internode_tcp_user_timeout_in_ms but used specifically for streaming connection.
+    public int internode_streaming_tcp_user_timeout_in_ms = 0;

Review comment:
       Just a nit -- I see this is following a similar pattern to the above variable, but what is the "user" meant to convey in this naming?

##########
File path: src/java/org/apache/cassandra/metrics/StreamingMetrics.java
##########
@@ -43,6 +44,8 @@
     public static final Counter totalOutgoingRepairSSTables = Metrics.counter(DefaultNameFactory.createMetricName(TYPE_NAME, "TotalOutgoingRepairSSTables", null));
     public final Counter incomingBytes;
     public final Counter outgoingBytes;
+    /* Measures the time taken for processing the incoming stream message after being deserialized, including the time to flush to disk. */
+    public final Timer incomingStreamMessageProcessTime;

Review comment:
       Should we add to metrics documentation?
   https://cassandra.apache.org/doc/latest/operating/metrics.html?highlight=metrics#streaming-metrics

##########
File path: src/java/org/apache/cassandra/streaming/StreamSession.java
##########
@@ -792,7 +792,21 @@ public void receive(IncomingStreamMessage message)
         // send back file received message
         messageSender.sendMessage(new ReceivedMessage(message.header.tableId, message.header.sequenceNumber));
         StreamHook.instance.reportIncomingStream(message.header.tableId, message.stream, this, message.header.sequenceNumber);
-        receivers.get(message.header.tableId).received(message.stream);
+        long receivedStartNanos = System.nanoTime();
+        try
+        {
+            receivers.get(message.header.tableId).received(message.stream);
+        }
+        finally
+        {
+            long latencyNanos = System.nanoTime() - receivedStartNanos;
+            metrics.incomingStreamMessageProcessTime.update(latencyNanos, TimeUnit.NANOSECONDS);
+            long latencyMs = TimeUnit.NANOSECONDS.toMillis(latencyNanos);
+            int timeout = DatabaseDescriptor.getInternodeStreamingTcpUserTimeoutInMS();
+            if (timeout > 0 && latencyMs > timeout)
+                logger.warn("Time taken ({} ms) for processing the incoming stream message ({}) exceeded internode streaming TCP user timeout ({} ms),",

Review comment:
       Is there anything we might tell the operator here about what to tune, change, or look into system-wise?

##########
File path: src/java/org/apache/cassandra/config/Config.java
##########
@@ -178,6 +178,8 @@
     // (it takes much longer than 30s) as of Linux 4.12. If you want something that high set this to 0
     // (which picks up the OS default) and configure the net.ipv4.tcp_retries2 sysctl to be ~8.
     public int internode_tcp_user_timeout_in_ms = 30000;
+    // Similar to internode_tcp_user_timeout_in_ms but used specifically for streaming connection.
+    public int internode_streaming_tcp_user_timeout_in_ms = 0;

Review comment:
       The [comment](https://github.com/apache/cassandra/blob/22abff779df097e0ef38180442e9c680b3d41187/src/java/org/apache/cassandra/config/Config.java#L43) at the top seems to indicate that JMX-modified settings should be declared volatile. Is that a concern here?

##########
File path: src/java/org/apache/cassandra/config/Config.java
##########
@@ -178,6 +178,8 @@
     // (it takes much longer than 30s) as of Linux 4.12. If you want something that high set this to 0
     // (which picks up the OS default) and configure the net.ipv4.tcp_retries2 sysctl to be ~8.
     public int internode_tcp_user_timeout_in_ms = 30000;
+    // Similar to internode_tcp_user_timeout_in_ms but used specifically for streaming connection.
+    public int internode_streaming_tcp_user_timeout_in_ms = 0;

Review comment:
       Do we need to document these in https://github.com/apache/cassandra-website/blob/trunk/src/doc/4.0-beta3/_sources/configuration/cass_yaml_file.rst.txt ? Worth adding some of your tl;dr from the ticket here, or in docs?

##########
File path: src/java/org/apache/cassandra/config/Config.java
##########
@@ -178,6 +178,8 @@
     // (it takes much longer than 30s) as of Linux 4.12. If you want something that high set this to 0
     // (which picks up the OS default) and configure the net.ipv4.tcp_retries2 sysctl to be ~8.
     public int internode_tcp_user_timeout_in_ms = 30000;
+    // Similar to internode_tcp_user_timeout_in_ms but used specifically for streaming connection.
+    public int internode_streaming_tcp_user_timeout_in_ms = 0;

Review comment:
       What does it mean for this timeout to default to zero instead of 30000 like its counterpart above?
   
   https://man7.org/linux/man-pages/man7/tcp.7.html
   > If the option value is specified as 0, TCP will use the system default.
   
   Seems sane to me, but it there a reason for the default to be different than `internode_tcp_user_timeout_in_ms`? I haven't been able to find out what the "system default" would typically be, and if it's much different from 30000. A brief look at the source gave me the impression that not setting it would actually cause this timeout mechanism to be bypassed completely, which is different behavior than we had before with a non-zero default.

##########
File path: src/java/org/apache/cassandra/metrics/StreamingMetrics.java
##########
@@ -43,6 +44,8 @@
     public static final Counter totalOutgoingRepairSSTables = Metrics.counter(DefaultNameFactory.createMetricName(TYPE_NAME, "TotalOutgoingRepairSSTables", null));
     public final Counter incomingBytes;
     public final Counter outgoingBytes;
+    /* Measures the time taken for processing the incoming stream message after being deserialized, including the time to flush to disk. */
+    public final Timer incomingStreamMessageProcessTime;

Review comment:
       Should we bootstrap a test for these metrics? I know we haven't bothered historically, but we're [trying to change that](https://issues.apache.org/jira/browse/CASSANDRA-16190) for 4.0+.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org