Re: [PR] Introduce delay analyzer to get watermark between sequence data and unsequece data based on ICDE 2022 [iotdb]

["jt2594838 (via GitHub)" <gi...@apache.org>]

jt2594838 commented on code in PR #16886:
URL: https://github.com/apache/iotdb/pull/16886#discussion_r2757682625


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java:
##########
@@ -1160,6 +1171,12 @@ public void insert(InsertRowNode insertRowNode) throws WriteProcessException {
       if (deleted) {
         return;
       }
+      long arrivalTime = System.currentTimeMillis();
+      long generationTime = insertRowNode.getTime();
+      if (delayAnalyzer != null) {
+        delayAnalyzer.update(generationTime, arrivalTime);
+      }

Review Comment:
   Put the first lines into the if-clause.
   The same below.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java:
##########
@@ -1144,6 +1119,51 @@ public void loadProperties(TrimProperties properties) throws BadNodeUrlException
             properties.getProperty(
                 "include_null_value_in_write_throughput_metric",
                 String.valueOf(conf.isIncludeNullValueInWriteThroughputMetric()))));
+
+    conf.setEnableDelayAnalyzer(
+        Boolean.parseBoolean(
+            properties.getProperty(
+                "enable_delay_analyzer",
+                ConfigurationFileUtils.getConfigurationDefaultValue("enable_delay_analyzer"))));
+
+    int delayAnalyzerWindowSize =
+        Integer.parseInt(
+            properties.getProperty(
+                "delay_analyzer_window_size",
+                ConfigurationFileUtils.getConfigurationDefaultValue("delay_analyzer_window_size")));
+    if (delayAnalyzerWindowSize > 0) {
+      conf.setDelayAnalyzerWindowSize(delayAnalyzerWindowSize);

Review Comment:
   May print a warn log



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java:
##########
@@ -398,6 +394,24 @@ public class IoTDBConfig {
   /** The interval to check whether unsequence memtables need flushing. Unit: ms */
   private long unseqMemtableFlushCheckInterval = 30 * 1000L;
 
+  /**
+   * Whether to enable delay analyzer for tracking data arrival delays and calculating safe
+   * watermarks
+   */
+  private boolean enableDelayAnalyzer = true;

Review Comment:
   Default to false



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org