You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "DanielWang2035 (via GitHub)" <gi...@apache.org> on 2023/07/13 14:02:22 UTC

[GitHub] [iotdb] DanielWang2035 opened a new pull request, #10550: [IOTDB-6067] Pipe: limit retry count when connection is lost in connector-v2

DanielWang2035 opened a new pull request, #10550:
URL: https://github.com/apache/iotdb/pull/10550

   Currently, iotdb-thrift-connector-v2 will keep retrying when the connection to receiver is lost, and the failure won't be known by PipeAgent.
   Fix:
   - move retry threads into one thread pool
   - if retry count exceeds limit, set the pipe status in current datanode to STOPPED.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [iotdb] SteveYurongSu commented on a diff in pull request #10550: [IOTDB-6067] Pipe: limit retry count when connection is lost in connector-v2

Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on code in PR #10550:
URL: https://github.com/apache/iotdb/pull/10550#discussion_r1271824150


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/task/subtask/connector/PipeConnectorSubtask.java:
##########
@@ -67,6 +69,11 @@ public PipeConnectorSubtask(
     super(taskID);
     this.inputPendingQueue = inputPendingQueue;
     this.outputPipeConnector = outputPipeConnector;
+    if (outputPipeConnector instanceof IoTDBThriftConnectorV2) {
+      retryPendingQueue = ((IoTDBThriftConnectorV2) outputPipeConnector).getRetryPendingQueue();

Review Comment:
   Try not to modify this class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [iotdb] SteveYurongSu commented on a diff in pull request #10550: [IOTDB-6067] Pipe: limit retry count when connection is lost in connector-v2

Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on code in PR #10550:
URL: https://github.com/apache/iotdb/pull/10550#discussion_r1264950817


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/PipeTaskAgent.java:
##########
@@ -315,6 +318,38 @@ private void dropAllPipeTasksWithoutLock() {
     }
   }
 
+  public synchronized void stopAllPipesWithCriticalException() {
+    acquireWriteLock();
+    try {
+      stopAllPipesWithCriticalExceptionInternal();
+    } finally {
+      releaseWriteLock();
+    }
+  }
+
+  private void stopAllPipesWithCriticalExceptionInternal() {
+    pipeMetaKeeper
+        .getPipeMetaList()
+        .forEach(
+            pipeMeta ->
+                pipeMeta
+                    .getRuntimeMeta()
+                    .getConsensusGroupId2TaskMetaMap()
+                    .values()
+                    .forEach(
+                        pipeTaskMeta -> {
+                          Iterator<PipeRuntimeException> exceptionIterator =
+                              pipeTaskMeta.getExceptionMessages().iterator();
+                          while (exceptionIterator.hasNext()) {
+                            if (exceptionIterator.next() instanceof PipeRuntimeCriticalException) {
+                              stopPipe(

Review Comment:
   check before stop



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [iotdb] DanielWang2035 commented on a diff in pull request #10550: [IOTDB-6067] Pipe: limit retry count when connection is lost in connector-v2

Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #10550:
URL: https://github.com/apache/iotdb/pull/10550#discussion_r1265194725


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/PipeTaskAgent.java:
##########
@@ -315,6 +318,38 @@ private void dropAllPipeTasksWithoutLock() {
     }
   }
 
+  public synchronized void stopAllPipesWithCriticalException() {
+    acquireWriteLock();
+    try {
+      stopAllPipesWithCriticalExceptionInternal();
+    } finally {
+      releaseWriteLock();
+    }
+  }
+
+  private void stopAllPipesWithCriticalExceptionInternal() {
+    pipeMetaKeeper
+        .getPipeMetaList()
+        .forEach(
+            pipeMeta ->
+                pipeMeta
+                    .getRuntimeMeta()
+                    .getConsensusGroupId2TaskMetaMap()
+                    .values()
+                    .forEach(
+                        pipeTaskMeta -> {
+                          Iterator<PipeRuntimeException> exceptionIterator =
+                              pipeTaskMeta.getExceptionMessages().iterator();
+                          while (exceptionIterator.hasNext()) {
+                            if (exceptionIterator.next() instanceof PipeRuntimeCriticalException) {
+                              stopPipe(

Review Comment:
   changed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [iotdb] SteveYurongSu commented on a diff in pull request #10550: [IOTDB-6067] Pipe: limit retry count when connection is lost in connector-v2

Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on code in PR #10550:
URL: https://github.com/apache/iotdb/pull/10550#discussion_r1271824546


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/v2/IoTDBThriftConnectorV2.java:
##########
@@ -76,6 +80,13 @@ public class IoTDBThriftConnectorV2 implements PipeConnector {
 
   private static final CommonConfig COMMON_CONFIG = CommonDescriptor.getInstance().getConfig();
 
+  private final BoundedBlockingPendingQueue<Event> retryPendingQueue =
+      new BoundedBlockingPendingQueue<>(1024);
+
+  private static final ExecutorService transferRetryExecutor =

Review Comment:
   double check init



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [iotdb] SteveYurongSu merged pull request #10550: [IOTDB-6067] Pipe: Improve the stability of iotdb-thrift-connector-v2 during fault tolerance (avoid OOM)

Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu merged PR #10550:
URL: https://github.com/apache/iotdb/pull/10550


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org