You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/03/21 13:05:18 UTC

[GitHub] [incubator-inlong] zk1510 opened a new pull request #3279: [INLONG-3274][Agent] when kafka topic deleted,then chang task status …

zk1510 opened a new pull request #3279:
URL: https://github.com/apache/incubator-inlong/pull/3279


   …to failed
   
   
   [Agent] when kafka topic deleted,then chang task status to failed
   
   Fixes #3274
   
   ### Motivation
   when kafka topic deleted,then chang task status to failed
   ### Modifications
   when kafka topic deleted,then chang task status to failed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-inlong] zk1510 commented on a change in pull request #3279: [INLONG-3274][Agent] Change the task status to failed when the Kafka topic was deleted

Posted by GitBox <gi...@apache.org>.
zk1510 commented on a change in pull request #3279:
URL: https://github.com/apache/incubator-inlong/pull/3279#discussion_r831712865



##########
File path: inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/KafkaReader.java
##########
@@ -143,7 +148,9 @@ public Message read() {
             }
         } else {
             // commit offset
-            consumer.commitAsync();
+            if (!isSourceExist()) {

Review comment:
       you shou see the method isSourceExist()




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-inlong] healchow commented on a change in pull request #3279: [INLONG-3274][Agent] Change the task status ro failed when the Kafka topic was deleted

Posted by GitBox <gi...@apache.org>.
healchow commented on a change in pull request #3279:
URL: https://github.com/apache/incubator-inlong/pull/3279#discussion_r831090578



##########
File path: inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/task/TaskWrapper.java
##########
@@ -79,20 +84,29 @@ public TaskWrapper(AgentManager manager, Task task) {
      * @return CompletableFuture
      */
     private CompletableFuture<?> submitReadThread() {
-        return CompletableFuture.runAsync(() -> {
-            Message message = null;
-            while (!isException() && !task.isReadFinished()) {
-                if (message == null || task.getChannel()
-                        .push(message, pushMaxWaitTime, TimeUnit.SECONDS)) {
-                    message = task.getReader().read();
-                }
+    return CompletableFuture.runAsync(

Review comment:
       The indentation of the code doesn't look right.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-inlong] dockerzhang commented on pull request #3279: [INLONG-3274][Agent] Change the task status to failed when the Kafka topic was deleted

Posted by GitBox <gi...@apache.org>.
dockerzhang commented on pull request #3279:
URL: https://github.com/apache/incubator-inlong/pull/3279#issuecomment-1073998462


   @EMsnap PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-inlong] EMsnap merged pull request #3279: [INLONG-3274][Agent] Change the task status to failed when the Kafka topic was deleted

Posted by GitBox <gi...@apache.org>.
EMsnap merged pull request #3279:
URL: https://github.com/apache/incubator-inlong/pull/3279


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-inlong] EMsnap commented on a change in pull request #3279: [INLONG-3274][Agent] Change the task status to failed when the Kafka topic was deleted

Posted by GitBox <gi...@apache.org>.
EMsnap commented on a change in pull request #3279:
URL: https://github.com/apache/incubator-inlong/pull/3279#discussion_r831712088



##########
File path: inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/KafkaReader.java
##########
@@ -143,7 +148,9 @@ public Message read() {
             }
         } else {
             // commit offset
-            consumer.commitAsync();
+            if (!isSourceExist()) {

Review comment:
       when source not exist commit offset? should commit offset when source exist




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-inlong] zk1510 commented on a change in pull request #3279: [INLONG-3274][Agent] Change the task status to failed when the Kafka topic was deleted

Posted by GitBox <gi...@apache.org>.
zk1510 commented on a change in pull request #3279:
URL: https://github.com/apache/incubator-inlong/pull/3279#discussion_r831714301



##########
File path: inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/KafkaReader.java
##########
@@ -143,7 +148,9 @@ public Message read() {
             }
         } else {
             // commit offset
-            consumer.commitAsync();
+            if (!isSourceExist()) {

Review comment:
       isSourceExist() return false indicates that  source does not exist




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-inlong] EMsnap commented on a change in pull request #3279: [INLONG-3274][Agent] Change the task status to failed when the Kafka topic was deleted

Posted by GitBox <gi...@apache.org>.
EMsnap commented on a change in pull request #3279:
URL: https://github.com/apache/incubator-inlong/pull/3279#discussion_r831807330



##########
File path: inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/plugin/Reader.java
##########
@@ -63,4 +63,10 @@
      * finish read
      */
     void finishRead();
+
+    /**
+     * source deleted

Review comment:
       source is exist

##########
File path: inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/KafkaSource.java
##########
@@ -107,9 +105,15 @@ public KafkaSource() {
             // example:0#110_1#666_2#222
             partitionOffsets = allPartitionOffsets.split(JOB_OFFSET_DELIMITER);
         }
+        //
+        props.put(KAFKA_SESSION_TIMEOUT, 30000);

Review comment:
       what's the purpose of doing this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-inlong] zk1510 commented on a change in pull request #3279: [INLONG-3274][Agent] Change the task status to failed when the Kafka topic was deleted

Posted by GitBox <gi...@apache.org>.
zk1510 commented on a change in pull request #3279:
URL: https://github.com/apache/incubator-inlong/pull/3279#discussion_r831714301



##########
File path: inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/KafkaReader.java
##########
@@ -143,7 +148,9 @@ public Message read() {
             }
         } else {
             // commit offset
-            consumer.commitAsync();
+            if (!isSourceExist()) {

Review comment:
       isSourceExist() return false indicates that  source does not exist




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-inlong] zk1510 commented on a change in pull request #3279: [INLONG-3274][Agent] Change the task status to failed when the Kafka topic was deleted

Posted by GitBox <gi...@apache.org>.
zk1510 commented on a change in pull request #3279:
URL: https://github.com/apache/incubator-inlong/pull/3279#discussion_r831812345



##########
File path: inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/KafkaSource.java
##########
@@ -107,9 +105,15 @@ public KafkaSource() {
             // example:0#110_1#666_2#222
             partitionOffsets = allPartitionOffsets.split(JOB_OFFSET_DELIMITER);
         }
+        //
+        props.put(KAFKA_SESSION_TIMEOUT, 30000);

Review comment:
       set the kafka consumer session timeout




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org