[GitHub] [kafka] guozhangwang commented on a change in pull request #8856: KAFKA-10150: task state transitions/management and committing cleanup

[GitBox <gi...@apache.org>]

guozhangwang commented on a change in pull request #8856:
URL: https://github.com/apache/kafka/pull/8856#discussion_r439493219



##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java
##########
@@ -220,12 +215,18 @@ public void handleAssignment(final Map<TaskId, Set<TopicPartition>> activeTasks,
             } else {
                 try {
                     task.suspend();
-                    final Map<TopicPartition, OffsetAndMetadata> committableOffsets = task.prepareCommit();
-
-                    tasksToClose.add(task);
-                    if (!committableOffsets.isEmpty()) {
-                        consumedOffsetsAndMetadataPerTask.put(task.id(), committableOffsets);
+                    if (task.commitNeeded()) {
+                        if (task.isActive()) {
+                            log.error("Active task {} was revoked and should have already been committed", task.id());
+                            throw new IllegalStateException("Revoked active task was not committed during handleRevocation");
+                        } else {
+                            task.prepareCommit();
+                            task.postCommit();
+                        }
                     }
+                    completeTaskCloseClean(task);

Review comment:
       When closing-clean a standby task, we would checkpoint the file and close the state store which would also flush it as well, so I think we do not need to call
   
   ```
   task.prepareCommit();
   task.postCommit();
   ```
   
   which is just to flush the stores and write checkpoint files, right?

##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java
##########
@@ -440,41 +402,35 @@ boolean tryToCompleteRestoration() {
      * @throws TaskMigratedException if the task producer got fenced (EOS only)
      */
     void handleRevocation(final Collection<TopicPartition> revokedPartitions) {
-        final Set<TopicPartition> remainingPartitions = new HashSet<>(revokedPartitions);
+        final Set<TopicPartition> remainingRevokedPartitions = new HashSet<>(revokedPartitions);

Review comment:
       nit: Add in the above javadoc that we should only revoke active tasks here?

##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java
##########
@@ -220,12 +215,18 @@ public void handleAssignment(final Map<TaskId, Set<TopicPartition>> activeTasks,
             } else {
                 try {
                     task.suspend();
-                    final Map<TopicPartition, OffsetAndMetadata> committableOffsets = task.prepareCommit();
-
-                    tasksToClose.add(task);
-                    if (!committableOffsets.isEmpty()) {
-                        consumedOffsetsAndMetadataPerTask.put(task.id(), committableOffsets);
+                    if (task.commitNeeded()) {

Review comment:
       nit: add a comment above `task.suspend()` that for active it should always be an no-op?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org