You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mjsax (via GitHub)" <gi...@apache.org> on 2026/06/23 04:38:53 UTC

[PR] KAFKA-20116: Forwad task-(end)-offset-sum to task assignor (4/N) [kafka]

mjsax opened a new pull request, #22644:
URL: https://github.com/apache/kafka/pull/22644

   This PR adds in-memory state to store task-offset-sum and task-end-offset-sum, and wiring to forward both fields into the "streams" task assignor.
   
   We also ungate the warmup-tasks field on the heartbeat request handler. The wiring to forward warmup-tasks field existed already.
   
   Part of KIP-1071.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20116: Forwad task-(end)-offset to task assignor (4/N) [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #22644:
URL: https://github.com/apache/kafka/pull/22644#discussion_r3465358925


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -2118,6 +2121,14 @@ private CoordinatorResult<StreamsGroupHeartbeatResult, CoordinatorRecord> stream
             );
         }
 
+        // Store the latest task changelog offsets/end-offsets reported by the member. These are transient telemetry
+        // (used by the assignor to estimate task lag) and are not persisted. Task offsets and end-offsets are reported
+        // independently: a null list means "unchanged since the last heartbeat", so we retain the previously reported
+        // value for whichever of the two is null and only update when at least one is reported.
+        if (taskOffsets != null || taskEndOffsets != null) {
+            group.updateTaskOffsets(memberId, group.taskOffsets(memberId).update(taskOffsets, taskEndOffsets));

Review Comment:
   If they are transient, we may want to discuss whether using timeline data structures is the correct approach here. Those are usually backed by persisted records.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20116: Forwad task-(end)-offset to task assignor (4/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #22644:
URL: https://github.com/apache/kafka/pull/22644#discussion_r3468642999


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -2118,6 +2121,14 @@ private CoordinatorResult<StreamsGroupHeartbeatResult, CoordinatorRecord> stream
             );
         }
 
+        // Store the latest task changelog offsets/end-offsets reported by the member. These are transient telemetry
+        // (used by the assignor to estimate task lag) and are not persisted. Task offsets and end-offsets are reported
+        // independently: a null list means "unchanged since the last heartbeat", so we retain the previously reported
+        // value for whichever of the two is null and only update when at least one is reported.
+        if (taskOffsets != null || taskEndOffsets != null) {
+            group.updateTaskOffsets(memberId, group.taskOffsets(memberId).update(taskOffsets, taskEndOffsets));

Review Comment:
   Ah. Good catch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20116: Forwad task-(end)-offset to task assignor (5/5) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #22644:
URL: https://github.com/apache/kafka/pull/22644


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org