You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lucasbru (via GitHub)" <gi...@apache.org> on 2025/09/09 12:23:13 UTC

[PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

lucasbru opened a new pull request, #20511:
URL: https://github.com/apache/kafka/pull/20511

   In the consumer, we invoke the consumer rebalance onPartitionRevoked or onPartitionLost callbacks, when the consumer closes. The point is that the application may want to commit, or wipe the state if we are closing unsuccessfully.
   
   In the StreamsRebalanceListener, we did not implement this behavior, which means when closing the consumer we may lose some progress, and in the worst case also miss that we have to wipe our local state state since we got fenced.
   
   In this PR we implement StreamsRebalanceListenerInvoker, very similarly to ConsumerRebalanceListenerInvoker and invoke it in Consumer.close.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru merged PR #20511:
URL: https://github.com/apache/kafka/pull/20511


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2337642213


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        this.listener = Optional.ofNullable(streamsRebalanceListener);
+    }
+
+    public Exception invokeAllTasksRevoked() {
+        if (listener.isPresent()) {
+            return invokeTasksRevoked(streamsRebalanceData.reconciledAssignment().activeTasks());
+        }
+
+        return null;
+    }
+
+    public Exception invokeTasksAssigned(final StreamsRebalanceData.Assignment assignment) {
+        if (listener.isPresent()) {
+            log.info("Adding newly assigned tasks: {}", assignment);

Review Comment:
   This log seems a bit confusing here on the callback invocations. Shouldn't it be more along the lines of "trigger task assigned callbacks" (maybe is me misreading, missing KS details. Ex. on the consumer side the assigned callback runs when the partitions were already assigned/added). If this applies, let's align also the logs for revoked/lost  



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -278,44 +269,28 @@ private void processStreamsOnAllTasksLostCallbackNeededEvent(final StreamsOnAllT
 
         private StreamsOnTasksRevokedCallbackCompletedEvent invokeOnTasksRevokedCallback(final Set<StreamsRebalanceData.TaskId> activeTasksToRevoke,
                                                                                          final CompletableFuture<Void> future) {
-            final Optional<Exception> exceptionFromCallback = streamsRebalanceListener().onTasksRevoked(activeTasksToRevoke);
+            final Optional<Exception> exceptionFromCallback = Optional.ofNullable(streamsRebalanceListenerInvoker().invokeTasksRevoked(activeTasksToRevoke));
             final Optional<KafkaException> error = exceptionFromCallback.map(e -> ConsumerUtils.maybeWrapAsKafkaException(e, "Task revocation callback throws an error"));

Review Comment:
   could we maybe simplify by pushing the optionals and maybeWrap logic to the completedEvent constructors?
   Keeping here simply
   
   ```
   Exception  error = invokeTasksRevoked(...)
   return new StreamsOnTasksRevokedCallbackCompletedEvent(future, error) 
   // and then in the constructor is where magic happens to shape that error, have a custom msg, etc. 
   // Applied consistently for the 3 StreamsOnTasksXXXCallbackCompletedEvent
   ```
   
   Just trying to detangle all this optionals a bit



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1964,7 +1957,11 @@ public void subscribe(Collection<String> topics, ConsumerRebalanceListener liste
 
     public void subscribe(Collection<String> topics, StreamsRebalanceListener streamsRebalanceListener) {
         subscribeInternal(topics, Optional.empty());
-        backgroundEventProcessor.setStreamsRebalanceListener(streamsRebalanceListener);
+        if (streamsRebalanceListenerInvoker.isPresent()) {
+            streamsRebalanceListenerInvoker.get().setRebalanceListener(streamsRebalanceListener);
+        } else {
+            throw new IllegalStateException("Consumer was not created to be used with Streams rebalance protocol events");
+        }

Review Comment:
   I would expect is probably best/clearer to do this setup/validation before any actual subscription action (`subscribeInternal`)?



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumerTest.java:
##########
@@ -2210,6 +2210,75 @@ private void markOffsetsReadyForCommitEvent() {
         }).when(applicationEventHandler).add(ArgumentMatchers.isA(CommitEvent.class));
     }
 
+    @Test
+    public void testCloseInvokesStreamsRebalanceListenerOnTasksRevokedWhenMemberEpochPositive() {
+        final String groupId = "streamsGroup";
+        final StreamsRebalanceData streamsRebalanceData = new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of());
+        
+        try (final MockedStatic<RequestManagers> requestManagers = mockStatic(RequestManagers.class)) {
+            consumer = newConsumerWithStreamRebalanceData(requiredConsumerConfigAndGroupId(groupId), streamsRebalanceData);
+            StreamsRebalanceListener mockStreamsListener = mock(StreamsRebalanceListener.class);
+            when(mockStreamsListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+            consumer.subscribe(singletonList("topic"), mockStreamsListener);
+            final MemberStateListener groupMetadataUpdateListener = captureGroupMetadataUpdateListener(requestManagers);
+            final int memberEpoch = 42;
+            final String memberId = "memberId";
+            groupMetadataUpdateListener.onMemberEpochUpdated(Optional.of(memberEpoch), memberId);
+            
+            consumer.close(CloseOptions.timeout(Duration.ZERO));
+            
+            verify(mockStreamsListener).onTasksRevoked(any());
+        }
+    }
+    
+    @Test
+    public void testCloseInvokesStreamsRebalanceListenerOnAllTasksLostWhenMemberEpochZeroOrNegative() {
+        // Test that close() calls streamsRebalanceListener.invokeAllTasksLost() when memberEpoch <= 0

Review Comment:
   this comment seems to state the same as the func name really, needed?



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null
+        invoker.setRebalanceListener(null);
+        
+        // Should behave as if no listener is set
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListenerOverwritesExisting() {
+        StreamsRebalanceListener firstListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        StreamsRebalanceListener secondListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(secondListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Set first listener
+        invoker.setRebalanceListener(firstListener);
+        
+        // Overwrite with second listener
+        invoker.setRebalanceListener(secondListener);
+        
+        // Should use second listener
+        invoker.invokeAllTasksRevoked();
+        verify(firstListener, never()).onTasksRevoked(any());
+        verify(secondListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithNoListener() {
+        // When no listener is set, should return null
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result);
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithListener() {
+        invoker.setRebalanceListener(mockListener);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        Exception result = invoker.invokeAllTasksRevoked();
+        
+        assertNull(result);
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }

Review Comment:
   This seems to be the same as `testSetRebalanceListener`? (I do like this name better though)



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumerTest.java:
##########
@@ -2210,6 +2210,75 @@ private void markOffsetsReadyForCommitEvent() {
         }).when(applicationEventHandler).add(ArgumentMatchers.isA(CommitEvent.class));
     }
 
+    @Test
+    public void testCloseInvokesStreamsRebalanceListenerOnTasksRevokedWhenMemberEpochPositive() {
+        final String groupId = "streamsGroup";
+        final StreamsRebalanceData streamsRebalanceData = new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of());
+        
+        try (final MockedStatic<RequestManagers> requestManagers = mockStatic(RequestManagers.class)) {
+            consumer = newConsumerWithStreamRebalanceData(requiredConsumerConfigAndGroupId(groupId), streamsRebalanceData);
+            StreamsRebalanceListener mockStreamsListener = mock(StreamsRebalanceListener.class);
+            when(mockStreamsListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+            consumer.subscribe(singletonList("topic"), mockStreamsListener);
+            final MemberStateListener groupMetadataUpdateListener = captureGroupMetadataUpdateListener(requestManagers);
+            final int memberEpoch = 42;
+            final String memberId = "memberId";
+            groupMetadataUpdateListener.onMemberEpochUpdated(Optional.of(memberEpoch), memberId);
+            
+            consumer.close(CloseOptions.timeout(Duration.ZERO));
+            
+            verify(mockStreamsListener).onTasksRevoked(any());
+        }
+    }
+    
+    @Test
+    public void testCloseInvokesStreamsRebalanceListenerOnAllTasksLostWhenMemberEpochZeroOrNegative() {
+        // Test that close() calls streamsRebalanceListener.invokeAllTasksLost() when memberEpoch <= 0
+        final String groupId = "streamsGroup";
+        final StreamsRebalanceData streamsRebalanceData = new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of());
+        
+        try (final MockedStatic<RequestManagers> requestManagers = mockStatic(RequestManagers.class)) {
+            consumer = newConsumerWithStreamRebalanceData(requiredConsumerConfigAndGroupId(groupId), streamsRebalanceData);
+            StreamsRebalanceListener mockStreamsListener = mock(StreamsRebalanceListener.class);
+            when(mockStreamsListener.onAllTasksLost()).thenReturn(Optional.empty());
+            consumer.subscribe(singletonList("topic"), mockStreamsListener);
+            final MemberStateListener groupMetadataUpdateListener = captureGroupMetadataUpdateListener(requestManagers);
+            final int memberEpoch = 0;
+            final String memberId = "memberId";
+            groupMetadataUpdateListener.onMemberEpochUpdated(Optional.of(memberEpoch), memberId);
+            
+            consumer.close(CloseOptions.timeout(Duration.ZERO));
+            
+            verify(mockStreamsListener).onAllTasksLost();
+        }
+    }
+    
+    @Test
+    public void testCloseWrapsStreamsRebalanceListenerException() {
+        final String groupId = "streamsGroup";
+        final StreamsRebalanceData streamsRebalanceData = new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of());
+        
+        try (final MockedStatic<RequestManagers> requestManagers = mockStatic(RequestManagers.class)) {
+            consumer = newConsumerWithStreamRebalanceData(requiredConsumerConfigAndGroupId(groupId), streamsRebalanceData);
+            StreamsRebalanceListener mockStreamsListener = mock(StreamsRebalanceListener.class);
+            RuntimeException testException = new RuntimeException("Test streams listener exception");
+            doThrow(testException).when(mockStreamsListener).onTasksRevoked(any());
+            consumer.subscribe(singletonList("topic"), mockStreamsListener);
+            final MemberStateListener groupMetadataUpdateListener = captureGroupMetadataUpdateListener(requestManagers);
+            final int memberEpoch = 1;
+            final String memberId = "memberId";
+            groupMetadataUpdateListener.onMemberEpochUpdated(Optional.of(memberEpoch), memberId);
+            
+            KafkaException thrownException = assertThrows(KafkaException.class, 
+                () -> consumer.close(CloseOptions.timeout(Duration.ZERO)));
+            
+            assertNotNull(thrownException.getCause());
+            assertTrue(thrownException.getCause() instanceof RuntimeException);

Review Comment:
   ```suggestion
               assertInstanceOf(RuntimeException.class, thrownException.getCause())
   ```



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null
+        invoker.setRebalanceListener(null);
+        
+        // Should behave as if no listener is set
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListenerOverwritesExisting() {
+        StreamsRebalanceListener firstListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        StreamsRebalanceListener secondListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(secondListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Set first listener
+        invoker.setRebalanceListener(firstListener);
+        
+        // Overwrite with second listener
+        invoker.setRebalanceListener(secondListener);
+        
+        // Should use second listener
+        invoker.invokeAllTasksRevoked();
+        verify(firstListener, never()).onTasksRevoked(any());
+        verify(secondListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithNoListener() {
+        // When no listener is set, should return null
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result);
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithListener() {
+        invoker.setRebalanceListener(mockListener);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        Exception result = invoker.invokeAllTasksRevoked();
+        
+        assertNull(result);
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testInvokeTasksAssignedWithNoListener() {
+        StreamsRebalanceData.Assignment assignment = createMockAssignment();
+        
+        Exception result = invoker.invokeTasksAssigned(assignment);
+        
+        assertNull(result);
+        verify(mockListener, never()).onTasksAssigned(any());
+    }
+
+    @Test
+    public void testInvokeTasksAssignedWithListener() {
+        invoker.setRebalanceListener(mockListener);
+        StreamsRebalanceData.Assignment assignment = createMockAssignment();
+        when(mockListener.onTasksAssigned(assignment)).thenReturn(Optional.empty());
+        
+        Exception result = invoker.invokeTasksAssigned(assignment);
+        
+        assertNull(result);
+        verify(mockListener).onTasksAssigned(eq(assignment));
+    }
+
+    @Test
+    public void testInvokeTasksAssignedWithWakeupException() {
+        invoker.setRebalanceListener(mockListener);
+        StreamsRebalanceData.Assignment assignment = createMockAssignment();
+        WakeupException wakeupException = new WakeupException();
+        doThrow(wakeupException).when(mockListener).onTasksAssigned(assignment);
+        
+        WakeupException thrownException = assertThrows(WakeupException.class, 
+            () -> invoker.invokeTasksAssigned(assignment));
+        
+        assertEquals(wakeupException, thrownException);
+        verify(mockListener).onTasksAssigned(eq(assignment));
+    }
+
+    @Test
+    public void testInvokeTasksAssignedWithInterruptException() {
+        invoker.setRebalanceListener(mockListener);
+        StreamsRebalanceData.Assignment assignment = createMockAssignment();
+        InterruptException interruptException = new InterruptException("Test interrupt");
+        doThrow(interruptException).when(mockListener).onTasksAssigned(assignment);
+        
+        InterruptException thrownException = assertThrows(InterruptException.class, 
+            () -> invoker.invokeTasksAssigned(assignment));
+        
+        assertEquals(interruptException, thrownException);
+        verify(mockListener).onTasksAssigned(eq(assignment));
+    }
+
+    @Test
+    public void testInvokeTasksAssignedWithOtherException() {
+        invoker.setRebalanceListener(mockListener);
+        StreamsRebalanceData.Assignment assignment = createMockAssignment();
+        RuntimeException runtimeException = new RuntimeException("Test exception");
+        doThrow(runtimeException).when(mockListener).onTasksAssigned(assignment);
+        
+        Exception result = invoker.invokeTasksAssigned(assignment);
+        
+        assertEquals(runtimeException, result);
+        verify(mockListener).onTasksAssigned(eq(assignment));
+    }
+
+    @Test
+    public void testInvokeTasksRevokedWithNoListener() {
+        Set<StreamsRebalanceData.TaskId> tasks = createMockTasks();
+        
+        Exception result = invoker.invokeTasksRevoked(tasks);
+        
+        assertNull(result);
+        verify(mockListener, never()).onTasksRevoked(any());
+    }
+
+    @Test
+    public void testInvokeTasksRevokedWithListener() {
+        invoker.setRebalanceListener(mockListener);
+        Set<StreamsRebalanceData.TaskId> tasks = createMockTasks();
+        when(mockListener.onTasksRevoked(tasks)).thenReturn(Optional.empty());
+        
+        Exception result = invoker.invokeTasksRevoked(tasks);
+        
+        assertNull(result);
+        verify(mockListener).onTasksRevoked(eq(tasks));
+    }
+
+    @Test
+    public void testInvokeTasksRevokedWithWakeupException() {
+        invoker.setRebalanceListener(mockListener);
+        Set<StreamsRebalanceData.TaskId> tasks = createMockTasks();
+        WakeupException wakeupException = new WakeupException();
+        doThrow(wakeupException).when(mockListener).onTasksRevoked(tasks);
+        
+        WakeupException thrownException = assertThrows(WakeupException.class, 
+            () -> invoker.invokeTasksRevoked(tasks));
+        
+        assertEquals(wakeupException, thrownException);
+        verify(mockListener).onTasksRevoked(eq(tasks));
+    }
+
+    @Test
+    public void testInvokeTasksRevokedWithInterruptException() {
+        invoker.setRebalanceListener(mockListener);
+        Set<StreamsRebalanceData.TaskId> tasks = createMockTasks();
+        InterruptException interruptException = new InterruptException("Test interrupt");
+        doThrow(interruptException).when(mockListener).onTasksRevoked(tasks);
+        
+        InterruptException thrownException = assertThrows(InterruptException.class, 
+            () -> invoker.invokeTasksRevoked(tasks));
+        
+        assertEquals(interruptException, thrownException);
+        verify(mockListener).onTasksRevoked(eq(tasks));
+    }
+
+    @Test
+    public void testInvokeTasksRevokedWithOtherException() {
+        invoker.setRebalanceListener(mockListener);
+        Set<StreamsRebalanceData.TaskId> tasks = createMockTasks();
+        RuntimeException runtimeException = new RuntimeException("Test exception");
+        doThrow(runtimeException).when(mockListener).onTasksRevoked(tasks);
+        
+        Exception result = invoker.invokeTasksRevoked(tasks);
+        
+        assertEquals(runtimeException, result);
+        verify(mockListener).onTasksRevoked(eq(tasks));
+    }
+
+    @Test
+    public void testInvokeAllTasksLostWithNoListener() {

Review Comment:
   isn't this testing the same as `testConstructorInitializesWithEmptyListener`?



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null
+        invoker.setRebalanceListener(null);
+        
+        // Should behave as if no listener is set
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListenerOverwritesExisting() {
+        StreamsRebalanceListener firstListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        StreamsRebalanceListener secondListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(secondListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Set first listener
+        invoker.setRebalanceListener(firstListener);
+        
+        // Overwrite with second listener
+        invoker.setRebalanceListener(secondListener);
+        
+        // Should use second listener
+        invoker.invokeAllTasksRevoked();
+        verify(firstListener, never()).onTasksRevoked(any());
+        verify(secondListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithNoListener() {
+        // When no listener is set, should return null
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result);
+    }

Review Comment:
   isn't this testing the same as `testConstructorInitializesWithEmptyListener`?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -187,23 +187,14 @@ public class AsyncKafkaConsumer<K, V> implements ConsumerDelegate<K, V> {
      */
     private class BackgroundEventProcessor implements EventProcessor<BackgroundEvent> {
 
-        private Optional<StreamsRebalanceListener> streamsRebalanceListener = Optional.empty();
-        private final Optional<StreamsRebalanceData> streamsRebalanceData;
+        private final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker;

Review Comment:
   I would expect it's optional because it will only exist if the AsycKafkaConsumer is being used with the streams rebalance protocol.
   
   But why do we need to declare it here in the internal class if it already exist in the parent `AsyncKafkaConsumer`? https://github.com/apache/kafka/blob/45a410b8a8fc3cc95c92ae8da9748990e9ddf1b8/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L345 
   (can we just use that one? as we do with the `rebalanceListenerInvoker`)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2339132075


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -187,23 +187,14 @@ public class AsyncKafkaConsumer<K, V> implements ConsumerDelegate<K, V> {
      */
     private class BackgroundEventProcessor implements EventProcessor<BackgroundEvent> {
 
-        private Optional<StreamsRebalanceListener> streamsRebalanceListener = Optional.empty();
-        private final Optional<StreamsRebalanceData> streamsRebalanceData;
+        private final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker;
 
         public BackgroundEventProcessor() {
-            this.streamsRebalanceData = Optional.empty();
+            this.streamsRebalanceListenerInvoker = Optional.empty();
         }
 
-        public BackgroundEventProcessor(final Optional<StreamsRebalanceData> streamsRebalanceData) {
-            this.streamsRebalanceData = streamsRebalanceData;
-        }
-
-        private void setStreamsRebalanceListener(final StreamsRebalanceListener streamsRebalanceListener) {
-            if (streamsRebalanceData.isEmpty()) {
-                throw new IllegalStateException("Background event processor was not created to be used with Streams " +
-                    "rebalance protocol events");
-            }
-            this.streamsRebalanceListener = Optional.of(streamsRebalanceListener);
+        public BackgroundEventProcessor(final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker) {

Review Comment:
   Removed the constructor.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -187,23 +187,14 @@ public class AsyncKafkaConsumer<K, V> implements ConsumerDelegate<K, V> {
      */
     private class BackgroundEventProcessor implements EventProcessor<BackgroundEvent> {
 
-        private Optional<StreamsRebalanceListener> streamsRebalanceListener = Optional.empty();
-        private final Optional<StreamsRebalanceData> streamsRebalanceData;
+        private final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker;

Review Comment:
   True. Removed the second field and used the reference in the outer class instead.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -278,44 +269,28 @@ private void processStreamsOnAllTasksLostCallbackNeededEvent(final StreamsOnAllT
 
         private StreamsOnTasksRevokedCallbackCompletedEvent invokeOnTasksRevokedCallback(final Set<StreamsRebalanceData.TaskId> activeTasksToRevoke,
                                                                                          final CompletableFuture<Void> future) {
-            final Optional<Exception> exceptionFromCallback = streamsRebalanceListener().onTasksRevoked(activeTasksToRevoke);
+            final Optional<Exception> exceptionFromCallback = Optional.ofNullable(streamsRebalanceListenerInvoker().invokeTasksRevoked(activeTasksToRevoke));
             final Optional<KafkaException> error = exceptionFromCallback.map(e -> ConsumerUtils.maybeWrapAsKafkaException(e, "Task revocation callback throws an error"));

Review Comment:
   This should work as expected. Indeed Optional.ofNullable will give Optional.empty for null, and the Optional.map will not be executed for Optional.empty.
   
   This is actually a suggested simplification by IntelliJ



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1964,7 +1957,11 @@ public void subscribe(Collection<String> topics, ConsumerRebalanceListener liste
 
     public void subscribe(Collection<String> topics, StreamsRebalanceListener streamsRebalanceListener) {
         subscribeInternal(topics, Optional.empty());
-        backgroundEventProcessor.setStreamsRebalanceListener(streamsRebalanceListener);
+        if (streamsRebalanceListenerInvoker.isPresent()) {
+            streamsRebalanceListenerInvoker.get().setRebalanceListener(streamsRebalanceListener);
+        } else {
+            throw new IllegalStateException("Consumer was not created to be used with Streams rebalance protocol events");
+        }

Review Comment:
   Done



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener

Review Comment:
   Done



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null

Review Comment:
   Done



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -278,44 +269,28 @@ private void processStreamsOnAllTasksLostCallbackNeededEvent(final StreamsOnAllT
 
         private StreamsOnTasksRevokedCallbackCompletedEvent invokeOnTasksRevokedCallback(final Set<StreamsRebalanceData.TaskId> activeTasksToRevoke,
                                                                                          final CompletableFuture<Void> future) {
-            final Optional<Exception> exceptionFromCallback = streamsRebalanceListener().onTasksRevoked(activeTasksToRevoke);
+            final Optional<Exception> exceptionFromCallback = Optional.ofNullable(streamsRebalanceListenerInvoker().invokeTasksRevoked(activeTasksToRevoke));
             final Optional<KafkaException> error = exceptionFromCallback.map(e -> ConsumerUtils.maybeWrapAsKafkaException(e, "Task revocation callback throws an error"));

Review Comment:
   I think I disagree here, since when you look at `invokeRabalanceCallbacks`, we are doing it outside the constructors in the consumer case as well. I would prefer to keep this consistent. So doing it inside the event constructors in one case but not the other would seem inconsistent to me. If you insist, I would prefer changing it on both codepaths.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1964,7 +1957,11 @@ public void subscribe(Collection<String> topics, ConsumerRebalanceListener liste
 
     public void subscribe(Collection<String> topics, StreamsRebalanceListener streamsRebalanceListener) {
         subscribeInternal(topics, Optional.empty());
-        backgroundEventProcessor.setStreamsRebalanceListener(streamsRebalanceListener);
+        if (streamsRebalanceListenerInvoker.isPresent()) {

Review Comment:
   Done



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;

Review Comment:
   The thing is that the invoker is constructed with the consumer, but the listener is only registered with the `subscribe` call. So I'd rather handle the intermediate state - where we have a streams consumer but no listener yet - explicitly. But I get your point that we do not really have to allow this case. So I'm throwing illegal state exceptions now when the listener is invoked but not yet registered.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        this.listener = Optional.ofNullable(streamsRebalanceListener);
+    }
+
+    public Exception invokeAllTasksRevoked() {
+        if (listener.isPresent()) {
+            return invokeTasksRevoked(streamsRebalanceData.reconciledAssignment().activeTasks());
+        }
+
+        return null;
+    }
+
+    public Exception invokeTasksAssigned(final StreamsRebalanceData.Assignment assignment) {
+        if (listener.isPresent()) {
+            log.info("Adding newly assigned tasks: {}", assignment);

Review Comment:
   The consumer rebalance listener logs `Adding newly assigned partitions:` in this case, I just copied its behavior for consistency. But generally, I agree. 
   
   I updated the log line indicate that the rebalance listener is invoked.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -278,44 +269,28 @@ private void processStreamsOnAllTasksLostCallbackNeededEvent(final StreamsOnAllT
 
         private StreamsOnTasksRevokedCallbackCompletedEvent invokeOnTasksRevokedCallback(final Set<StreamsRebalanceData.TaskId> activeTasksToRevoke,
                                                                                          final CompletableFuture<Void> future) {
-            final Optional<Exception> exceptionFromCallback = streamsRebalanceListener().onTasksRevoked(activeTasksToRevoke);
+            final Optional<Exception> exceptionFromCallback = Optional.ofNullable(streamsRebalanceListenerInvoker().invokeTasksRevoked(activeTasksToRevoke));
             final Optional<KafkaException> error = exceptionFromCallback.map(e -> ConsumerUtils.maybeWrapAsKafkaException(e, "Task revocation callback throws an error"));
             return new StreamsOnTasksRevokedCallbackCompletedEvent(future, error);
         }
 
         private StreamsOnTasksAssignedCallbackCompletedEvent invokeOnTasksAssignedCallback(final StreamsRebalanceData.Assignment assignment,
                                                                                            final CompletableFuture<Void> future) {
             final Optional<KafkaException> error;
-            final Optional<Exception> exceptionFromCallback = streamsRebalanceListener().onTasksAssigned(assignment);
-            if (exceptionFromCallback.isPresent()) {
-                error = Optional.of(ConsumerUtils.maybeWrapAsKafkaException(exceptionFromCallback.get(), "Task assignment callback throws an error"));
-            } else {
-                error = Optional.empty();
-                streamsRebalanceData().setReconciledAssignment(assignment);
-            }
+            final Optional<Exception> exceptionFromCallback = Optional.ofNullable(streamsRebalanceListenerInvoker().invokeTasksAssigned(assignment));
+            error = exceptionFromCallback.map(e -> ConsumerUtils.maybeWrapAsKafkaException(e, "Task assignment callback throws an error"));

Review Comment:
   See above.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1532,21 +1512,34 @@ private void runRebalanceCallbacksOnClose() {
 

Review Comment:
   Urgs. I think `rebalanceListenerInvoker` is never null. It's created even when no rebalanceListener is defined. I removed that part of the if condition, since it's confusing. 



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        this.listener = Optional.ofNullable(streamsRebalanceListener);

Review Comment:
   Throwing nullpointerException instead now.



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumerTest.java:
##########
@@ -2210,6 +2210,75 @@ private void markOffsetsReadyForCommitEvent() {
         }).when(applicationEventHandler).add(ArgumentMatchers.isA(CommitEvent.class));
     }
 
+    @Test
+    public void testCloseInvokesStreamsRebalanceListenerOnTasksRevokedWhenMemberEpochPositive() {
+        final String groupId = "streamsGroup";
+        final StreamsRebalanceData streamsRebalanceData = new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of());
+        
+        try (final MockedStatic<RequestManagers> requestManagers = mockStatic(RequestManagers.class)) {
+            consumer = newConsumerWithStreamRebalanceData(requiredConsumerConfigAndGroupId(groupId), streamsRebalanceData);
+            StreamsRebalanceListener mockStreamsListener = mock(StreamsRebalanceListener.class);
+            when(mockStreamsListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+            consumer.subscribe(singletonList("topic"), mockStreamsListener);
+            final MemberStateListener groupMetadataUpdateListener = captureGroupMetadataUpdateListener(requestManagers);
+            final int memberEpoch = 42;
+            final String memberId = "memberId";
+            groupMetadataUpdateListener.onMemberEpochUpdated(Optional.of(memberEpoch), memberId);
+            
+            consumer.close(CloseOptions.timeout(Duration.ZERO));
+            
+            verify(mockStreamsListener).onTasksRevoked(any());
+        }
+    }
+    
+    @Test
+    public void testCloseInvokesStreamsRebalanceListenerOnAllTasksLostWhenMemberEpochZeroOrNegative() {
+        // Test that close() calls streamsRebalanceListener.invokeAllTasksLost() when memberEpoch <= 0

Review Comment:
   Done



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null
+        invoker.setRebalanceListener(null);
+        
+        // Should behave as if no listener is set
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListenerOverwritesExisting() {
+        StreamsRebalanceListener firstListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        StreamsRebalanceListener secondListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(secondListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Set first listener
+        invoker.setRebalanceListener(firstListener);
+        
+        // Overwrite with second listener
+        invoker.setRebalanceListener(secondListener);
+        
+        // Should use second listener
+        invoker.invokeAllTasksRevoked();
+        verify(firstListener, never()).onTasksRevoked(any());
+        verify(secondListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithNoListener() {
+        // When no listener is set, should return null
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result);
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithListener() {
+        invoker.setRebalanceListener(mockListener);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        Exception result = invoker.invokeAllTasksRevoked();
+        
+        assertNull(result);
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }

Review Comment:
   Done



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null
+        invoker.setRebalanceListener(null);
+        
+        // Should behave as if no listener is set
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListenerOverwritesExisting() {
+        StreamsRebalanceListener firstListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        StreamsRebalanceListener secondListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);

Review Comment:
   Done



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumerTest.java:
##########
@@ -2210,6 +2210,75 @@ private void markOffsetsReadyForCommitEvent() {
         }).when(applicationEventHandler).add(ArgumentMatchers.isA(CommitEvent.class));
     }
 
+    @Test
+    public void testCloseInvokesStreamsRebalanceListenerOnTasksRevokedWhenMemberEpochPositive() {
+        final String groupId = "streamsGroup";
+        final StreamsRebalanceData streamsRebalanceData = new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of());
+        
+        try (final MockedStatic<RequestManagers> requestManagers = mockStatic(RequestManagers.class)) {
+            consumer = newConsumerWithStreamRebalanceData(requiredConsumerConfigAndGroupId(groupId), streamsRebalanceData);
+            StreamsRebalanceListener mockStreamsListener = mock(StreamsRebalanceListener.class);
+            when(mockStreamsListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+            consumer.subscribe(singletonList("topic"), mockStreamsListener);
+            final MemberStateListener groupMetadataUpdateListener = captureGroupMetadataUpdateListener(requestManagers);
+            final int memberEpoch = 42;
+            final String memberId = "memberId";
+            groupMetadataUpdateListener.onMemberEpochUpdated(Optional.of(memberEpoch), memberId);
+            
+            consumer.close(CloseOptions.timeout(Duration.ZERO));
+            
+            verify(mockStreamsListener).onTasksRevoked(any());
+        }
+    }
+    
+    @Test
+    public void testCloseInvokesStreamsRebalanceListenerOnAllTasksLostWhenMemberEpochZeroOrNegative() {
+        // Test that close() calls streamsRebalanceListener.invokeAllTasksLost() when memberEpoch <= 0
+        final String groupId = "streamsGroup";
+        final StreamsRebalanceData streamsRebalanceData = new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of());
+        
+        try (final MockedStatic<RequestManagers> requestManagers = mockStatic(RequestManagers.class)) {
+            consumer = newConsumerWithStreamRebalanceData(requiredConsumerConfigAndGroupId(groupId), streamsRebalanceData);
+            StreamsRebalanceListener mockStreamsListener = mock(StreamsRebalanceListener.class);
+            when(mockStreamsListener.onAllTasksLost()).thenReturn(Optional.empty());
+            consumer.subscribe(singletonList("topic"), mockStreamsListener);
+            final MemberStateListener groupMetadataUpdateListener = captureGroupMetadataUpdateListener(requestManagers);
+            final int memberEpoch = 0;
+            final String memberId = "memberId";
+            groupMetadataUpdateListener.onMemberEpochUpdated(Optional.of(memberEpoch), memberId);
+            
+            consumer.close(CloseOptions.timeout(Duration.ZERO));
+            
+            verify(mockStreamsListener).onAllTasksLost();
+        }
+    }
+    
+    @Test
+    public void testCloseWrapsStreamsRebalanceListenerException() {
+        final String groupId = "streamsGroup";
+        final StreamsRebalanceData streamsRebalanceData = new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of());
+        
+        try (final MockedStatic<RequestManagers> requestManagers = mockStatic(RequestManagers.class)) {
+            consumer = newConsumerWithStreamRebalanceData(requiredConsumerConfigAndGroupId(groupId), streamsRebalanceData);
+            StreamsRebalanceListener mockStreamsListener = mock(StreamsRebalanceListener.class);
+            RuntimeException testException = new RuntimeException("Test streams listener exception");
+            doThrow(testException).when(mockStreamsListener).onTasksRevoked(any());
+            consumer.subscribe(singletonList("topic"), mockStreamsListener);
+            final MemberStateListener groupMetadataUpdateListener = captureGroupMetadataUpdateListener(requestManagers);
+            final int memberEpoch = 1;
+            final String memberId = "memberId";
+            groupMetadataUpdateListener.onMemberEpochUpdated(Optional.of(memberEpoch), memberId);
+            
+            KafkaException thrownException = assertThrows(KafkaException.class, 
+                () -> consumer.close(CloseOptions.timeout(Duration.ZERO)));
+            
+            assertNotNull(thrownException.getCause());
+            assertTrue(thrownException.getCause() instanceof RuntimeException);

Review Comment:
   Done



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null
+        invoker.setRebalanceListener(null);
+        
+        // Should behave as if no listener is set
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListenerOverwritesExisting() {
+        StreamsRebalanceListener firstListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        StreamsRebalanceListener secondListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(secondListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Set first listener
+        invoker.setRebalanceListener(firstListener);
+        
+        // Overwrite with second listener
+        invoker.setRebalanceListener(secondListener);
+        
+        // Should use second listener
+        invoker.invokeAllTasksRevoked();
+        verify(firstListener, never()).onTasksRevoked(any());
+        verify(secondListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithNoListener() {
+        // When no listener is set, should return null
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result);
+    }

Review Comment:
   Done



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null
+        invoker.setRebalanceListener(null);
+        
+        // Should behave as if no listener is set
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListenerOverwritesExisting() {
+        StreamsRebalanceListener firstListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        StreamsRebalanceListener secondListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(secondListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Set first listener
+        invoker.setRebalanceListener(firstListener);
+        
+        // Overwrite with second listener
+        invoker.setRebalanceListener(secondListener);
+        
+        // Should use second listener
+        invoker.invokeAllTasksRevoked();
+        verify(firstListener, never()).onTasksRevoked(any());
+        verify(secondListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithNoListener() {
+        // When no listener is set, should return null
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result);
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithListener() {
+        invoker.setRebalanceListener(mockListener);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        Exception result = invoker.invokeAllTasksRevoked();
+        
+        assertNull(result);
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testInvokeTasksAssignedWithNoListener() {
+        StreamsRebalanceData.Assignment assignment = createMockAssignment();
+        
+        Exception result = invoker.invokeTasksAssigned(assignment);
+        
+        assertNull(result);
+        verify(mockListener, never()).onTasksAssigned(any());
+    }
+
+    @Test
+    public void testInvokeTasksAssignedWithListener() {
+        invoker.setRebalanceListener(mockListener);
+        StreamsRebalanceData.Assignment assignment = createMockAssignment();
+        when(mockListener.onTasksAssigned(assignment)).thenReturn(Optional.empty());
+        
+        Exception result = invoker.invokeTasksAssigned(assignment);
+        
+        assertNull(result);
+        verify(mockListener).onTasksAssigned(eq(assignment));
+    }
+
+    @Test
+    public void testInvokeTasksAssignedWithWakeupException() {
+        invoker.setRebalanceListener(mockListener);
+        StreamsRebalanceData.Assignment assignment = createMockAssignment();
+        WakeupException wakeupException = new WakeupException();
+        doThrow(wakeupException).when(mockListener).onTasksAssigned(assignment);
+        
+        WakeupException thrownException = assertThrows(WakeupException.class, 
+            () -> invoker.invokeTasksAssigned(assignment));
+        
+        assertEquals(wakeupException, thrownException);
+        verify(mockListener).onTasksAssigned(eq(assignment));
+    }
+
+    @Test
+    public void testInvokeTasksAssignedWithInterruptException() {
+        invoker.setRebalanceListener(mockListener);
+        StreamsRebalanceData.Assignment assignment = createMockAssignment();
+        InterruptException interruptException = new InterruptException("Test interrupt");
+        doThrow(interruptException).when(mockListener).onTasksAssigned(assignment);
+        
+        InterruptException thrownException = assertThrows(InterruptException.class, 
+            () -> invoker.invokeTasksAssigned(assignment));
+        
+        assertEquals(interruptException, thrownException);
+        verify(mockListener).onTasksAssigned(eq(assignment));
+    }
+
+    @Test
+    public void testInvokeTasksAssignedWithOtherException() {
+        invoker.setRebalanceListener(mockListener);
+        StreamsRebalanceData.Assignment assignment = createMockAssignment();
+        RuntimeException runtimeException = new RuntimeException("Test exception");
+        doThrow(runtimeException).when(mockListener).onTasksAssigned(assignment);
+        
+        Exception result = invoker.invokeTasksAssigned(assignment);
+        
+        assertEquals(runtimeException, result);
+        verify(mockListener).onTasksAssigned(eq(assignment));
+    }
+
+    @Test
+    public void testInvokeTasksRevokedWithNoListener() {
+        Set<StreamsRebalanceData.TaskId> tasks = createMockTasks();
+        
+        Exception result = invoker.invokeTasksRevoked(tasks);
+        
+        assertNull(result);
+        verify(mockListener, never()).onTasksRevoked(any());
+    }
+
+    @Test
+    public void testInvokeTasksRevokedWithListener() {
+        invoker.setRebalanceListener(mockListener);
+        Set<StreamsRebalanceData.TaskId> tasks = createMockTasks();
+        when(mockListener.onTasksRevoked(tasks)).thenReturn(Optional.empty());
+        
+        Exception result = invoker.invokeTasksRevoked(tasks);
+        
+        assertNull(result);
+        verify(mockListener).onTasksRevoked(eq(tasks));
+    }
+
+    @Test
+    public void testInvokeTasksRevokedWithWakeupException() {
+        invoker.setRebalanceListener(mockListener);
+        Set<StreamsRebalanceData.TaskId> tasks = createMockTasks();
+        WakeupException wakeupException = new WakeupException();
+        doThrow(wakeupException).when(mockListener).onTasksRevoked(tasks);
+        
+        WakeupException thrownException = assertThrows(WakeupException.class, 
+            () -> invoker.invokeTasksRevoked(tasks));
+        
+        assertEquals(wakeupException, thrownException);
+        verify(mockListener).onTasksRevoked(eq(tasks));
+    }
+
+    @Test
+    public void testInvokeTasksRevokedWithInterruptException() {
+        invoker.setRebalanceListener(mockListener);
+        Set<StreamsRebalanceData.TaskId> tasks = createMockTasks();
+        InterruptException interruptException = new InterruptException("Test interrupt");
+        doThrow(interruptException).when(mockListener).onTasksRevoked(tasks);
+        
+        InterruptException thrownException = assertThrows(InterruptException.class, 
+            () -> invoker.invokeTasksRevoked(tasks));
+        
+        assertEquals(interruptException, thrownException);
+        verify(mockListener).onTasksRevoked(eq(tasks));
+    }
+
+    @Test
+    public void testInvokeTasksRevokedWithOtherException() {
+        invoker.setRebalanceListener(mockListener);
+        Set<StreamsRebalanceData.TaskId> tasks = createMockTasks();
+        RuntimeException runtimeException = new RuntimeException("Test exception");
+        doThrow(runtimeException).when(mockListener).onTasksRevoked(tasks);
+        
+        Exception result = invoker.invokeTasksRevoked(tasks);
+        
+        assertEquals(runtimeException, result);
+        verify(mockListener).onTasksRevoked(eq(tasks));
+    }
+
+    @Test
+    public void testInvokeAllTasksLostWithNoListener() {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "UladzislauBlok (via GitHub)" <gi...@apache.org>.
UladzislauBlok commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2337131031


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -187,23 +187,14 @@ public class AsyncKafkaConsumer<K, V> implements ConsumerDelegate<K, V> {
      */
     private class BackgroundEventProcessor implements EventProcessor<BackgroundEvent> {
 
-        private Optional<StreamsRebalanceListener> streamsRebalanceListener = Optional.empty();
-        private final Optional<StreamsRebalanceData> streamsRebalanceData;
+        private final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker;

Review Comment:
   out of curiosity: why this is optional, if we anyway will throw exception at line 292 in case of null?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2333356980


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/DefaultStreamsRebalanceListenerTest.java:
##########
@@ -170,39 +167,56 @@ void testOnTasksAssigned() {
         );
         inOrder.verify(streamThread).setState(StreamThread.State.PARTITIONS_ASSIGNED);
         inOrder.verify(taskManager).handleRebalanceComplete();
+        inOrder.verify(streamsRebalanceData).setReconciledAssignment(assignment);
     }
 
     @Test
     void testOnTasksAssignedWithException() {
         final Exception exception = new RuntimeException("sample exception");
         doThrow(exception).when(taskManager).handleAssignment(any(), any());
 
-        createRebalanceListenerWithRebalanceData(new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of()));
-        final Optional<Exception> result = defaultStreamsRebalanceListener.onTasksAssigned(new StreamsRebalanceData.Assignment(Set.of(), Set.of(), Set.of()));
-        assertTrue(defaultStreamsRebalanceListener.onAllTasksLost().isEmpty());
+        final StreamsRebalanceData streamsRebalanceData = mock(StreamsRebalanceData.class);
+        when(streamsRebalanceData.subtopologies()).thenReturn(Map.of());
+        createRebalanceListenerWithRebalanceData(streamsRebalanceData);
+        
+        final Optional<Exception> result = defaultStreamsRebalanceListener.onTasksAssigned(
+            new StreamsRebalanceData.Assignment(Set.of(), Set.of(), Set.of())
+        );
+        
         assertTrue(result.isPresent());
         assertEquals(exception, result.get());
-        verify(taskManager).handleLostAll();
+        verify(taskManager).handleAssignment(any(), any());
         verify(streamThread, never()).setState(StreamThread.State.PARTITIONS_ASSIGNED);
         verify(taskManager, never()).handleRebalanceComplete();
+        verify(streamsRebalanceData, never()).setReconciledAssignment(any());
     }
 
     @Test
     void testOnAllTasksLost() {
-        createRebalanceListenerWithRebalanceData(new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of()));
+        final StreamsRebalanceData streamsRebalanceData = mock(StreamsRebalanceData.class);
+        when(streamsRebalanceData.subtopologies()).thenReturn(Map.of());
+        createRebalanceListenerWithRebalanceData(streamsRebalanceData);
+        
         assertTrue(defaultStreamsRebalanceListener.onAllTasksLost().isEmpty());
-        verify(taskManager).handleLostAll();
+        

Review Comment:
   Lines 199 and 201 contain unnecessary trailing whitespace that should be removed.



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/DefaultStreamsRebalanceListenerTest.java:
##########
@@ -170,39 +167,56 @@ void testOnTasksAssigned() {
         );
         inOrder.verify(streamThread).setState(StreamThread.State.PARTITIONS_ASSIGNED);
         inOrder.verify(taskManager).handleRebalanceComplete();
+        inOrder.verify(streamsRebalanceData).setReconciledAssignment(assignment);
     }
 
     @Test
     void testOnTasksAssignedWithException() {
         final Exception exception = new RuntimeException("sample exception");
         doThrow(exception).when(taskManager).handleAssignment(any(), any());
 
-        createRebalanceListenerWithRebalanceData(new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of()));
-        final Optional<Exception> result = defaultStreamsRebalanceListener.onTasksAssigned(new StreamsRebalanceData.Assignment(Set.of(), Set.of(), Set.of()));
-        assertTrue(defaultStreamsRebalanceListener.onAllTasksLost().isEmpty());
+        final StreamsRebalanceData streamsRebalanceData = mock(StreamsRebalanceData.class);
+        when(streamsRebalanceData.subtopologies()).thenReturn(Map.of());
+        createRebalanceListenerWithRebalanceData(streamsRebalanceData);
+        
+        final Optional<Exception> result = defaultStreamsRebalanceListener.onTasksAssigned(
+            new StreamsRebalanceData.Assignment(Set.of(), Set.of(), Set.of())
+        );
+        

Review Comment:
   There are multiple trailing whitespace issues in this test file. Lines 181 and 185 have unnecessary trailing spaces that should be removed for consistency.
   ```suggestion
   
           final Optional<Exception> result = defaultStreamsRebalanceListener.onTasksAssigned(
               new StreamsRebalanceData.Assignment(Set.of(), Set.of(), Set.of())
           );
   ```



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1532,21 +1512,32 @@ private void runRebalanceCallbacksOnClose() {
 
         int memberEpoch = groupMetadata.get().get().generationId();
 
-        Set<TopicPartition> assignedPartitions = groupAssignmentSnapshot.get();
+        final Exception error;
+
+        if (streamsRebalanceListenerInvoker.isPresent()) {
 
-        if (assignedPartitions.isEmpty())
-            // Nothing to revoke.
-            return;
+            if (memberEpoch > 0)
+                error = streamsRebalanceListenerInvoker.get().invokeAllTasksRevoked();
+            else
+                error = streamsRebalanceListenerInvoker.get().invokeAllTasksLost();
 
-        SortedSet<TopicPartition> droppedPartitions = new TreeSet<>(TOPIC_PARTITION_COMPARATOR);
-        droppedPartitions.addAll(assignedPartitions);
+        } else {
 
-        final Exception error;
+            Set<TopicPartition> assignedPartitions = groupAssignmentSnapshot.get();
 
-        if (memberEpoch > 0)
-            error = rebalanceListenerInvoker.invokePartitionsRevoked(droppedPartitions);
-        else
-            error = rebalanceListenerInvoker.invokePartitionsLost(droppedPartitions);
+            if (assignedPartitions.isEmpty())
+                // Nothing to revoke.
+                return;
+
+            SortedSet<TopicPartition> droppedPartitions = new TreeSet<>(TOPIC_PARTITION_COMPARATOR);
+            droppedPartitions.addAll(assignedPartitions);
+
+            if (memberEpoch > 0)
+                error = rebalanceListenerInvoker.invokePartitionsRevoked(droppedPartitions);
+            else
+                error = rebalanceListenerInvoker.invokePartitionsLost(droppedPartitions);

Review Comment:
   The if-else statements on lines 1535-1538 should use braces for consistency with the project's coding style, even for single-line statements.
   ```suggestion
               if (memberEpoch > 0) {
                   error = rebalanceListenerInvoker.invokePartitionsRevoked(droppedPartitions);
               } else {
                   error = rebalanceListenerInvoker.invokePartitionsLost(droppedPartitions);
               }
   ```



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1532,21 +1512,32 @@ private void runRebalanceCallbacksOnClose() {
 
         int memberEpoch = groupMetadata.get().get().generationId();
 
-        Set<TopicPartition> assignedPartitions = groupAssignmentSnapshot.get();
+        final Exception error;
+
+        if (streamsRebalanceListenerInvoker.isPresent()) {
 
-        if (assignedPartitions.isEmpty())
-            // Nothing to revoke.
-            return;
+            if (memberEpoch > 0)
+                error = streamsRebalanceListenerInvoker.get().invokeAllTasksRevoked();
+            else
+                error = streamsRebalanceListenerInvoker.get().invokeAllTasksLost();

Review Comment:
   The if-else statements on lines 1519-1522 should use braces for consistency with the project's coding style, even for single-line statements.
   ```suggestion
               if (memberEpoch > 0) {
                   error = streamsRebalanceListenerInvoker.get().invokeAllTasksRevoked();
               } else {
                   error = streamsRebalanceListenerInvoker.get().invokeAllTasksLost();
               }
   ```



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/DefaultStreamsRebalanceListenerTest.java:
##########
@@ -170,39 +167,56 @@ void testOnTasksAssigned() {
         );
         inOrder.verify(streamThread).setState(StreamThread.State.PARTITIONS_ASSIGNED);
         inOrder.verify(taskManager).handleRebalanceComplete();
+        inOrder.verify(streamsRebalanceData).setReconciledAssignment(assignment);
     }
 
     @Test
     void testOnTasksAssignedWithException() {
         final Exception exception = new RuntimeException("sample exception");
         doThrow(exception).when(taskManager).handleAssignment(any(), any());
 
-        createRebalanceListenerWithRebalanceData(new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of()));
-        final Optional<Exception> result = defaultStreamsRebalanceListener.onTasksAssigned(new StreamsRebalanceData.Assignment(Set.of(), Set.of(), Set.of()));
-        assertTrue(defaultStreamsRebalanceListener.onAllTasksLost().isEmpty());
+        final StreamsRebalanceData streamsRebalanceData = mock(StreamsRebalanceData.class);
+        when(streamsRebalanceData.subtopologies()).thenReturn(Map.of());
+        createRebalanceListenerWithRebalanceData(streamsRebalanceData);
+        
+        final Optional<Exception> result = defaultStreamsRebalanceListener.onTasksAssigned(
+            new StreamsRebalanceData.Assignment(Set.of(), Set.of(), Set.of())
+        );
+        
         assertTrue(result.isPresent());
         assertEquals(exception, result.get());
-        verify(taskManager).handleLostAll();
+        verify(taskManager).handleAssignment(any(), any());
         verify(streamThread, never()).setState(StreamThread.State.PARTITIONS_ASSIGNED);
         verify(taskManager, never()).handleRebalanceComplete();
+        verify(streamsRebalanceData, never()).setReconciledAssignment(any());
     }
 
     @Test
     void testOnAllTasksLost() {
-        createRebalanceListenerWithRebalanceData(new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of()));
+        final StreamsRebalanceData streamsRebalanceData = mock(StreamsRebalanceData.class);
+        when(streamsRebalanceData.subtopologies()).thenReturn(Map.of());
+        createRebalanceListenerWithRebalanceData(streamsRebalanceData);
+        
         assertTrue(defaultStreamsRebalanceListener.onAllTasksLost().isEmpty());
-        verify(taskManager).handleLostAll();
+        
+        final InOrder inOrder = inOrder(taskManager, streamsRebalanceData);
+        inOrder.verify(taskManager).handleLostAll();
+        inOrder.verify(streamsRebalanceData).setReconciledAssignment(StreamsRebalanceData.Assignment.EMPTY);
     }
 
     @Test
     void testOnAllTasksLostWithException() {
         final Exception exception = new RuntimeException("sample exception");
         doThrow(exception).when(taskManager).handleLostAll();
 
-        createRebalanceListenerWithRebalanceData(new StreamsRebalanceData(UUID.randomUUID(), Optional.empty(), Map.of(), Map.of()));
+        final StreamsRebalanceData streamsRebalanceData = mock(StreamsRebalanceData.class);
+        when(streamsRebalanceData.subtopologies()).thenReturn(Map.of());
+        createRebalanceListenerWithRebalanceData(streamsRebalanceData);
+        

Review Comment:
   Line 215 contains unnecessary trailing whitespace that should be removed.
   ```suggestion
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2339120707


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -278,44 +269,28 @@ private void processStreamsOnAllTasksLostCallbackNeededEvent(final StreamsOnAllT
 
         private StreamsOnTasksRevokedCallbackCompletedEvent invokeOnTasksRevokedCallback(final Set<StreamsRebalanceData.TaskId> activeTasksToRevoke,
                                                                                          final CompletableFuture<Void> future) {
-            final Optional<Exception> exceptionFromCallback = streamsRebalanceListener().onTasksRevoked(activeTasksToRevoke);
+            final Optional<Exception> exceptionFromCallback = Optional.ofNullable(streamsRebalanceListenerInvoker().invokeTasksRevoked(activeTasksToRevoke));
             final Optional<KafkaException> error = exceptionFromCallback.map(e -> ConsumerUtils.maybeWrapAsKafkaException(e, "Task revocation callback throws an error"));

Review Comment:
   +1
   It's reasonable to encapsulate these logics into the constructor, and it is also much easier to trace the whole flow of event construction. (We don't need to look at this helper method to understand the error shaping, etc)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2338979903


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null

Review Comment:
   ditto



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener

Review Comment:
   Do we need this comment?
   I suppose the test case name is clear enough.



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null
+        invoker.setRebalanceListener(null);
+        
+        // Should behave as if no listener is set
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListenerOverwritesExisting() {
+        StreamsRebalanceListener firstListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        StreamsRebalanceListener secondListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(secondListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Set first listener
+        invoker.setRebalanceListener(firstListener);
+        
+        // Overwrite with second listener
+        invoker.setRebalanceListener(secondListener);
+        
+        // Should use second listener
+        invoker.invokeAllTasksRevoked();
+        verify(firstListener, never()).onTasksRevoked(any());
+        verify(secondListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithNoListener() {
+        // When no listener is set, should return null
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result);
+    }
+
+    @Test
+    public void testInvokeAllTasksRevokedWithListener() {
+        invoker.setRebalanceListener(mockListener);
+        
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        Exception result = invoker.invokeAllTasksRevoked();
+        
+        assertNull(result);
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }

Review Comment:
   Yes, they are same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2337597497


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -187,23 +187,14 @@ public class AsyncKafkaConsumer<K, V> implements ConsumerDelegate<K, V> {
      */
     private class BackgroundEventProcessor implements EventProcessor<BackgroundEvent> {
 
-        private Optional<StreamsRebalanceListener> streamsRebalanceListener = Optional.empty();
-        private final Optional<StreamsRebalanceData> streamsRebalanceData;
+        private final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker;

Review Comment:
   AsyncConsumer can be used with "plain consumer" or inside "Kafka Streams". We would use this new handler only for the KS case, but not the "plain consumer" case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "UladzislauBlok (via GitHub)" <gi...@apache.org>.
UladzislauBlok commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2337685271


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        this.listener = Optional.ofNullable(streamsRebalanceListener);
+    }
+
+    public Exception invokeAllTasksRevoked() {
+        if (listener.isPresent()) {
+            return invokeTasksRevoked(streamsRebalanceData.reconciledAssignment().activeTasks());
+        }
+
+        return null;
+    }
+
+    public Exception invokeTasksAssigned(final StreamsRebalanceData.Assignment assignment) {
+        if (listener.isPresent()) {
+            log.info("Adding newly assigned tasks: {}", assignment);
+            try {
+                listener.get().onTasksAssigned(assignment);
+            } catch (WakeupException | InterruptException e) {

Review Comment:
   Okay, it's custom one, but bad



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        this.listener = Optional.ofNullable(streamsRebalanceListener);
+    }
+
+    public Exception invokeAllTasksRevoked() {
+        if (listener.isPresent()) {
+            return invokeTasksRevoked(streamsRebalanceData.reconciledAssignment().activeTasks());
+        }
+
+        return null;
+    }
+
+    public Exception invokeTasksAssigned(final StreamsRebalanceData.Assignment assignment) {
+        if (listener.isPresent()) {
+            log.info("Adding newly assigned tasks: {}", assignment);
+            try {
+                listener.get().onTasksAssigned(assignment);
+            } catch (WakeupException | InterruptException e) {

Review Comment:
   Okay, it's custom one, my bad



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2349320450


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        Objects.requireNonNull(streamsRebalanceListener, "StreamsRebalanceListener cannot be null");
+        if (listener.isPresent() && listener.get() != streamsRebalanceListener) {
+            throw new IllegalStateException("StreamsRebalanceListener can only be set once");

Review Comment:
   Streams will always use the same rebalance listener to subscribe



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2337597497


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -187,23 +187,14 @@ public class AsyncKafkaConsumer<K, V> implements ConsumerDelegate<K, V> {
      */
     private class BackgroundEventProcessor implements EventProcessor<BackgroundEvent> {
 
-        private Optional<StreamsRebalanceListener> streamsRebalanceListener = Optional.empty();
-        private final Optional<StreamsRebalanceData> streamsRebalanceData;
+        private final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker;

Review Comment:
   AsyncConsumer can be used with "plain consumer" or inside "Kafka Streams". We would use this new handler only for the KS case, but not the "plain consumer" case.
   
   And L292 is only executed for the KS case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #20511:
URL: https://github.com/apache/kafka/pull/20511#issuecomment-3279704361

   Wow, so many reviews. Thanks a lot. I addressed the comments @mjsax @frankvicky @lianetm @UladzislauBlok 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2349332471


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1527,26 +1495,39 @@ private void autoCommitOnClose(final Timer timer) {
     }
 
     private void runRebalanceCallbacksOnClose() {
-        if (groupMetadata.get().isEmpty() || rebalanceListenerInvoker == null)
+        if (groupMetadata.get().isEmpty())
             return;
 
         int memberEpoch = groupMetadata.get().get().generationId();
 
-        Set<TopicPartition> assignedPartitions = groupAssignmentSnapshot.get();
+        final Exception error;
 
-        if (assignedPartitions.isEmpty())
-            // Nothing to revoke.
-            return;
+        if (streamsRebalanceListenerInvoker.isPresent()) {

Review Comment:
   Wow, that close is sneaky. Fixed.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1527,26 +1495,39 @@ private void autoCommitOnClose(final Timer timer) {
     }
 
     private void runRebalanceCallbacksOnClose() {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2352207430


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        Objects.requireNonNull(streamsRebalanceListener, "StreamsRebalanceListener cannot be null");
+        if (listener.isPresent() && listener.get() != streamsRebalanceListener) {
+            throw new IllegalStateException("StreamsRebalanceListener can only be set once");

Review Comment:
   That's actually not true - I think I looked at the wrong code. So I changed it make to allow overwriting the rebalance listener on subscribe



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2348999339


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1527,26 +1495,39 @@ private void autoCommitOnClose(final Timer timer) {
     }
 
     private void runRebalanceCallbacksOnClose() {
-        if (groupMetadata.get().isEmpty() || rebalanceListenerInvoker == null)
+        if (groupMetadata.get().isEmpty())
             return;
 
         int memberEpoch = groupMetadata.get().get().generationId();
 
-        Set<TopicPartition> assignedPartitions = groupAssignmentSnapshot.get();
+        final Exception error;
 
-        if (assignedPartitions.isEmpty())
-            // Nothing to revoke.
-            return;
+        if (streamsRebalanceListenerInvoker.isPresent()) {

Review Comment:
   the `streamsRebalanceListenerInvoker` could be null here in the case where `close` is called from the constructor upon a failure building the consumer (ln 509), so maybe consider adding a check here? (we've been seeing those noisy NPE logs on close recently actually) 



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        Objects.requireNonNull(streamsRebalanceListener, "StreamsRebalanceListener cannot be null");
+        if (listener.isPresent() && listener.get() != streamsRebalanceListener) {
+            throw new IllegalStateException("StreamsRebalanceListener can only be set once");

Review Comment:
   given this validation, should we clear the listener on `unsubscribe`? (otherwise, I expect that subscribe+unsubscribe+subscribe would fail on the last subscribe with this error). 



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1527,26 +1495,39 @@ private void autoCommitOnClose(final Timer timer) {
     }
 
     private void runRebalanceCallbacksOnClose() {

Review Comment:
   since this func is being extended to cover streams tasks now too, should we tweak the error message to make it more generic? (above, `Failed to release group assignment`). Up to you, but maybe something along the lines of "Failed running callbacks" would apply better to both cases



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -278,44 +269,28 @@ private void processStreamsOnAllTasksLostCallbackNeededEvent(final StreamsOnAllT
 
         private StreamsOnTasksRevokedCallbackCompletedEvent invokeOnTasksRevokedCallback(final Set<StreamsRebalanceData.TaskId> activeTasksToRevoke,
                                                                                          final CompletableFuture<Void> future) {
-            final Optional<Exception> exceptionFromCallback = streamsRebalanceListener().onTasksRevoked(activeTasksToRevoke);
+            final Optional<Exception> exceptionFromCallback = Optional.ofNullable(streamsRebalanceListenerInvoker().invokeTasksRevoked(activeTasksToRevoke));
             final Optional<KafkaException> error = exceptionFromCallback.map(e -> ConsumerUtils.maybeWrapAsKafkaException(e, "Task revocation callback throws an error"));

Review Comment:
   uhm I see, I'm ok leaving like this then, was just trying to simplify the optionals readability, gets heavy, but I see the trade-of with the existing logic you mentioned. 



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        this.listener = Optional.ofNullable(streamsRebalanceListener);
+    }
+
+    public Exception invokeAllTasksRevoked() {
+        if (listener.isPresent()) {
+            return invokeTasksRevoked(streamsRebalanceData.reconciledAssignment().activeTasks());
+        }
+
+        return null;
+    }
+
+    public Exception invokeTasksAssigned(final StreamsRebalanceData.Assignment assignment) {
+        if (listener.isPresent()) {
+            log.info("Adding newly assigned tasks: {}", assignment);

Review Comment:
   > The consumer rebalance listener logs Adding newly assigned partitions: in this case
   
   I see, well, that's wrong too :) at that point the partitions have been already added really. I will fix it on the consumer side ;) 



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1532,21 +1512,34 @@ private void runRebalanceCallbacksOnClose() {
 

Review Comment:
   ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #20511:
URL: https://github.com/apache/kafka/pull/20511#issuecomment-3377762730

   Cherry-picked to `4.1` branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "UladzislauBlok (via GitHub)" <gi...@apache.org>.
UladzislauBlok commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2337336868


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvokerTest.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class StreamsRebalanceListenerInvokerTest {
+
+    @Mock
+    private StreamsRebalanceListener mockListener;
+
+    @Mock
+    private StreamsRebalanceData streamsRebalanceData;
+
+    private StreamsRebalanceListenerInvoker invoker;
+    private final LogContext logContext = new LogContext();
+
+    @BeforeEach
+    public void setup() {
+        invoker = new StreamsRebalanceListenerInvoker(logContext, streamsRebalanceData);
+    }
+
+    @Test
+    public void testConstructorInitializesWithEmptyListener() {
+        // When invoker is constructed, it should have no listener set initially
+        // This is verified by testing that invoke methods return null when no listener is present
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListener() {
+        // Test setting a listener
+        invoker.setRebalanceListener(mockListener);
+        
+        // Verify listener is set by checking that methods no longer return null immediately
+        // (we'll mock the dependencies needed for actual invocation)
+        StreamsRebalanceData.Assignment mockAssignment = createMockAssignment();
+        when(streamsRebalanceData.reconciledAssignment()).thenReturn(mockAssignment);
+        when(mockListener.onTasksRevoked(any())).thenReturn(Optional.empty());
+        
+        // Should now invoke the listener instead of returning null immediately
+        Exception result = invoker.invokeAllTasksRevoked();
+        assertNull(result); // No exception thrown by mock listener
+        verify(mockListener).onTasksRevoked(eq(mockAssignment.activeTasks()));
+    }
+
+    @Test
+    public void testSetRebalanceListenerWithNull() {
+        // Test setting listener to null
+        invoker.setRebalanceListener(null);
+        
+        // Should behave as if no listener is set
+        assertNull(invoker.invokeAllTasksRevoked());
+        assertNull(invoker.invokeAllTasksLost());
+    }
+
+    @Test
+    public void testSetRebalanceListenerOverwritesExisting() {
+        StreamsRebalanceListener firstListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);
+        StreamsRebalanceListener secondListener = org.mockito.Mockito.mock(StreamsRebalanceListener.class);

Review Comment:
   minor: full import?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -187,23 +187,14 @@ public class AsyncKafkaConsumer<K, V> implements ConsumerDelegate<K, V> {
      */
     private class BackgroundEventProcessor implements EventProcessor<BackgroundEvent> {
 
-        private Optional<StreamsRebalanceListener> streamsRebalanceListener = Optional.empty();
-        private final Optional<StreamsRebalanceData> streamsRebalanceData;
+        private final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker;

Review Comment:
   out of curiosity: why this is optional, if we anyway we'll throw exception at line 292 in case of null?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        this.listener = Optional.ofNullable(streamsRebalanceListener);
+    }
+
+    public Exception invokeAllTasksRevoked() {
+        if (listener.isPresent()) {
+            return invokeTasksRevoked(streamsRebalanceData.reconciledAssignment().activeTasks());
+        }
+
+        return null;
+    }
+
+    public Exception invokeTasksAssigned(final StreamsRebalanceData.Assignment assignment) {
+        if (listener.isPresent()) {
+            log.info("Adding newly assigned tasks: {}", assignment);
+            try {
+                listener.get().onTasksAssigned(assignment);
+            } catch (WakeupException | InterruptException e) {

Review Comment:
   No interruption reset?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "UladzislauBlok (via GitHub)" <gi...@apache.org>.
UladzislauBlok commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2337131031


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -187,23 +187,14 @@ public class AsyncKafkaConsumer<K, V> implements ConsumerDelegate<K, V> {
      */
     private class BackgroundEventProcessor implements EventProcessor<BackgroundEvent> {
 
-        private Optional<StreamsRebalanceListener> streamsRebalanceListener = Optional.empty();
-        private final Optional<StreamsRebalanceData> streamsRebalanceData;
+        private final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker;

Review Comment:
   out of curiosity: why this is optional, if we anyway will throw exception at line 292 in case of empty optional?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19694: Trigger StreamsRebalanceListener in Consumer.close [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20511:
URL: https://github.com/apache/kafka/pull/20511#discussion_r2337602146


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -187,23 +187,14 @@ public class AsyncKafkaConsumer<K, V> implements ConsumerDelegate<K, V> {
      */
     private class BackgroundEventProcessor implements EventProcessor<BackgroundEvent> {
 
-        private Optional<StreamsRebalanceListener> streamsRebalanceListener = Optional.empty();
-        private final Optional<StreamsRebalanceData> streamsRebalanceData;
+        private final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker;
 
         public BackgroundEventProcessor() {
-            this.streamsRebalanceData = Optional.empty();
+            this.streamsRebalanceListenerInvoker = Optional.empty();
         }
 
-        public BackgroundEventProcessor(final Optional<StreamsRebalanceData> streamsRebalanceData) {
-            this.streamsRebalanceData = streamsRebalanceData;
-        }
-
-        private void setStreamsRebalanceListener(final StreamsRebalanceListener streamsRebalanceListener) {
-            if (streamsRebalanceData.isEmpty()) {
-                throw new IllegalStateException("Background event processor was not created to be used with Streams " +
-                    "rebalance protocol events");
-            }
-            this.streamsRebalanceListener = Optional.of(streamsRebalanceListener);
+        public BackgroundEventProcessor(final Optional<StreamsRebalanceListenerInvoker> streamsRebalanceListenerInvoker) {

Review Comment:
   For this constructor, it seems not to make sense to pass in an `Optional`? If we are in the KS case, and use this constructor, we must pass the `StreamsRebalanceListenerInvoker`.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -278,44 +269,28 @@ private void processStreamsOnAllTasksLostCallbackNeededEvent(final StreamsOnAllT
 
         private StreamsOnTasksRevokedCallbackCompletedEvent invokeOnTasksRevokedCallback(final Set<StreamsRebalanceData.TaskId> activeTasksToRevoke,
                                                                                          final CompletableFuture<Void> future) {
-            final Optional<Exception> exceptionFromCallback = streamsRebalanceListener().onTasksRevoked(activeTasksToRevoke);
+            final Optional<Exception> exceptionFromCallback = Optional.ofNullable(streamsRebalanceListenerInvoker().invokeTasksRevoked(activeTasksToRevoke));
             final Optional<KafkaException> error = exceptionFromCallback.map(e -> ConsumerUtils.maybeWrapAsKafkaException(e, "Task revocation callback throws an error"));

Review Comment:
   Would this work as expected, given that `exceptionFromCallback` is created as ofNullable? It seems, if `e == null`, we would set `error = new KafkaException("Task assignment callback throws an error", null)` what does not look correct?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;

Review Comment:
   Given that we use `StreamsRebalanceListenerInvoker` as `Optional` (what does make sense), I am wondering if we should use `Optional` for the `listener`? We would only use `StreamsRebalanceListenerInvoker` for the KS case, but if we are in the KS case, we must have a `StreamsRebalanceListener`, right?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -278,44 +269,28 @@ private void processStreamsOnAllTasksLostCallbackNeededEvent(final StreamsOnAllT
 
         private StreamsOnTasksRevokedCallbackCompletedEvent invokeOnTasksRevokedCallback(final Set<StreamsRebalanceData.TaskId> activeTasksToRevoke,
                                                                                          final CompletableFuture<Void> future) {
-            final Optional<Exception> exceptionFromCallback = streamsRebalanceListener().onTasksRevoked(activeTasksToRevoke);
+            final Optional<Exception> exceptionFromCallback = Optional.ofNullable(streamsRebalanceListenerInvoker().invokeTasksRevoked(activeTasksToRevoke));
             final Optional<KafkaException> error = exceptionFromCallback.map(e -> ConsumerUtils.maybeWrapAsKafkaException(e, "Task revocation callback throws an error"));
             return new StreamsOnTasksRevokedCallbackCompletedEvent(future, error);
         }
 
         private StreamsOnTasksAssignedCallbackCompletedEvent invokeOnTasksAssignedCallback(final StreamsRebalanceData.Assignment assignment,
                                                                                            final CompletableFuture<Void> future) {
             final Optional<KafkaException> error;
-            final Optional<Exception> exceptionFromCallback = streamsRebalanceListener().onTasksAssigned(assignment);
-            if (exceptionFromCallback.isPresent()) {
-                error = Optional.of(ConsumerUtils.maybeWrapAsKafkaException(exceptionFromCallback.get(), "Task assignment callback throws an error"));
-            } else {
-                error = Optional.empty();
-                streamsRebalanceData().setReconciledAssignment(assignment);
-            }
+            final Optional<Exception> exceptionFromCallback = Optional.ofNullable(streamsRebalanceListenerInvoker().invokeTasksAssigned(assignment));
+            error = exceptionFromCallback.map(e -> ConsumerUtils.maybeWrapAsKafkaException(e, "Task assignment callback throws an error"));

Review Comment:
   As above (same elsewhere).



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsRebalanceListenerInvoker.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
+import org.apache.kafka.common.utils.LogContext;
+
+import org.slf4j.Logger;
+
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * This class encapsulates the invocation of the callback methods defined in the {@link StreamsRebalanceListener}
+ * interface. When streams group task assignment changes, these methods are invoked. This class wraps those
+ * callback calls with some logging and error handling.
+ */
+public class StreamsRebalanceListenerInvoker {
+
+    private final Logger log;
+
+    private final StreamsRebalanceData streamsRebalanceData;
+    private Optional<StreamsRebalanceListener> listener;
+
+    StreamsRebalanceListenerInvoker(LogContext logContext, StreamsRebalanceData streamsRebalanceData) {
+        this.log = logContext.logger(getClass());
+        this.listener = Optional.empty();
+        this.streamsRebalanceData = streamsRebalanceData;
+    }
+
+    public void setRebalanceListener(StreamsRebalanceListener streamsRebalanceListener) {
+        this.listener = Optional.ofNullable(streamsRebalanceListener);

Review Comment:
   I don't think that `streamsRebalanceListener` should be allowed to be `null`?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1964,7 +1957,11 @@ public void subscribe(Collection<String> topics, ConsumerRebalanceListener liste
 
     public void subscribe(Collection<String> topics, StreamsRebalanceListener streamsRebalanceListener) {
         subscribeInternal(topics, Optional.empty());
-        backgroundEventProcessor.setStreamsRebalanceListener(streamsRebalanceListener);
+        if (streamsRebalanceListenerInvoker.isPresent()) {

Review Comment:
   Can we simplify this to `orElseThrow` ?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:
##########
@@ -1532,21 +1512,34 @@ private void runRebalanceCallbacksOnClose() {
 

Review Comment:
   Cannot comment above, but there is:
   ```
           if (groupMetadata.get().isEmpty() || rebalanceListenerInvoker == null)
               return;
   ```
   
   I assume we only have `rebalanceListenerInvoker` set, for the plain consumer case? So for KS case, we would `return` above? Don't we need to change this code, too?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org