You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "arunkumarucet (via GitHub)" <gi...@apache.org> on 2026/02/04 15:54:21 UTC

[PR] Add Kafka 4.x client support and deprecate Kafka 2.0 [pinot]

arunkumarucet opened a new pull request, #17633:
URL: https://github.com/apache/pinot/pull/17633

   ## Summary
   - Adds support for Kafka 4.x clients with a new `pinot-kafka-4.0` module
   - **Deprecates Kafka 2.0** - all Kafka 2.0 consumer factory references now redirect to Kafka 3.0
   - Introduces Kafka 4.1.1 client dependency
   
   ## Changes
   
   ### Kafka 2.0 Deprecation
   - Redirect handled through PluginManager.java
   - Users specifying Kafka 2.0 consumer factory classes will automatically use Kafka 3.0:
     - `org.apache.pinot.core.realtime.impl.kafka2.KafkaConsumerFactory` → `kafka30.KafkaConsumerFactory`
     - `org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory` → `kafka30.KafkaConsumerFactory`
   - No configuration changes required - existing tables using Kafka 2.0 will seamlessly use Kafka 3.0
   
   ### New Module: `pinot-kafka-4.0`
   - Created new stream ingestion module for Kafka 4.x compatibility
   - Implements `KafkaConsumerFactory`, `KafkaPartitionLevelConsumer`, `KafkaStreamMetadataProvider`, and related classes
   - Uses Testcontainers for unit testing (gracefully skips when Docker unavailable)
   
   ### Backward Compatibility Mappings (PluginManager)
   | User Config | Actual Class Used |
   |-------------|-------------------|
   | `kafka20.KafkaConsumerFactory` | `kafka30.KafkaConsumerFactory` |
   | `kafka2.KafkaConsumerFactory` (old) | `kafka30.KafkaConsumerFactory` |
   | `kafka3.KafkaConsumerFactory` (old) | `kafka30.KafkaConsumerFactory` |
   | `kafka4.KafkaConsumerFactory` (old) | `kafka40.KafkaConsumerFactory` |
   
   ### Kafka 4.x Key Changes
   - Kafka 4.x removed ZooKeeper dependency entirely (KRaft-only)
   - Kafka 4.x removed Scala dependency (pure Java clients)
   - Updated `ConsumerRecord` API usage for Kafka 4.x compatibility
   
   ### Testing
   - Added `LLCRealtimeKafka4ClusterIntegrationTest` for end-to-end validation
   - Unit tests use Testcontainers with Docker skip logic for environments without Docker
   - All existing Kafka 3.0 tests continue to pass
   
   ## Test plan
   - [x] Unit tests pass (non-Docker tests)
   - [x] `LLCRealtimeKafka4ClusterIntegrationTest` passes (16 tests)
   - [x] Existing Kafka 3.0 integration tests unaffected
   - [x] Build compiles successfully
   
   ## Configuration
   To use Kafka 4.x, specify in your table config:
   `"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory"
   `
   ## Migration Notes
   - Kafka 2.0 users: No action required. Your existing configuration will automatically use Kafka 3.0.
   - Kafka 3.0 users: No changes needed.
   - Kafka 4.x users: Update your consumer factory class to kafka40.KafkaConsumerFactory.
   
   ## Notes
   - Kafka 4.x clients are backward compatible with Kafka 3.x brokers
   - The integration tests use the existing Kafka 2.x embedded server (client compatibility verified)
   - Unit tests requiring Docker will skip gracefully on machines without Docker


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add Kafka 4.x client support [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #17633:
URL: https://github.com/apache/pinot/pull/17633


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add Kafka 4.x client support [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #17633:
URL: https://github.com/apache/pinot/pull/17633#discussion_r2768943631


##########
pinot-spi/src/main/java/org/apache/pinot/spi/plugin/PluginManager.java:
##########
@@ -95,10 +95,13 @@ public class PluginManager {
       put("org.apache.pinot.filesystem.LocalPinotFS", "org.apache.pinot.spi.filesystem.LocalPinotFS");
 
       // StreamConsumerFactory
+      // Old-style class names mapped to current plugin packages
       put("org.apache.pinot.core.realtime.impl.kafka2.KafkaConsumerFactory",
           "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory");
       put("org.apache.pinot.core.realtime.impl.kafka3.KafkaConsumerFactory",
           "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory");
+      put("org.apache.pinot.core.realtime.impl.kafka4.KafkaConsumerFactory",
+          "org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory");

Review Comment:
   No need for kafka4. 
   It's legacy class rename.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add Kafka 4.x client support [pinot]

Posted by "arunkumarucet (via GitHub)" <gi...@apache.org>.
arunkumarucet commented on PR #17633:
URL: https://github.com/apache/pinot/pull/17633#issuecomment-3857965281

   > lgtm otherwise
   
   Thanks @xiangfu0 , have fixed the review comments. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add Kafka 4.x client support and deprecate Kafka 2.0 [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17633:
URL: https://github.com/apache/pinot/pull/17633#issuecomment-3848601752

   ### :x: 7 Tests Failed:
   | Tests completed | Failed | Passed | Skipped |
   |---|---|---|---|
   | 10106 | 7 | 10099 | 14 |
   <details><summary>View the top 3 failed test(s) by shortest run time</summary>
   
   > <pre><code class="language-python">org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerBackwardCompatibilityTest::testOffsetsExpired</code></pre>
   > <details><summary>Stack Traces | 0.001s run time</summary>
   > 
   > > <pre><code class="language-python">org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory</code></pre>
   > 
   > </details>
   
   
   > <pre><code class="language-python">org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest::testOffsetsExpired</code></pre>
   > <details><summary>Stack Traces | 0.002s run time</summary>
   > 
   > > <pre><code class="language-python">org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory</code></pre>
   > 
   > </details>
   
   
   > <pre><code class="language-python">org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerBackwardCompatibilityTest::testConsumer</code></pre>
   > <details><summary>Stack Traces | 0.022s run time</summary>
   > 
   > > <pre><code class="language-python">org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory</code></pre>
   > 
   > </details>
   
   
   > <pre><code class="language-python">org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest::testConsumer</code></pre>
   > <details><summary>Stack Traces | 0.024s run time</summary>
   > 
   > > <pre><code class="language-python">org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory</code></pre>
   > 
   > </details>
   
   </details>
   <details><summary>View the full list of 3 :snowflake: flaky test(s)</summary>
   
   > <pre><code class="language-python">org.apache.pinot.spi.plugin.PluginManagerTest::testBackwardCompatible</code></pre>
   > **Flake rate in main:** 86.05% (Passed 12 times, Failed 74 times)
   > <details><summary>Stack Traces | 0.002s run time</summary>
   > 
   > > <pre><code class="language-python">expected [org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory] but found [org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory]</code></pre>
   > 
   > </details>
   
   
   > <pre><code class="language-python">org.apache.pinot.spi.plugin.PluginManagerTest::testBackwardCompatible</code></pre>
   > **Flake rate in main:** 86.05% (Passed 12 times, Failed 74 times)
   > <details><summary>Stack Traces | 0.005s run time</summary>
   > 
   > > <pre><code class="language-python">expected [org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory] but found [org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory]</code></pre>
   > 
   > </details>
   
   
   > <pre><code class="language-python">org.apache.pinot.spi.plugin.PluginManagerTest::testBackwardCompatible</code></pre>
   > **Flake rate in main:** 86.05% (Passed 12 times, Failed 74 times)
   > <details><summary>Stack Traces | 0.002s run time</summary>
   > 
   > > <pre><code class="language-python">expected [org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory] but found [org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory]</code></pre>
   > 
   > </details>
   
   </details>
   
   To view more test analytics, go to the [Test Analytics Dashboard](https://app.codecov.io/gh/apache/pinot/tests/arunkumarucet%3Amaster)
   <sub>📋 Got 3 mins? [Take this short survey](https://forms.gle/22i53Qa1CySZjA6c7) to help us improve Test Analytics.</sub>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add Kafka 4.x client support and deprecate Kafka 2.0 [pinot]

Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on PR #17633:
URL: https://github.com/apache/pinot/pull/17633#issuecomment-3851563161

   Theres a PR already to remove 2.0 https://github.com/apache/pinot/pull/17602


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add Kafka 4.x client support [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17633:
URL: https://github.com/apache/pinot/pull/17633#discussion_r2768602893


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-4.0/src/test/java/org/apache/pinot/plugin/stream/kafka40/utils/MiniKafkaCluster.java:
##########
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka40.utils;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.admin.RecordsToDelete;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.TopicPartition;
+import org.testcontainers.kafka.KafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+
+/**
+ * MiniKafkaCluster for Kafka 4.x using Testcontainers with KRaft mode (no ZooKeeper).
+ * Uses the apache/kafka-native image for fast startup.
+ */
+public final class MiniKafkaCluster implements Closeable {
+
+  private static final String KAFKA_IMAGE = "apache/kafka:4.0.0";
+
+  private final KafkaContainer _kafkaContainer;
+  private final AdminClient _adminClient;

Review Comment:
   The field `_adminClient` is declared but initialized to null in the constructor and never assigned a value. It is also checked for null in the `close()` method but can never be non-null. This field should either be removed or properly initialized if it's needed for future functionality.



##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-4.0/src/main/java/org/apache/pinot/plugin/stream/kafka40/KafkaStreamMetadataProvider.java:
##########
@@ -0,0 +1,287 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka40;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.time.Clock;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.stream.Collectors;
+import org.apache.commons.collections4.CollectionUtils;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.ListTopicsResult;
+import org.apache.kafka.clients.consumer.OffsetAndTimestamp;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.TimeoutException;
+import org.apache.pinot.plugin.stream.kafka.KafkaConsumerPartitionLag;
+import org.apache.pinot.spi.stream.ConsumerPartitionState;
+import org.apache.pinot.spi.stream.LongMsgOffset;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.PartitionLagState;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamMessageMetadata;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.TransientConsumerException;
+import org.apache.pinot.spi.utils.TimeUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class KafkaStreamMetadataProvider extends KafkaPartitionLevelConnectionHandler
+    implements StreamMetadataProvider {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(KafkaStreamMetadataProvider.class);
+
+  public KafkaStreamMetadataProvider(String clientId, StreamConfig streamConfig) {
+    this(clientId, streamConfig, Integer.MIN_VALUE);
+  }
+
+  public KafkaStreamMetadataProvider(String clientId, StreamConfig streamConfig, int partition) {
+    super(clientId, streamConfig, partition);
+  }
+
+  @Override
+  public int fetchPartitionCount(long timeoutMillis) {
+    try {
+      List<PartitionInfo> partitionInfos = _consumer.partitionsFor(_topic, Duration.ofMillis(timeoutMillis));
+      if (CollectionUtils.isNotEmpty(partitionInfos)) {
+        return partitionInfos.size();
+      }
+      throw new RuntimeException(String.format("Failed to fetch partition information for topic: %s", _topic));
+    } catch (TimeoutException e) {
+      throw new TransientConsumerException(e);
+    }
+  }
+
+  @Override
+  public Set<Integer> fetchPartitionIds(long timeoutMillis) {
+    try {
+      List<PartitionInfo> partitionInfos = _consumer.partitionsFor(_topic, Duration.ofMillis(timeoutMillis));
+      if (CollectionUtils.isEmpty(partitionInfos)) {
+        throw new RuntimeException(String.format("Failed to fetch partition information for topic: %s", _topic));
+      }
+      Set<Integer> partitionIds = Sets.newHashSetWithExpectedSize(partitionInfos.size());
+      for (PartitionInfo partitionInfo : partitionInfos) {
+        partitionIds.add(partitionInfo.partition());
+      }
+      return partitionIds;
+    } catch (TimeoutException e) {
+      throw new TransientConsumerException(e);
+    }
+  }
+
+  @Override
+  public Map<Integer, StreamPartitionMsgOffset> fetchLatestStreamOffset(Set<Integer> partitionIds, long timeoutMillis) {
+    List<TopicPartition> topicPartitions = new ArrayList<>(partitionIds.size());
+    for (Integer streamPartition: partitionIds) {
+      topicPartitions.add(new TopicPartition(_topic, streamPartition));
+    }
+    try {
+      Map<TopicPartition, Long> topicPartitionToLatestOffsetMap =
+          _consumer.endOffsets(topicPartitions, Duration.ofMillis(timeoutMillis));
+
+      Map<Integer, StreamPartitionMsgOffset> partitionIdToLatestOffset =
+          new HashMap<>(topicPartitionToLatestOffsetMap.size());
+      for (Map.Entry<TopicPartition, Long> entry : topicPartitionToLatestOffsetMap.entrySet()) {
+        partitionIdToLatestOffset.put(entry.getKey().partition(), new LongMsgOffset(entry.getValue()));
+      }
+
+      return partitionIdToLatestOffset;
+    } catch (TimeoutException e) {
+      throw new TransientConsumerException(e);
+    }
+  }
+
+  @Override
+  public StreamPartitionMsgOffset fetchStreamPartitionOffset(OffsetCriteria offsetCriteria, long timeoutMillis) {
+    Preconditions.checkNotNull(offsetCriteria);
+    long offset;
+    try {
+      if (offsetCriteria.isLargest()) {
+        offset = _consumer.endOffsets(Collections.singletonList(_topicPartition), Duration.ofMillis(timeoutMillis))
+            .get(_topicPartition);
+      } else if (offsetCriteria.isSmallest()) {
+        offset =
+            _consumer.beginningOffsets(Collections.singletonList(_topicPartition), Duration.ofMillis(timeoutMillis))
+                .get(_topicPartition);
+      } else if (offsetCriteria.isPeriod()) {
+        OffsetAndTimestamp offsetAndTimestamp = _consumer.offsetsForTimes(Collections.singletonMap(_topicPartition,
+                Clock.systemUTC().millis() - TimeUtils.convertPeriodToMillis(offsetCriteria.getOffsetString())))
+            .get(_topicPartition);
+        if (offsetAndTimestamp == null) {
+          offset = _consumer.endOffsets(Collections.singletonList(_topicPartition), Duration.ofMillis(timeoutMillis))
+              .get(_topicPartition);
+          LOGGER.warn(
+              "initial offset type is period and its value evaluates to null hence proceeding with offset {} for "
+                  + "topic {} partition {}", offset, _topicPartition.topic(), _topicPartition.partition());
+        } else {
+          offset = offsetAndTimestamp.offset();
+        }
+      } else if (offsetCriteria.isTimestamp()) {
+        OffsetAndTimestamp offsetAndTimestamp = _consumer.offsetsForTimes(Collections.singletonMap(_topicPartition,
+            TimeUtils.convertTimestampToMillis(offsetCriteria.getOffsetString()))).get(_topicPartition);
+        if (offsetAndTimestamp == null) {
+          offset = _consumer.endOffsets(Collections.singletonList(_topicPartition), Duration.ofMillis(timeoutMillis))
+              .get(_topicPartition);
+          LOGGER.warn(
+              "initial offset type is timestamp and its value evaluates to null hence proceeding with offset {} for "
+                  + "topic {} partition {}", offset, _topicPartition.topic(), _topicPartition.partition());
+        } else {
+          offset = offsetAndTimestamp.offset();
+        }
+      } else {
+        throw new IllegalArgumentException("Unknown initial offset value " + offsetCriteria);
+      }
+      return new LongMsgOffset(offset);
+    } catch (TimeoutException e) {
+      throw new TransientConsumerException(e);
+    }
+  }
+
+  @Override
+  public Map<String, PartitionLagState> getCurrentPartitionLagState(
+      Map<String, ConsumerPartitionState> currentPartitionStateMap) {
+    Map<String, PartitionLagState> perPartitionLag = new HashMap<>();
+    for (Map.Entry<String, ConsumerPartitionState> entry : currentPartitionStateMap.entrySet()) {
+      ConsumerPartitionState partitionState = entry.getValue();
+      // Compute records-lag
+      StreamPartitionMsgOffset currentOffset = partitionState.getCurrentOffset();
+      StreamPartitionMsgOffset upstreamLatest = partitionState.getUpstreamLatestOffset();
+      String offsetLagString = "UNKNOWN";
+
+      if (currentOffset instanceof LongMsgOffset && upstreamLatest instanceof LongMsgOffset) {
+        long offsetLag = ((LongMsgOffset) upstreamLatest).getOffset() - ((LongMsgOffset) currentOffset).getOffset();
+        offsetLagString = String.valueOf(offsetLag);
+      }
+
+      // Compute record-availability
+      String availabilityLagMs = "UNKNOWN";
+      StreamMessageMetadata lastProcessedMessageMetadata = partitionState.getLastProcessedRowMetadata();
+      if (lastProcessedMessageMetadata != null && partitionState.getLastProcessedTimeMs() > 0) {
+        long availabilityLag =
+            partitionState.getLastProcessedTimeMs() - lastProcessedMessageMetadata.getRecordIngestionTimeMs();
+        availabilityLagMs = String.valueOf(availabilityLag);
+      }
+
+      perPartitionLag.put(entry.getKey(), new KafkaConsumerPartitionLag(offsetLagString, availabilityLagMs));
+    }
+    return perPartitionLag;
+  }
+
+  @Override
+  public List<TopicMetadata> getTopics() {
+    try {
+      AdminClient adminClient = getOrCreateSharedAdminClient();
+      ListTopicsResult result = adminClient.listTopics();
+      if (result == null) {
+        return Collections.emptyList();
+      }
+      return result.names()
+          .get()
+          .stream()
+          .map(topic -> new KafkaTopicMetadata().setName(topic))
+          .collect(Collectors.toList());
+    } catch (ExecutionException | InterruptedException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  @Override
+  public boolean supportsOffsetLag() {
+    return true;
+  }
+
+  public static class KafkaTopicMetadata implements TopicMetadata {
+    private String _name;
+
+    public String getName() {
+      return _name;
+    }
+
+    public KafkaTopicMetadata setName(String name) {
+      _name = name;
+      return this;
+    }
+  }
+
+
+
+  @Override
+  public StreamPartitionMsgOffset getOffsetAtTimestamp(int partitionId, long timestampMillis, long timeoutMillis) {
+    try {
+      OffsetAndTimestamp offsetAndTimestamp = _consumer.offsetsForTimes(Map.of(_topicPartition, timestampMillis),
+          Duration.ofMillis(timeoutMillis)).get(_topicPartition);
+      if (offsetAndTimestamp == null) {
+        return null;
+      }
+      return new LongMsgOffset(offsetAndTimestamp.offset());
+    } catch (Exception e) {
+      LOGGER.warn("Failed to get offset at timestamp {} for partition {}", timestampMillis, partitionId, e);
+      return null;
+    }
+  }
+
+  @Override
+  public Map<String, StreamPartitionMsgOffset> getStreamStartOffsets() {
+    List<PartitionInfo> partitionInfos = _consumer.partitionsFor(_topic);
+    Map<TopicPartition, Long> startOffsets = _consumer.beginningOffsets(
+        partitionInfos.stream()
+            .filter(info -> info != null)
+            .map(info -> new TopicPartition(_topic, info.partition()))

Review Comment:
   Filtering for null PartitionInfo objects is unnecessary. The `partitionsFor()` method returns a list of non-null PartitionInfo objects. If there are no partitions, it returns null or an empty list, but individual elements in the list are never null. This null check adds unnecessary overhead and can be removed.



##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-4.0/src/main/java/org/apache/pinot/plugin/stream/kafka40/KafkaStreamMetadataProvider.java:
##########
@@ -0,0 +1,287 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka40;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.time.Clock;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.stream.Collectors;
+import org.apache.commons.collections4.CollectionUtils;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.ListTopicsResult;
+import org.apache.kafka.clients.consumer.OffsetAndTimestamp;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.TimeoutException;
+import org.apache.pinot.plugin.stream.kafka.KafkaConsumerPartitionLag;
+import org.apache.pinot.spi.stream.ConsumerPartitionState;
+import org.apache.pinot.spi.stream.LongMsgOffset;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.PartitionLagState;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamMessageMetadata;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.TransientConsumerException;
+import org.apache.pinot.spi.utils.TimeUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class KafkaStreamMetadataProvider extends KafkaPartitionLevelConnectionHandler
+    implements StreamMetadataProvider {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(KafkaStreamMetadataProvider.class);
+
+  public KafkaStreamMetadataProvider(String clientId, StreamConfig streamConfig) {
+    this(clientId, streamConfig, Integer.MIN_VALUE);
+  }
+
+  public KafkaStreamMetadataProvider(String clientId, StreamConfig streamConfig, int partition) {
+    super(clientId, streamConfig, partition);
+  }
+
+  @Override
+  public int fetchPartitionCount(long timeoutMillis) {
+    try {
+      List<PartitionInfo> partitionInfos = _consumer.partitionsFor(_topic, Duration.ofMillis(timeoutMillis));
+      if (CollectionUtils.isNotEmpty(partitionInfos)) {
+        return partitionInfos.size();
+      }
+      throw new RuntimeException(String.format("Failed to fetch partition information for topic: %s", _topic));
+    } catch (TimeoutException e) {
+      throw new TransientConsumerException(e);
+    }
+  }
+
+  @Override
+  public Set<Integer> fetchPartitionIds(long timeoutMillis) {
+    try {
+      List<PartitionInfo> partitionInfos = _consumer.partitionsFor(_topic, Duration.ofMillis(timeoutMillis));
+      if (CollectionUtils.isEmpty(partitionInfos)) {
+        throw new RuntimeException(String.format("Failed to fetch partition information for topic: %s", _topic));
+      }
+      Set<Integer> partitionIds = Sets.newHashSetWithExpectedSize(partitionInfos.size());
+      for (PartitionInfo partitionInfo : partitionInfos) {
+        partitionIds.add(partitionInfo.partition());
+      }
+      return partitionIds;
+    } catch (TimeoutException e) {
+      throw new TransientConsumerException(e);
+    }
+  }
+
+  @Override
+  public Map<Integer, StreamPartitionMsgOffset> fetchLatestStreamOffset(Set<Integer> partitionIds, long timeoutMillis) {
+    List<TopicPartition> topicPartitions = new ArrayList<>(partitionIds.size());
+    for (Integer streamPartition: partitionIds) {
+      topicPartitions.add(new TopicPartition(_topic, streamPartition));
+    }
+    try {
+      Map<TopicPartition, Long> topicPartitionToLatestOffsetMap =
+          _consumer.endOffsets(topicPartitions, Duration.ofMillis(timeoutMillis));
+
+      Map<Integer, StreamPartitionMsgOffset> partitionIdToLatestOffset =
+          new HashMap<>(topicPartitionToLatestOffsetMap.size());
+      for (Map.Entry<TopicPartition, Long> entry : topicPartitionToLatestOffsetMap.entrySet()) {
+        partitionIdToLatestOffset.put(entry.getKey().partition(), new LongMsgOffset(entry.getValue()));
+      }
+
+      return partitionIdToLatestOffset;
+    } catch (TimeoutException e) {
+      throw new TransientConsumerException(e);
+    }
+  }
+
+  @Override
+  public StreamPartitionMsgOffset fetchStreamPartitionOffset(OffsetCriteria offsetCriteria, long timeoutMillis) {
+    Preconditions.checkNotNull(offsetCriteria);
+    long offset;
+    try {
+      if (offsetCriteria.isLargest()) {
+        offset = _consumer.endOffsets(Collections.singletonList(_topicPartition), Duration.ofMillis(timeoutMillis))
+            .get(_topicPartition);
+      } else if (offsetCriteria.isSmallest()) {
+        offset =
+            _consumer.beginningOffsets(Collections.singletonList(_topicPartition), Duration.ofMillis(timeoutMillis))
+                .get(_topicPartition);
+      } else if (offsetCriteria.isPeriod()) {
+        OffsetAndTimestamp offsetAndTimestamp = _consumer.offsetsForTimes(Collections.singletonMap(_topicPartition,
+                Clock.systemUTC().millis() - TimeUtils.convertPeriodToMillis(offsetCriteria.getOffsetString())))
+            .get(_topicPartition);
+        if (offsetAndTimestamp == null) {
+          offset = _consumer.endOffsets(Collections.singletonList(_topicPartition), Duration.ofMillis(timeoutMillis))
+              .get(_topicPartition);
+          LOGGER.warn(
+              "initial offset type is period and its value evaluates to null hence proceeding with offset {} for "
+                  + "topic {} partition {}", offset, _topicPartition.topic(), _topicPartition.partition());
+        } else {
+          offset = offsetAndTimestamp.offset();
+        }
+      } else if (offsetCriteria.isTimestamp()) {
+        OffsetAndTimestamp offsetAndTimestamp = _consumer.offsetsForTimes(Collections.singletonMap(_topicPartition,
+            TimeUtils.convertTimestampToMillis(offsetCriteria.getOffsetString()))).get(_topicPartition);
+        if (offsetAndTimestamp == null) {
+          offset = _consumer.endOffsets(Collections.singletonList(_topicPartition), Duration.ofMillis(timeoutMillis))
+              .get(_topicPartition);
+          LOGGER.warn(
+              "initial offset type is timestamp and its value evaluates to null hence proceeding with offset {} for "
+                  + "topic {} partition {}", offset, _topicPartition.topic(), _topicPartition.partition());
+        } else {
+          offset = offsetAndTimestamp.offset();
+        }
+      } else {
+        throw new IllegalArgumentException("Unknown initial offset value " + offsetCriteria);
+      }
+      return new LongMsgOffset(offset);
+    } catch (TimeoutException e) {
+      throw new TransientConsumerException(e);
+    }
+  }
+
+  @Override
+  public Map<String, PartitionLagState> getCurrentPartitionLagState(
+      Map<String, ConsumerPartitionState> currentPartitionStateMap) {
+    Map<String, PartitionLagState> perPartitionLag = new HashMap<>();
+    for (Map.Entry<String, ConsumerPartitionState> entry : currentPartitionStateMap.entrySet()) {
+      ConsumerPartitionState partitionState = entry.getValue();
+      // Compute records-lag
+      StreamPartitionMsgOffset currentOffset = partitionState.getCurrentOffset();
+      StreamPartitionMsgOffset upstreamLatest = partitionState.getUpstreamLatestOffset();
+      String offsetLagString = "UNKNOWN";
+
+      if (currentOffset instanceof LongMsgOffset && upstreamLatest instanceof LongMsgOffset) {
+        long offsetLag = ((LongMsgOffset) upstreamLatest).getOffset() - ((LongMsgOffset) currentOffset).getOffset();
+        offsetLagString = String.valueOf(offsetLag);
+      }
+
+      // Compute record-availability
+      String availabilityLagMs = "UNKNOWN";
+      StreamMessageMetadata lastProcessedMessageMetadata = partitionState.getLastProcessedRowMetadata();
+      if (lastProcessedMessageMetadata != null && partitionState.getLastProcessedTimeMs() > 0) {
+        long availabilityLag =
+            partitionState.getLastProcessedTimeMs() - lastProcessedMessageMetadata.getRecordIngestionTimeMs();
+        availabilityLagMs = String.valueOf(availabilityLag);
+      }
+
+      perPartitionLag.put(entry.getKey(), new KafkaConsumerPartitionLag(offsetLagString, availabilityLagMs));
+    }
+    return perPartitionLag;
+  }
+
+  @Override
+  public List<TopicMetadata> getTopics() {
+    try {
+      AdminClient adminClient = getOrCreateSharedAdminClient();
+      ListTopicsResult result = adminClient.listTopics();
+      if (result == null) {
+        return Collections.emptyList();
+      }
+      return result.names()
+          .get()
+          .stream()
+          .map(topic -> new KafkaTopicMetadata().setName(topic))
+          .collect(Collectors.toList());
+    } catch (ExecutionException | InterruptedException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  @Override
+  public boolean supportsOffsetLag() {
+    return true;
+  }
+
+  public static class KafkaTopicMetadata implements TopicMetadata {
+    private String _name;
+
+    public String getName() {
+      return _name;
+    }
+
+    public KafkaTopicMetadata setName(String name) {
+      _name = name;
+      return this;
+    }
+  }
+
+
+
+  @Override
+  public StreamPartitionMsgOffset getOffsetAtTimestamp(int partitionId, long timestampMillis, long timeoutMillis) {
+    try {
+      OffsetAndTimestamp offsetAndTimestamp = _consumer.offsetsForTimes(Map.of(_topicPartition, timestampMillis),
+          Duration.ofMillis(timeoutMillis)).get(_topicPartition);
+      if (offsetAndTimestamp == null) {
+        return null;
+      }
+      return new LongMsgOffset(offsetAndTimestamp.offset());
+    } catch (Exception e) {
+      LOGGER.warn("Failed to get offset at timestamp {} for partition {}", timestampMillis, partitionId, e);
+      return null;
+    }
+  }
+
+  @Override
+  public Map<String, StreamPartitionMsgOffset> getStreamStartOffsets() {
+    List<PartitionInfo> partitionInfos = _consumer.partitionsFor(_topic);
+    Map<TopicPartition, Long> startOffsets = _consumer.beginningOffsets(
+        partitionInfos.stream()
+            .filter(info -> info != null)
+            .map(info -> new TopicPartition(_topic, info.partition()))
+            .collect(Collectors.toList()));
+    return startOffsets.entrySet().stream().collect(
+        Collectors.toMap(
+            entry -> String.valueOf(entry.getKey().partition()),
+            entry -> new LongMsgOffset(entry.getValue()),
+            (existingValue, newValue) -> newValue
+        ));
+  }
+
+  @Override
+  public Map<String, StreamPartitionMsgOffset> getStreamEndOffsets() {
+    List<PartitionInfo> partitionInfos = _consumer.partitionsFor(_topic);
+    Map<TopicPartition, Long> startOffsets = _consumer.endOffsets(
+        partitionInfos.stream()
+            .filter(info -> info != null)
+            .map(info -> new TopicPartition(_topic, info.partition()))

Review Comment:
   Same as the previous comment - filtering for null PartitionInfo objects is unnecessary. The `partitionsFor()` method returns a list of non-null PartitionInfo objects.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org