You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "apoorvmittal10 (via GitHub)" <gi...@apache.org> on 2026/05/27 16:26:15 UTC
[PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
apoorvmittal10 opened a new pull request, #22389:
URL: https://github.com/apache/kafka/pull/22389
The PR adds abstracted classes to move replica manager dependency from share fetch related classes.
The change is also needed by Share Group DLQ manager to read data for specific offset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #22389:
URL: https://github.com/apache/kafka/pull/22389#discussion_r3316076130
##########
server/src/main/java/org/apache/kafka/server/share/LogReader.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.server.share;
+
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.server.storage.log.FetchParams;
+import org.apache.kafka.storage.internals.log.LogReadResult;
+
+import java.util.LinkedHashMap;
+import java.util.Set;
+
+/**
+ * Abstraction for reading records from log.
+ */
+public interface LogReader {
+
+ /**
+ * Read records for the given partitions starting at the specified offsets.
+ *
+ * @param fetchParams The fetch parameters (isolation level, maxBytes, etc.)
+ * @param partitionsToFetch The set of partitions to actually fetch (after filtering erroneous ones)
+ * @param topicPartitionFetchOffsets The fetch offset per partition
+ * @param partitionMaxBytes The max bytes per partition
+ * @return A map of partition to log read result
+ */
+ LinkedHashMap<TopicIdPartition, LogReadResult> read(
Review Comment:
Since it is part ReplicaManager public API - there could be custom code using those and that might prevent this component from being an easy replacement.
We could acknowledge this though and mention the divergence as part of the java doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #22389:
URL: https://github.com/apache/kafka/pull/22389#discussion_r3315956736
##########
server/src/main/java/org/apache/kafka/server/share/LogReader.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.server.share;
+
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.server.storage.log.FetchParams;
+import org.apache.kafka.storage.internals.log.LogReadResult;
+
+import java.util.LinkedHashMap;
+import java.util.Set;
+
+/**
+ * Abstraction for reading records from log.
+ */
+public interface LogReader {
+
+ /**
+ * Read records for the given partitions starting at the specified offsets.
+ *
+ * @param fetchParams The fetch parameters (isolation level, maxBytes, etc.)
+ * @param partitionsToFetch The set of partitions to actually fetch (after filtering erroneous ones)
+ * @param topicPartitionFetchOffsets The fetch offset per partition
+ * @param partitionMaxBytes The max bytes per partition
+ * @return A map of partition to log read result
+ */
+ LinkedHashMap<TopicIdPartition, LogReadResult> read(
Review Comment:
This refactor changes the semantics of the underlying replicaManager API. The original scala impl accepts a `List` of tuples of (topicIdPartitions, PartitionData) which implies same topic topicIdPartition could be included in parameters multiple times with different offset values which the hashmap disallows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22389:
URL: https://github.com/apache/kafka/pull/22389#discussion_r3432367498
##########
core/src/main/java/kafka/server/share/ReplicaManagerLogReader.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.server.share;
+
+import kafka.server.QuotaFactory;
+import kafka.server.ReplicaManager;
+
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.requests.FetchRequest;
+import org.apache.kafka.server.share.LogReader;
+import org.apache.kafka.server.storage.log.FetchParams;
+import org.apache.kafka.storage.internals.log.LogReadResult;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.LinkedHashMap;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+import scala.collection.Seq;
+import scala.jdk.javaapi.CollectionConverters;
+import scala.runtime.BoxedUnit;
+
+/**
+ * Implementation of {@link LogReader} that reads records from the local log
+ * via {@link ReplicaManager#readFromLog}.
+ */
+public class ReplicaManagerLogReader implements LogReader {
+
+ private static final Logger log = LoggerFactory.getLogger(ReplicaManagerLogReader.class);
+
+ private final ReplicaManager replicaManager;
+
+ public ReplicaManagerLogReader(ReplicaManager replicaManager) {
Review Comment:
Yes, I agree that a more powerful interface will likely be required down the road, but I don't see that as being mutually exclusive with starting with a simple interface for now. My point is that as long as we establish proper modularization from day one, we leave the door open for future evolution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22389:
URL: https://github.com/apache/kafka/pull/22389#discussion_r3413431225
##########
core/src/main/java/kafka/server/share/ReplicaManagerLogReader.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.server.share;
+
+import kafka.server.QuotaFactory;
+import kafka.server.ReplicaManager;
+
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.requests.FetchRequest;
+import org.apache.kafka.server.share.LogReader;
+import org.apache.kafka.server.storage.log.FetchParams;
+import org.apache.kafka.storage.internals.log.LogReadResult;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.LinkedHashMap;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+import scala.collection.Seq;
+import scala.jdk.javaapi.CollectionConverters;
+import scala.runtime.BoxedUnit;
+
+/**
+ * Implementation of {@link LogReader} that reads records from the local log
+ * via {@link ReplicaManager#readFromLog}.
+ */
+public class ReplicaManagerLogReader implements LogReader {
+
+ private static final Logger log = LoggerFactory.getLogger(ReplicaManagerLogReader.class);
+
+ private final ReplicaManager replicaManager;
+
+ public ReplicaManagerLogReader(ReplicaManager replicaManager) {
Review Comment:
Out of curiosity, have you considered using lambda functions instead of `ReplicaManager`, and moving those classes out of the core module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #22389:
URL: https://github.com/apache/kafka/pull/22389#discussion_r3316008095
##########
server/src/main/java/org/apache/kafka/server/share/LogReader.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.server.share;
+
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.server.storage.log.FetchParams;
+import org.apache.kafka.storage.internals.log.LogReadResult;
+
+import java.util.LinkedHashMap;
+import java.util.Set;
+
+/**
+ * Abstraction for reading records from log.
+ */
+public interface LogReader {
+
+ /**
+ * Read records for the given partitions starting at the specified offsets.
+ *
+ * @param fetchParams The fetch parameters (isolation level, maxBytes, etc.)
+ * @param partitionsToFetch The set of partitions to actually fetch (after filtering erroneous ones)
+ * @param topicPartitionFetchOffsets The fetch offset per partition
+ * @param partitionMaxBytes The max bytes per partition
+ * @return A map of partition to log read result
+ */
+ LinkedHashMap<TopicIdPartition, LogReadResult> read(
Review Comment:
But I don't think there is any use case where we send same topic id partition with different partition data ever. Do you think there is any?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on PR #22389:
URL: https://github.com/apache/kafka/pull/22389#issuecomment-4556588505
@apoorvmittal10 Build failed I'm afraid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on PR #22389:
URL: https://github.com/apache/kafka/pull/22389#issuecomment-4561576617
Thanks for the PR - overall looks good but some observations:
The interfaces and classes though directly usable in share group based code - do not have any dependency with share related client or server artifacts.
Wouldn't it be better to move them to neutral packages (not including `.share`) so that that can we adopted in other components as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on PR #22389:
URL: https://github.com/apache/kafka/pull/22389#issuecomment-4561672763
> Wouldn't it be better to move them to neutral packages (not including `.share`) so that their general nature could be highlighted?
Yeah, I thought about it and moved them to share package as of now because the usage is primarily there i.e. DelayedShareFetch, SharePartition, SharePartitionManager and the DLQ classes. I was of the opinion that if we do see more usage outside share groups then we should move the classes outside share package in future but not right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 merged PR #22389:
URL: https://github.com/apache/kafka/pull/22389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20610: Added abstraction classes for share fetch reads (1/N) [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #22389:
URL: https://github.com/apache/kafka/pull/22389#discussion_r3426803601
##########
core/src/main/java/kafka/server/share/ReplicaManagerLogReader.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.server.share;
+
+import kafka.server.QuotaFactory;
+import kafka.server.ReplicaManager;
+
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.requests.FetchRequest;
+import org.apache.kafka.server.share.LogReader;
+import org.apache.kafka.server.storage.log.FetchParams;
+import org.apache.kafka.storage.internals.log.LogReadResult;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.LinkedHashMap;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+import scala.collection.Seq;
+import scala.jdk.javaapi.CollectionConverters;
+import scala.runtime.BoxedUnit;
+
+/**
+ * Implementation of {@link LogReader} that reads records from the local log
+ * via {@link ReplicaManager#readFromLog}.
+ */
+public class ReplicaManagerLogReader implements LogReader {
+
+ private static final Logger log = LoggerFactory.getLogger(ReplicaManagerLogReader.class);
+
+ private final ReplicaManager replicaManager;
+
+ public ReplicaManagerLogReader(ReplicaManager replicaManager) {
Review Comment:
I did but there are couple of things, the shim/interface is needed outside of core module for DLQStateManager to read messages from the log. Also I think we need to further enhance this interface to maybe have consolidated definitions for reading from tiered log as again the DLQStateManager shall need something in near future. We might need to further enhance the interface to work with CompletableFuture while making abstraction for reading data where internally the implementation decides to read from local log or tiered log. Making things common will clean up a lot code in DelayedShareFetch which is very complex today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org