You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "joshua2519 (via GitHub)" <gi...@apache.org> on 2026/03/19 09:54:26 UTC
[PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
joshua2519 opened a new pull request, #21816:
URL: https://github.com/apache/kafka/pull/21816
Fetch.forPartition used Map.of() which creates immutable maps. If add() is called on a forPartition-created Fetch as the target, putAll on the immutable nextOffsetAndMetadata would throw UnsupportedOperationException. Replace with HashMap to ensure mutability.
See https://github.com/apache/kafka/pull/21726#discussion_r2958164222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
Posted by "joshua2519 (via GitHub)" <gi...@apache.org>.
joshua2519 commented on PR #21816:
URL: https://github.com/apache/kafka/pull/21816#issuecomment-4091198426
> @joshua2519 Would you mind adding a unit test to immortalize this brain fart?
I am happy to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #21816:
URL: https://github.com/apache/kafka/pull/21816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21816:
URL: https://github.com/apache/kafka/pull/21816#discussion_r2959003168
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetch.java:
##########
@@ -62,6 +58,21 @@ private Fetch(
this.nextOffsetAndMetadata = nextOffsetAndMetadata;
}
+ private Fetch(
Review Comment:
Could we consolidate these two constructors? For instance, we could pass null to partition and other arguments to represent an 'empty' state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
Posted by "joshua2519 (via GitHub)" <gi...@apache.org>.
joshua2519 commented on code in PR #21816:
URL: https://github.com/apache/kafka/pull/21816#discussion_r2960894651
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetch.java:
##########
@@ -62,6 +58,21 @@ private Fetch(
this.nextOffsetAndMetadata = nextOffsetAndMetadata;
}
+ private Fetch(
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
Posted by "joshua2519 (via GitHub)" <gi...@apache.org>.
joshua2519 commented on PR #21816:
URL: https://github.com/apache/kafka/pull/21816#issuecomment-4095063221
Added FetchTest to cover forPartition mutability, add() merging, positionAdvanced propagation, and unmodifiable return maps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21816:
URL: https://github.com/apache/kafka/pull/21816#discussion_r2964159428
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetch.java:
##########
@@ -43,25 +43,26 @@ public static <K, V> Fetch<K, V> forPartition(
boolean positionAdvanced,
OffsetAndMetadata nextOffsetAndMetadata
) {
- Map<TopicPartition, List<ConsumerRecord<K, V>>> recordsMap = records.isEmpty()
- ? Map.of()
- : Map.of(partition, records);
- Map<TopicPartition, OffsetAndMetadata> nextOffsetAndMetadataMap = Map.of(partition, nextOffsetAndMetadata);
- return new Fetch<>(recordsMap, positionAdvanced, records.size(), nextOffsetAndMetadataMap);
+ return new Fetch<>(positionAdvanced, partition, records, nextOffsetAndMetadata);
}
private Fetch(
- Map<TopicPartition, List<ConsumerRecord<K, V>>> records,
boolean positionAdvanced,
- int numRecords,
- Map<TopicPartition, OffsetAndMetadata> nextOffsetAndMetadata
+ TopicPartition partition,
+ List<ConsumerRecord<K, V>> records,
+ OffsetAndMetadata offsetAndMetadata
) {
- this.records = records;
+ this.records = new HashMap<>();
+ if (partition != null && !records.isEmpty()) {
+ this.records.put(partition, records);
+ }
this.positionAdvanced = positionAdvanced;
- this.numRecords = numRecords;
- this.nextOffsetAndMetadata = nextOffsetAndMetadata;
+ this.numRecords = records.size();
+ this.nextOffsetAndMetadata = new HashMap<>();
+ if (partition != null && offsetAndMetadata != null) {
Review Comment:
My bad! Seeing this merged constructor in action makes me realize the `null` checks make it a bit clunky. Let's go back to your original approach with the two separate constructors. Sorry for the back-and-forth
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetchTest.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class FetchTest {
+
+ private static final TopicPartition TP0 = new TopicPartition("topic", 0);
+ private static final TopicPartition TP1 = new TopicPartition("topic", 1);
+
+ @Test
+ void testAddToForPartitionFetch() {
+ List<ConsumerRecord<String, String>> records0 = List.of(
Review Comment:
```java
var records0 = List.of(new ConsumerRecord<>("topic", 0, 0, "key0", "value0"));
var target = Fetch.forPartition(TP0, records0, true, new OffsetAndMetadata(1, Optional.empty(), ""));
var records1 = List.of(new ConsumerRecord<>("topic", 1, 0, "key1", "value1"));
var source = Fetch.forPartition(TP1, records1, true, new OffsetAndMetadata(1, Optional.empty(), ""));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
Posted by "joshua2519 (via GitHub)" <gi...@apache.org>.
joshua2519 commented on code in PR #21816:
URL: https://github.com/apache/kafka/pull/21816#discussion_r2965944974
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetch.java:
##########
@@ -43,11 +43,7 @@ public static <K, V> Fetch<K, V> forPartition(
boolean positionAdvanced,
OffsetAndMetadata nextOffsetAndMetadata
) {
- Map<TopicPartition, List<ConsumerRecord<K, V>>> recordsMap = records.isEmpty()
- ? Map.of()
- : Map.of(partition, records);
- Map<TopicPartition, OffsetAndMetadata> nextOffsetAndMetadataMap = Map.of(partition, nextOffsetAndMetadata);
- return new Fetch<>(recordsMap, positionAdvanced, records.size(), nextOffsetAndMetadataMap);
+ return new Fetch<>(positionAdvanced, partition, records, nextOffsetAndMetadata);
}
private Fetch(
Review Comment:
Good idea — this also eliminates the constructor that accepts external maps.
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #21816:
URL: https://github.com/apache/kafka/pull/21816#issuecomment-4091165444
@joshua2519 Would you mind adding a unit test to immortalize this brain fart?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
Posted by "joshua2519 (via GitHub)" <gi...@apache.org>.
joshua2519 commented on code in PR #21816:
URL: https://github.com/apache/kafka/pull/21816#discussion_r2965222338
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetchTest.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class FetchTest {
+
+ private static final TopicPartition TP0 = new TopicPartition("topic", 0);
+ private static final TopicPartition TP1 = new TopicPartition("topic", 1);
+
+ @Test
+ void testAddToForPartitionFetch() {
+ List<ConsumerRecord<String, String>> records0 = List.of(
Review Comment:
Done
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetch.java:
##########
@@ -43,25 +43,26 @@ public static <K, V> Fetch<K, V> forPartition(
boolean positionAdvanced,
OffsetAndMetadata nextOffsetAndMetadata
) {
- Map<TopicPartition, List<ConsumerRecord<K, V>>> recordsMap = records.isEmpty()
- ? Map.of()
- : Map.of(partition, records);
- Map<TopicPartition, OffsetAndMetadata> nextOffsetAndMetadataMap = Map.of(partition, nextOffsetAndMetadata);
- return new Fetch<>(recordsMap, positionAdvanced, records.size(), nextOffsetAndMetadataMap);
+ return new Fetch<>(positionAdvanced, partition, records, nextOffsetAndMetadata);
}
private Fetch(
- Map<TopicPartition, List<ConsumerRecord<K, V>>> records,
boolean positionAdvanced,
- int numRecords,
- Map<TopicPartition, OffsetAndMetadata> nextOffsetAndMetadata
+ TopicPartition partition,
+ List<ConsumerRecord<K, V>> records,
+ OffsetAndMetadata offsetAndMetadata
) {
- this.records = records;
+ this.records = new HashMap<>();
+ if (partition != null && !records.isEmpty()) {
+ this.records.put(partition, records);
+ }
this.positionAdvanced = positionAdvanced;
- this.numRecords = numRecords;
- this.nextOffsetAndMetadata = nextOffsetAndMetadata;
+ this.numRecords = records.size();
+ this.nextOffsetAndMetadata = new HashMap<>();
+ if (partition != null && offsetAndMetadata != null) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Use mutable maps in Fetch.forPartition to avoid potential UnsupportedOperationException [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21816:
URL: https://github.com/apache/kafka/pull/21816#discussion_r2965685962
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetch.java:
##########
@@ -43,11 +43,7 @@ public static <K, V> Fetch<K, V> forPartition(
boolean positionAdvanced,
OffsetAndMetadata nextOffsetAndMetadata
) {
- Map<TopicPartition, List<ConsumerRecord<K, V>>> recordsMap = records.isEmpty()
- ? Map.of()
- : Map.of(partition, records);
- Map<TopicPartition, OffsetAndMetadata> nextOffsetAndMetadataMap = Map.of(partition, nextOffsetAndMetadata);
- return new Fetch<>(recordsMap, positionAdvanced, records.size(), nextOffsetAndMetadataMap);
+ return new Fetch<>(positionAdvanced, partition, records, nextOffsetAndMetadata);
}
private Fetch(
Review Comment:
Maybe we could make this constructor more simple.
```java
public static <K, V> Fetch<K, V> empty() {
return new Fetch<>(false, 0);
}
public static <K, V> Fetch<K, V> forPartition(
TopicPartition partition,
List<ConsumerRecord<K, V>> records,
boolean positionAdvanced,
OffsetAndMetadata nextOffsetAndMetadata
) {
return new Fetch<>(positionAdvanced, partition, records, nextOffsetAndMetadata);
}
private Fetch(
boolean positionAdvanced,
int numRecords
) {
this.records = new HashMap<>();
this.positionAdvanced = positionAdvanced;
this.numRecords = numRecords;
this.nextOffsetAndMetadata = new HashMap<>();
}
private Fetch(
boolean positionAdvanced,
TopicPartition partition,
List<ConsumerRecord<K, V>> records,
OffsetAndMetadata offsetAndMetadata
) {
this.records = new HashMap<>();
if (!records.isEmpty()) {
this.records.put(partition, records);
}
this.positionAdvanced = positionAdvanced;
this.numRecords = records.size();
this.nextOffsetAndMetadata = new HashMap<>();
this.nextOffsetAndMetadata.put(partition, offsetAndMetadata);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org