You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dajac (via GitHub)" <gi...@apache.org> on 2025/01/30 09:47:39 UTC
[PR] KAFKA-18672; CoordinatorRecordSerde must validate value version [kafka]
dajac opened a new pull request, #18749:
URL: https://github.com/apache/kafka/pull/18749
CoordinatorRecordSerde does not validate the version of the value to check whether the version is supported by the current version of the software. This is problematic if a future and unsupported version of the record is read by an older version of the software because it would misinterpret the bytes. Hence CoordinatorRecordSerde must throw an error if the version is unknown. This is also consistent with the handling in the old coordinator.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18672; CoordinatorRecordSerde must validate value version [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #18749:
URL: https://github.com/apache/kafka/pull/18749#discussion_r1939129722
##########
coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRecordSerde.java:
##########
@@ -76,6 +76,11 @@ public CoordinatorRecord deserialize(
final ApiMessage valueMessage = apiMessageValueFor(recordType);
final short valueVersion = readVersion(valueBuffer, "value");
+
+ if (valueVersion < valueMessage.lowestSupportedVersion() || valueVersion > valueMessage.highestSupportedVersion()) {
+ throw new UnknownRecordVersionException(recordType, valueVersion);
+ }
Review Comment:
That's a good question. I think that we would need to do a new KIP or to extend KIP-915: Txn and Group Coordinator Downgrade Foundation for this. In KIP-915, we defined that using tagged fields should be the way forward to handle backward compatibility. Hence, I am not sure that ignore newer version is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18672; CoordinatorRecordSerde must validate value version [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #18749:
URL: https://github.com/apache/kafka/pull/18749#discussion_r1935311563
##########
coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/Deserializer.java:
##########
@@ -24,6 +24,42 @@
* @param <T> The record type.
*/
public interface Deserializer<T> {
+ /**
+ * UnknownRecordTypeException is thrown when the Deserializer encounters
+ * an unknown record type.
+ */
+ class UnknownRecordTypeException extends RuntimeException {
Review Comment:
I just moved this one from the Loader to here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18672; CoordinatorRecordSerde must validate value version [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #18749:
URL: https://github.com/apache/kafka/pull/18749#discussion_r1938757450
##########
coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRecordSerde.java:
##########
@@ -76,6 +76,11 @@ public CoordinatorRecord deserialize(
final ApiMessage valueMessage = apiMessageValueFor(recordType);
final short valueVersion = readVersion(valueBuffer, "value");
+
+ if (valueVersion < valueMessage.lowestSupportedVersion() || valueVersion > valueMessage.highestSupportedVersion()) {
+ throw new UnknownRecordVersionException(recordType, valueVersion);
+ }
Review Comment:
The `UnknownRecordTypeException` is caught as the aborted upgrade might leave behind some "new" records. Should we also consider handling `UnknownRecordVersionException`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18672; CoordinatorRecordSerde must validate value version [kafka]
Posted by "jeffkbkim (via GitHub)" <gi...@apache.org>.
jeffkbkim commented on code in PR #18749:
URL: https://github.com/apache/kafka/pull/18749#discussion_r1935896162
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupCoordinatorRecordSerdeTest.java:
##########
@@ -198,6 +198,34 @@ public void testDeserializeWithInvalidValueBytes() {
ex.getMessage());
}
+ @Test
+ public void testDeserializeWithInvalidValueVersion() {
+ GroupCoordinatorRecordSerde serde = new GroupCoordinatorRecordSerde();
+
+ ApiMessage key = new ConsumerGroupMetadataKey().setGroupId("foo");
+ ByteBuffer keyBuffer = MessageUtil.toCoordinatorTypePrefixedByteBuffer(key);
+
+ ByteBuffer valueBuffer1 = ByteBuffer.allocate(2);
+ valueBuffer1.putShort((short) (ConsumerGroupMetadataValue.HIGHEST_SUPPORTED_VERSION + 1));
+ valueBuffer1.rewind();
+
+ Deserializer.UnknownRecordVersionException ex =
+ assertThrows(Deserializer.UnknownRecordVersionException.class,
+ () -> serde.deserialize(keyBuffer, valueBuffer1));
+ assertEquals(key.apiKey(), ex.type());
+ assertEquals(ConsumerGroupMetadataValue.HIGHEST_SUPPORTED_VERSION + 1, ex.unknownVersion());
+
+ keyBuffer.rewind();
+ ByteBuffer valueBuffer2 = ByteBuffer.allocate(2);
+ valueBuffer2.putShort((short) (ConsumerGroupMetadataValue.HIGHEST_SUPPORTED_VERSION - 1));
Review Comment:
should this be LOWEST_SUPPORTED_VERSION - 1? also in L226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18672; CoordinatorRecordSerde must validate value version [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #18749:
URL: https://github.com/apache/kafka/pull/18749#discussion_r1935310879
##########
coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRecordSerde.java:
##########
@@ -76,6 +76,11 @@ public CoordinatorRecord deserialize(
final ApiMessage valueMessage = apiMessageValueFor(recordType);
final short valueVersion = readVersion(valueBuffer, "value");
+
+ if (valueVersion < valueMessage.lowestSupportedVersion() || valueVersion > valueMessage.highestSupportedVersion()) {
+ throw new UnknownRecordVersionException(recordType, valueVersion);
+ }
Review Comment:
This is the main change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18672; CoordinatorRecordSerde must validate value version [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac merged PR #18749:
URL: https://github.com/apache/kafka/pull/18749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18672; CoordinatorRecordSerde must validate value version [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #18749:
URL: https://github.com/apache/kafka/pull/18749#discussion_r1937293367
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupCoordinatorRecordSerdeTest.java:
##########
@@ -198,6 +198,34 @@ public void testDeserializeWithInvalidValueBytes() {
ex.getMessage());
}
+ @Test
+ public void testDeserializeWithInvalidValueVersion() {
+ GroupCoordinatorRecordSerde serde = new GroupCoordinatorRecordSerde();
+
+ ApiMessage key = new ConsumerGroupMetadataKey().setGroupId("foo");
+ ByteBuffer keyBuffer = MessageUtil.toCoordinatorTypePrefixedByteBuffer(key);
+
+ ByteBuffer valueBuffer1 = ByteBuffer.allocate(2);
+ valueBuffer1.putShort((short) (ConsumerGroupMetadataValue.HIGHEST_SUPPORTED_VERSION + 1));
+ valueBuffer1.rewind();
+
+ Deserializer.UnknownRecordVersionException ex =
+ assertThrows(Deserializer.UnknownRecordVersionException.class,
+ () -> serde.deserialize(keyBuffer, valueBuffer1));
+ assertEquals(key.apiKey(), ex.type());
+ assertEquals(ConsumerGroupMetadataValue.HIGHEST_SUPPORTED_VERSION + 1, ex.unknownVersion());
+
+ keyBuffer.rewind();
+ ByteBuffer valueBuffer2 = ByteBuffer.allocate(2);
+ valueBuffer2.putShort((short) (ConsumerGroupMetadataValue.HIGHEST_SUPPORTED_VERSION - 1));
Review Comment:
correct. in practice, it does not change much because they are both set to zero.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org