You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "jsancio (via GitHub)" <gi...@apache.org> on 2025/04/29 20:16:34 UTC
[PR] MINOR; Log message for unexpected buffer allocation [kafka]
jsancio opened a new pull request, #19596:
URL: https://github.com/apache/kafka/pull/19596
Log a message when reading a batch that is larger than the currently allocated batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR; Log message for unexpected buffer allocation [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #19596:
URL: https://github.com/apache/kafka/pull/19596#discussion_r2067638473
##########
raft/src/main/java/org/apache/kafka/snapshot/Snapshots.java:
##########
@@ -159,23 +160,26 @@ public static Path markForDelete(Path logDir, OffsetAndEpoch snapshotId) {
}
}
- public static long lastContainedLogTimestamp(RawSnapshotReader reader) {
- try (RecordsSnapshotReader<ByteBuffer> recordsSnapshotReader =
- RecordsSnapshotReader.of(
- reader,
- IdentitySerde.INSTANCE,
- new BufferSupplier.GrowableBufferSupplier(),
- KafkaRaftClient.MAX_BATCH_SIZE_BYTES,
- true
- )
- ) {
- return recordsSnapshotReader.lastContainedLogTimestamp();
+ public static long lastContainedLogTimestamp(RawSnapshotReader reader, LogContext logContext) {
+ try (var bufferSupplier = new BufferSupplier.GrowableBufferSupplier()) {
+ try (RecordsSnapshotReader<ByteBuffer> recordsSnapshotReader =
+ RecordsSnapshotReader.of(
+ reader,
+ IdentitySerde.INSTANCE,
+ bufferSupplier,
+ KafkaRaftClient.MAX_BATCH_SIZE_BYTES,
+ true,
+ logContext
+ )
+ ) {
+ return recordsSnapshotReader.lastContainedLogTimestamp();
+ }
}
Review Comment:
How about using a single try block for this?
```suggestion
try (var bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
RecordsSnapshotReader<ByteBuffer> recordsSnapshotReader = RecordsSnapshotReader.of(
reader,
IdentitySerde.INSTANCE,
bufferSupplier,
KafkaRaftClient.MAX_BATCH_SIZE_BYTES,
true,
logContext
)) {
return recordsSnapshotReader.lastContainedLogTimestamp();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR; Log message for unexpected buffer allocation [kafka]
Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on code in PR #19596:
URL: https://github.com/apache/kafka/pull/19596#discussion_r2067345623
##########
core/src/main/scala/kafka/server/DynamicBrokerConfig.scala:
##########
@@ -204,38 +207,42 @@ object DynamicBrokerConfig {
props.put(key, value)
}
}
- raftManager.replicatedLog.latestSnapshotId().ifPresent(latestSnapshotId => {
- raftManager.replicatedLog.readSnapshot(latestSnapshotId).ifPresent(rawSnapshotReader => {
- val reader = RecordsSnapshotReader.of(
- rawSnapshotReader,
- raftManager.recordSerde,
- BufferSupplier.create(),
- KafkaRaftClient.MAX_BATCH_SIZE_BYTES,
- true
- )
- val dynamicPerBrokerConfigs = new Properties()
- val dynamicDefaultConfigs = new Properties()
- while (reader.hasNext) {
- val batch = reader.next()
- batch.forEach(record => {
- if (record.message().apiKey() == MetadataRecordType.CONFIG_RECORD.id) {
- val configRecord = record.message().asInstanceOf[ConfigRecord]
- if (DynamicBrokerConfig.AllDynamicConfigs.contains(configRecord.name()) &&
- configRecord.resourceType() == ConfigResource.Type.BROKER.id()) {
- if (configRecord.resourceName().isEmpty) {
- putOrRemoveIfNull(dynamicDefaultConfigs, configRecord.name(), configRecord.value())
- } else if (configRecord.resourceName() == config.brokerId.toString) {
- putOrRemoveIfNull(dynamicPerBrokerConfigs, configRecord.name(), configRecord.value())
- }
+ raftManager.replicatedLog.latestSnapshotId().ifPresent { latestSnapshotId =>
+ raftManager.replicatedLog.readSnapshot(latestSnapshotId).ifPresent { rawSnapshotReader =>
+ Using.resource(
+ RecordsSnapshotReader.of(
+ rawSnapshotReader,
+ raftManager.recordSerde,
+ BufferSupplier.create(),
+ KafkaRaftClient.MAX_BATCH_SIZE_BYTES,
+ true,
+ logContext
+ )
+ ) { reader =>
Review Comment:
This wasn't closing the snapshot reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR; Log message for unexpected buffer allocation [kafka]
Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on code in PR #19596:
URL: https://github.com/apache/kafka/pull/19596#discussion_r2074196046
##########
raft/src/main/java/org/apache/kafka/snapshot/Snapshots.java:
##########
@@ -159,23 +160,26 @@ public static Path markForDelete(Path logDir, OffsetAndEpoch snapshotId) {
}
}
- public static long lastContainedLogTimestamp(RawSnapshotReader reader) {
- try (RecordsSnapshotReader<ByteBuffer> recordsSnapshotReader =
- RecordsSnapshotReader.of(
- reader,
- IdentitySerde.INSTANCE,
- new BufferSupplier.GrowableBufferSupplier(),
- KafkaRaftClient.MAX_BATCH_SIZE_BYTES,
- true
- )
- ) {
- return recordsSnapshotReader.lastContainedLogTimestamp();
+ public static long lastContainedLogTimestamp(RawSnapshotReader reader, LogContext logContext) {
+ try (var bufferSupplier = new BufferSupplier.GrowableBufferSupplier()) {
+ try (RecordsSnapshotReader<ByteBuffer> recordsSnapshotReader =
+ RecordsSnapshotReader.of(
+ reader,
+ IdentitySerde.INSTANCE,
+ bufferSupplier,
+ KafkaRaftClient.MAX_BATCH_SIZE_BYTES,
+ true,
+ logContext
+ )
+ ) {
+ return recordsSnapshotReader.lastContainedLogTimestamp();
+ }
}
Review Comment:
Done. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR; Log message for unexpected buffer allocation [kafka]
Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio merged PR #19596:
URL: https://github.com/apache/kafka/pull/19596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org