You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/04/07 05:30:24 UTC
[GitHub] [kafka] chia7712 opened a new pull request #10498: MINOR: remove checksumOrNull from Record
chia7712 opened a new pull request #10498:
URL: https://github.com/apache/kafka/pull/10498
Reference: https://github.com/apache/kafka/pull/10470#discussion_r608347559
This PR includes following task
1. rewrite the checksum of `DumpLogSegments`
2. remove `checksumOrNull` from `Record` (blocked by #10470)
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a change in pull request #10498: MINOR: remove `checksumOrNull`,`isValid`, and `ensureValid` from Record
Posted by GitBox <gi...@apache.org>.
chia7712 commented on a change in pull request #10498:
URL: https://github.com/apache/kafka/pull/10498#discussion_r613773275
##########
File path: clients/src/main/java/org/apache/kafka/common/record/Record.java
##########
@@ -132,4 +121,16 @@
* @return the array of headers
*/
Header[] headers();
+
+ static void ensureValid(Record record) {
+ if (record instanceof AbstractLegacyRecordBatch) {
+ ((AbstractLegacyRecordBatch) record).ensureValid();
+ }
+ }
+
+ static boolean isValid(Record record) {
+ if (record instanceof AbstractLegacyRecordBatch) {
+ return ((AbstractLegacyRecordBatch) record).isValid();
+ } else return true;
Review comment:
The method “isVaild” is meaningless for v2 and later. It seems to me it is not worth being a interface. If this change is overkill, alternative is that we can add default implementation to those stale methods. The v2 and later can skip implementation of stale methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a change in pull request #10498: MINOR: remove `checksumOrNull`,`isValid`, and `ensureValid` from Record
Posted by GitBox <gi...@apache.org>.
ijuma commented on a change in pull request #10498:
URL: https://github.com/apache/kafka/pull/10498#discussion_r613768029
##########
File path: clients/src/main/java/org/apache/kafka/common/record/Record.java
##########
@@ -132,4 +121,16 @@
* @return the array of headers
*/
Header[] headers();
+
+ static void ensureValid(Record record) {
+ if (record instanceof AbstractLegacyRecordBatch) {
+ ((AbstractLegacyRecordBatch) record).ensureValid();
+ }
+ }
+
+ static boolean isValid(Record record) {
+ if (record instanceof AbstractLegacyRecordBatch) {
+ return ((AbstractLegacyRecordBatch) record).isValid();
+ } else return true;
Review comment:
Why is this change an improvement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 merged pull request #10498: MINOR: remove `checksumOrNull` and `isValid` from Record
Posted by GitBox <gi...@apache.org>.
chia7712 merged pull request #10498:
URL: https://github.com/apache/kafka/pull/10498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a change in pull request #10498: MINOR: remove `checksumOrNull` and `isValid` from Record
Posted by GitBox <gi...@apache.org>.
ijuma commented on a change in pull request #10498:
URL: https://github.com/apache/kafka/pull/10498#discussion_r614068123
##########
File path: core/src/main/scala/kafka/tools/DumpLogSegments.scala
##########
@@ -279,8 +279,10 @@ object DumpLogSegments {
if (batch.magic >= RecordBatch.MAGIC_VALUE_V2) {
print(" sequence: " + record.sequence + " headerKeys: " + record.headers.map(_.key).mkString("[", ",", "]"))
- } else {
- print(s" crc: ${record.checksumOrNull} isvalid: ${record.isValid}")
+ }
+ record match {
+ case r: AbstractLegacyRecordBatch => print(s" crc: ${r.checksum}}")
Review comment:
Shall we include the `isValid` part here too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a change in pull request #10498: MINOR: remove `checksumOrNull` and `isValid` from Record
Posted by GitBox <gi...@apache.org>.
chia7712 commented on a change in pull request #10498:
URL: https://github.com/apache/kafka/pull/10498#discussion_r614588327
##########
File path: core/src/main/scala/kafka/tools/DumpLogSegments.scala
##########
@@ -279,8 +279,10 @@ object DumpLogSegments {
if (batch.magic >= RecordBatch.MAGIC_VALUE_V2) {
print(" sequence: " + record.sequence + " headerKeys: " + record.headers.map(_.key).mkString("[", ",", "]"))
- } else {
- print(s" crc: ${record.checksumOrNull} isvalid: ${record.isValid}")
+ }
+ record match {
+ case r: AbstractLegacyRecordBatch => print(s" crc: ${r.checksum}}")
Review comment:
sure!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a change in pull request #10498: MINOR: remove checksumOrNull from Record
Posted by GitBox <gi...@apache.org>.
chia7712 commented on a change in pull request #10498:
URL: https://github.com/apache/kafka/pull/10498#discussion_r613753836
##########
File path: core/src/main/scala/kafka/tools/DumpLogSegments.scala
##########
@@ -280,7 +284,7 @@ object DumpLogSegments {
if (batch.magic >= RecordBatch.MAGIC_VALUE_V2) {
print(" sequence: " + record.sequence + " headerKeys: " + record.headers.map(_.key).mkString("[", ",", "]"))
} else {
- print(s" crc: ${record.checksumOrNull} isvalid: ${record.isValid}")
+ print(s" crc: $checksum isvalid: ${record.isValid}")
Review comment:
yep, it printed the checksum twice if magic code is smaller than 2. Will remove the duplicate print.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a change in pull request #10498: MINOR: remove checksumOrNull from Record
Posted by GitBox <gi...@apache.org>.
ijuma commented on a change in pull request #10498:
URL: https://github.com/apache/kafka/pull/10498#discussion_r613742636
##########
File path: core/src/main/scala/kafka/tools/DumpLogSegments.scala
##########
@@ -270,7 +270,11 @@ object DumpLogSegments {
var prefix = s"${RecordIndent} "
if (!skipRecordMetadata) {
- print(s"${prefix}offset: ${record.offset} isValid: ${record.isValid} crc: ${record.checksumOrNull}" +
+ val checksum = record match {
+ case r: AbstractLegacyRecordBatch => r.checksum().toString
+ case _ => "null"
+ }
Review comment:
I think we should skip the checksum string altogether at record level for message format v2 and later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a change in pull request #10498: MINOR: remove `checksumOrNull`,`isValid`, and `ensureValid` from Record
Posted by GitBox <gi...@apache.org>.
chia7712 commented on a change in pull request #10498:
URL: https://github.com/apache/kafka/pull/10498#discussion_r613782042
##########
File path: clients/src/main/java/org/apache/kafka/common/record/Record.java
##########
@@ -132,4 +121,16 @@
* @return the array of headers
*/
Header[] headers();
+
+ static void ensureValid(Record record) {
+ if (record instanceof AbstractLegacyRecordBatch) {
+ ((AbstractLegacyRecordBatch) record).ensureValid();
+ }
+ }
+
+ static boolean isValid(Record record) {
+ if (record instanceof AbstractLegacyRecordBatch) {
+ return ((AbstractLegacyRecordBatch) record).isValid();
+ } else return true;
Review comment:
thanks for nice suggestions. will copy that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a change in pull request #10498: MINOR: remove `checksumOrNull`,`isValid`, and `ensureValid` from Record
Posted by GitBox <gi...@apache.org>.
chia7712 commented on a change in pull request #10498:
URL: https://github.com/apache/kafka/pull/10498#discussion_r613761674
##########
File path: core/src/main/scala/kafka/tools/DumpLogSegments.scala
##########
@@ -270,7 +270,11 @@ object DumpLogSegments {
var prefix = s"${RecordIndent} "
if (!skipRecordMetadata) {
- print(s"${prefix}offset: ${record.offset} isValid: ${record.isValid} crc: ${record.checksumOrNull}" +
+ val checksum = record match {
+ case r: AbstractLegacyRecordBatch => r.checksum().toString
+ case _ => "null"
+ }
Review comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a change in pull request #10498: MINOR: remove `checksumOrNull`,`isValid`, and `ensureValid` from Record
Posted by GitBox <gi...@apache.org>.
ijuma commented on a change in pull request #10498:
URL: https://github.com/apache/kafka/pull/10498#discussion_r613777759
##########
File path: clients/src/main/java/org/apache/kafka/common/record/Record.java
##########
@@ -132,4 +121,16 @@
* @return the array of headers
*/
Header[] headers();
+
+ static void ensureValid(Record record) {
+ if (record instanceof AbstractLegacyRecordBatch) {
+ ((AbstractLegacyRecordBatch) record).ensureValid();
+ }
+ }
+
+ static boolean isValid(Record record) {
+ if (record instanceof AbstractLegacyRecordBatch) {
+ return ((AbstractLegacyRecordBatch) record).isValid();
+ } else return true;
Review comment:
Polymorphism is a fine way to deal with that though. It's more confusing if we have a combination of polymorphism and pattern matching (well, the if/else version of it which doesn't include exhaustiveness checking).
I don't see an issue with having an implementation that always return true or is a no-op for v2. A default may be ok, but it introduces more coupling between the interface and implementations. It seems that `isValid` is only used by `DumpLogSegments` (outside of tests), so we can maybe remove that altogether. I would probably leave `ensureValid` as it was before this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a change in pull request #10498: MINOR: remove checksumOrNull from Record
Posted by GitBox <gi...@apache.org>.
ijuma commented on a change in pull request #10498:
URL: https://github.com/apache/kafka/pull/10498#discussion_r613743249
##########
File path: core/src/main/scala/kafka/tools/DumpLogSegments.scala
##########
@@ -280,7 +284,7 @@ object DumpLogSegments {
if (batch.magic >= RecordBatch.MAGIC_VALUE_V2) {
print(" sequence: " + record.sequence + " headerKeys: " + record.headers.map(_.key).mkString("[", ",", "]"))
} else {
- print(s" crc: ${record.checksumOrNull} isvalid: ${record.isValid}")
+ print(s" crc: $checksum isvalid: ${record.isValid}")
Review comment:
Are we printing the checksum twice or am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org