You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/04/14 04:09:35 UTC
[GitHub] [kafka] tang7526 opened a new pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
tang7526 opened a new pull request #10534:
URL: https://github.com/apache/kafka/pull/10534
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 commented on pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 commented on pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#issuecomment-849014645
> @tang7526: Thanks for the PR. A few comments below. Also, could we add a unit test for this?
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 commented on a change in pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 commented on a change in pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#discussion_r637506676
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
+ ensureOffsetInRange(batch.lastOffset)
+
+ if (batch.maxTimestamp > maxTimestampSoFarTmp) {
Review comment:
You mean remove line 178?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 commented on a change in pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 commented on a change in pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#discussion_r640016337
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
+ ensureOffsetInRange(batch.lastOffset)
+
+ if (batch.maxTimestamp > maxTimestampSoFarTmp) {
+ maxTimestampSoFarTmp = batch.maxTimestamp
+ offsetOfMaxTimestampSoFarTmp = batch.lastOffset
+ }
+
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ if (batch.lastOffset > originalLastOffset) {
+ offsetIndex.append(batch.lastOffset, validBytes)
+ timeIndex.maybeAppend(maxTimestampSoFarTmp, offsetOfMaxTimestampSoFarTmp)
+ }
+ lastIndexEntry = validBytes
Review comment:
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 commented on a change in pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 commented on a change in pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#discussion_r640027627
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
+ ensureOffsetInRange(batch.lastOffset)
+
+ if (batch.maxTimestamp > maxTimestampSoFarTmp) {
+ maxTimestampSoFarTmp = batch.maxTimestamp
+ offsetOfMaxTimestampSoFarTmp = batch.lastOffset
+ }
+
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
Review comment:
hmmm.... I don't know how to use `bytesSinceLastIndexEntry` in here.
I have added a `val accumulatedBytes` and reset it to 0 after each index insertion.
Is it OK?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 commented on a change in pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 commented on a change in pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#discussion_r640016337
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
+ ensureOffsetInRange(batch.lastOffset)
+
+ if (batch.maxTimestamp > maxTimestampSoFarTmp) {
+ maxTimestampSoFarTmp = batch.maxTimestamp
+ offsetOfMaxTimestampSoFarTmp = batch.lastOffset
+ }
+
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ if (batch.lastOffset > originalLastOffset) {
+ offsetIndex.append(batch.lastOffset, validBytes)
+ timeIndex.maybeAppend(maxTimestampSoFarTmp, offsetOfMaxTimestampSoFarTmp)
+ }
+ lastIndexEntry = validBytes
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 removed a comment on pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 removed a comment on pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#issuecomment-825051965
@chia7712 Could you help to review this PR? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 commented on a change in pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 commented on a change in pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#discussion_r640027627
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
+ ensureOffsetInRange(batch.lastOffset)
+
+ if (batch.maxTimestamp > maxTimestampSoFarTmp) {
+ maxTimestampSoFarTmp = batch.maxTimestamp
+ offsetOfMaxTimestampSoFarTmp = batch.lastOffset
+ }
+
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
Review comment:
hmmm.... I don't know how to consider `bytesSinceLastIndexEntry` into here.
I have added a `accumulatedBytes` and reset it to 0 after each index insertion.
Is it OK?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 commented on a change in pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 commented on a change in pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#discussion_r640024597
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
+ ensureOffsetInRange(batch.lastOffset)
+
+ if (batch.maxTimestamp > maxTimestampSoFarTmp) {
Review comment:
We need to add previous missing indexes if a follower restarts after being down for a long time.
So I think we need to recheck batches to append missing `offsetIndex` and `timeIndex`.
Is it right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 commented on pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 commented on pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#issuecomment-825051965
@chia7712 Could you help to review this PR? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#discussion_r620704283
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
+ ensureOffsetInRange(batch.lastOffset)
+
+ if (batch.maxTimestamp > maxTimestampSoFarTmp) {
+ maxTimestampSoFarTmp = batch.maxTimestamp
+ offsetOfMaxTimestampSoFarTmp = batch.lastOffset
+ }
+
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ if (batch.lastOffset > originalLastOffset) {
+ offsetIndex.append(batch.lastOffset, validBytes)
+ timeIndex.maybeAppend(maxTimestampSoFarTmp, offsetOfMaxTimestampSoFarTmp)
+ }
+ lastIndexEntry = validBytes
Review comment:
Could we just have a single val accumulatedBytes and reset it to 0 after each index insertion?
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
+ ensureOffsetInRange(batch.lastOffset)
+
+ if (batch.maxTimestamp > maxTimestampSoFarTmp) {
+ maxTimestampSoFarTmp = batch.maxTimestamp
+ offsetOfMaxTimestampSoFarTmp = batch.lastOffset
+ }
+
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
Review comment:
It seems that we need to take bytesSinceLastIndexEntry into consideration and we also need to update bytesSinceLastIndexEntry properly here instead of the caller.
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
Review comment:
We already did validation in Log.analyzeAndValidateRecords(). So we don't need to do this again here.
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
+ ensureOffsetInRange(batch.lastOffset)
+
+ if (batch.maxTimestamp > maxTimestampSoFarTmp) {
Review comment:
Since this is done in the caller for all batches, we don't need to do it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 commented on a change in pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 commented on a change in pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#discussion_r637506753
##########
File path: core/src/main/scala/kafka/log/LogSegment.scala
##########
@@ -162,15 +162,43 @@ class LogSegment private[log] (val log: FileRecords,
// append the messages
val appendedBytes = log.append(records)
trace(s"Appended $appendedBytes to ${log.file} at end offset $largestOffset")
+
+
+ def appendIndex(): Unit = {
+ var validBytes = 0
+ var maxTimestampSoFarTmp = RecordBatch.NO_TIMESTAMP
+ var offsetOfMaxTimestampSoFarTmp = 0L
+ var lastIndexEntry = 0
+ val originalLastOffset = offsetIndex.lastOffset
+
+ for (batch <- log.batches.asScala) {
+ batch.ensureValid()
Review comment:
OK, I will remove line 175.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tang7526 commented on pull request #10534: KAFKA-806: Index may not always observe log.index.interval.bytes
Posted by GitBox <gi...@apache.org>.
tang7526 commented on pull request #10534:
URL: https://github.com/apache/kafka/pull/10534#issuecomment-819237139
@junrao Could you help to review this PR? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org