You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "kamalcph (via GitHub)" <gi...@apache.org> on 2023/12/03 16:15:56 UTC

[PR] MINOR: Allow local-log segment deletion when log-start-offset incremented [kafka]

kamalcph opened a new pull request, #14905:
URL: https://github.com/apache/kafka/pull/14905

   `DELETE_RECORDS` API can move the log-start-offset beyond the highest-copied-remote-offset. In such cases, we should allow deletion of local-log segments since they are not eligible for upload to remote storage.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Allow local-log segment deletion when log-start-offset incremented [kafka]

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on PR #14905:
URL: https://github.com/apache/kafka/pull/14905#issuecomment-1842245405

   
   > Does that mean currently, when the log start offset incremented over the candidate segments, we won't delete until the segments are uploaded to the remote storage? 
   
   The local-log segments won't be eligible for deletion and they are not eligible for upload to remote storage as well since the log-start-offset might already be moved. Once the next-set of segments gets uploaded, then the `highest-copied-remote-offset` will be higher than not-uploaded local-log segments, then they became eligible for deletion. This patch is raised to address edge cases where the topic becomes stale/deprecated post the log-start-offset update.
   
   > If so, does that mean those uploaded segments will be deleted next time when when enter `cleanupExpiredRemoteLogSegments` because of `isSegmentBreachByLogStartOffset`?
   
   Yes, correct. The uploaded remote segments will be cleaned up by `cleanupExpiredRemoteLogSegments` due to `isSegmentBreachByLogStartOffset`.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Allow local-log segment deletion when log-start-offset incremented [kafka]

Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #14905:
URL: https://github.com/apache/kafka/pull/14905#issuecomment-1842685829

   Jenkins test failures are unrelated to this PR, merging it to trunk.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Allow local-log segment deletion when log-start-offset incremented [kafka]

Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd merged PR #14905:
URL: https://github.com/apache/kafka/pull/14905


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Allow local-log segment deletion when log-start-offset incremented [kafka]

Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14905:
URL: https://github.com/apache/kafka/pull/14905#discussion_r1417134578


##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1443,11 +1443,14 @@ class UnifiedLog(@volatile var logStartOffset: Long,
    * @return the segments ready to be deleted
    */
   private[log] def deletableSegments(predicate: (LogSegment, Option[LogSegment]) => Boolean): Iterable[LogSegment] = {
-    def isSegmentEligibleForDeletion(upperBoundOffset: Long): Boolean = {
+    def isSegmentEligibleForDeletion(nextSegmentOpt: Option[LogSegment], upperBoundOffset: Long): Boolean = {
+      val allowDeletionDueToLogStartOffsetIncremented = nextSegmentOpt.isDefined && logStartOffset >= nextSegmentOpt.get.baseOffset
       // Segments are eligible for deletion when:
       //    1. they are uploaded to the remote storage
+      //    2. log-start-offset was incremented higher than the largest offset in the candidate segment
       if (remoteLogEnabled()) {
-        upperBoundOffset > 0 && upperBoundOffset - 1 <= highestOffsetInRemoteStorage
+        (upperBoundOffset > 0 && upperBoundOffset - 1 <= highestOffsetInRemoteStorage) ||
+          allowDeletionDueToLogStartOffsetIncremented

Review Comment:
   Note: This check can not be added as part of `deleteLogStartOffsetBreachedSegments()` as the current  `upperBoundOffset > 0 && upperBoundOffset - 1 <= highestOffsetInRemoteStorage` becomes false and does not allow the segment to be deleted. 
   Added a note here that may help give clarity to the readers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org