You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/04/24 01:13:06 UTC
[GitHub] [kafka] steverod opened a new pull request #8543: [KAFKA-9826] Handle an unaligned first dirty offset during log cleani…
steverod opened a new pull request #8543:
URL: https://github.com/apache/kafka/pull/8543
…ng. (#8469)
In KAFKA-9826, a log whose first dirty offset was past the start of the active segment and past the last cleaned point resulted in an endless cycle of picking the segment to clean and discarding it. Though this didn't interfere with cleaning other log segments, it kept the log cleaner thread continuously busy (potentially wasting CPU and impacting other running threads) and filled the logs with lots of extraneous messages.
This was determined to be because the active segment was getting mistakenly picked for cleaning, and because the logSegments code handles (start == end) cases only for (start, end) on a segment boundary: the intent is to return a null list, but if they're not on a segment boundary, the routine returns that segment.
This fix has two parts:
It changes logSegments to handle start==end by returning an empty List always.
It changes the definition of calculateCleanableBytes to not consider anything past the UncleanableOffset; previously, it would potentially shift the UncleanableOffset to match the firstDirtyOffset even if the firstDirtyOffset was past the firstUncleanableOffset. This has no real effect now in the context of the fix for (1) but it makes the code read more like the model that the code is attempting to follow.
These changes require modifications to a few test cases that handled this particular test case; they were introduced in the context of KAFKA-8764. Those situations are now handled elsewhere in code, but the tests themselves allowed a DirtyOffset in the active segment, and expected an active segment to be selected for cleaning.
Reviewer: Jun Rao <ju...@gmail.com>
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on pull request #8543: [KAFKA-9826] Handle an unaligned first dirty offset during log cleani…
Posted by GitBox <gi...@apache.org>.
junrao commented on pull request #8543:
URL: https://github.com/apache/kafka/pull/8543#issuecomment-620879490
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] steverod commented on pull request #8543: [KAFKA-9826] Handle an unaligned first dirty offset during log cleani…
Posted by GitBox <gi...@apache.org>.
steverod commented on pull request #8543:
URL: https://github.com/apache/kafka/pull/8543#issuecomment-620710529
> @steverod : There seems to be compilation errors in JDK 8 test?
Hi @junrao -- Yes, there are, but they aren't mine (!!). No, really.
> @steverod : There seems to be compilation errors in JDK 8 test?
This was pre-existing in 2.4 and is being fixed by https://github.com/apache/kafka/pull/8562 (backport of some test fixes).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] steverod commented on pull request #8543: [KAFKA-9826] Handle an unaligned first dirty offset during log cleani…
Posted by GitBox <gi...@apache.org>.
steverod commented on pull request #8543:
URL: https://github.com/apache/kafka/pull/8543#issuecomment-621336009
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on pull request #8543: [KAFKA-9826] Handle an unaligned first dirty offset during log cleani…
Posted by GitBox <gi...@apache.org>.
junrao commented on pull request #8543:
URL: https://github.com/apache/kafka/pull/8543#issuecomment-621374929
The failed test `BranchedMultiLevelRepartitionConnectedTopologyTest` seems unrelated to the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] steverod commented on pull request #8543: [KAFKA-9826] Handle an unaligned first dirty offset during log cleani…
Posted by GitBox <gi...@apache.org>.
steverod commented on pull request #8543:
URL: https://github.com/apache/kafka/pull/8543#issuecomment-619214390
> is this duplicate to #8542?
Yes, this is aimed at 2.4 rather than 2.5. Took a best guess for propagation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on pull request #8543: [KAFKA-9826] Handle an unaligned first dirty offset during log cleani…
Posted by GitBox <gi...@apache.org>.
junrao commented on pull request #8543:
URL: https://github.com/apache/kafka/pull/8543#issuecomment-619128544
@chia7712 : It's the same patch, but for a different branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on pull request #8543: [KAFKA-9826] Handle an unaligned first dirty offset during log cleani…
Posted by GitBox <gi...@apache.org>.
chia7712 commented on pull request #8543:
URL: https://github.com/apache/kafka/pull/8543#issuecomment-619070458
is this duplicate to #8542?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on pull request #8543: [KAFKA-9826] Handle an unaligned first dirty offset during log cleani…
Posted by GitBox <gi...@apache.org>.
junrao commented on pull request #8543:
URL: https://github.com/apache/kafka/pull/8543#issuecomment-619129858
@steverod : There seems to be compilation errors in JDK 8 test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org