You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/04/30 03:15:45 UTC
[GitHub] [kafka] wenbingshen opened a new pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
wenbingshen opened a new pull request #10617:
URL: https://github.com/apache/kafka/pull/10617
The detailed stack information is linked to this jiraId [KAFKA-12734](https://issues.apache.org/jira/browse/KAFKA-12734)
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen commented on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #10617:
URL: https://github.com/apache/kafka/pull/10617#issuecomment-830757283
@junrao I did some thinking again today. The timeindex problems of [KAFKA-10471](https://issues.apache.org/jira/browse/KAFKA-10471) and [KAFKA-12734](https://issues.apache.org/jira/browse/KAFKA-12734) all come from `reset the index size of the currently active log segment to allow more entries`. Compared with directly deleting the cleanShutDownFiles file, does the impact of only detecting the index file of the active segment have less impact?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen commented on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #10617:
URL: https://github.com/apache/kafka/pull/10617#issuecomment-829766268
@junrao @ijuma @chia7712 Can you take a look at this PR? Thanks. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
junrao commented on pull request #10617:
URL: https://github.com/apache/kafka/pull/10617#issuecomment-831371437
@wenbingshen : It would be useful to be clear what the PR wants to address. To me, if the index is corrupted because of our recovery logic (e.g., not handling cleaning shutdown file properly), we should just fix the cause. It's possible that after a clean restart, the index is corrupted due to storage issues. In that case, validating the index when the index is opened could potentially be useful. If we want to do that, it seems that we should we just do the validation on every index open, instead of just for the active segment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen commented on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #10617:
URL: https://github.com/apache/kafka/pull/10617#issuecomment-830514376
> @wenbingshen: Thanks for the PR. We recently fixed https://issues.apache.org/jira/browse/KAFKA-10471. Do you think this is the same problem or a different one?
@junrao Wow, great, I think they are the same problem. As you said, implementing sanityCheck is a solution, but the state consistency of the clean shutdown file is the root cause of the problem. I will apply KAFKA-10471 The patch to our company's kafka. Thank you very much for your help.
@ijuma @chia7712 @hachikuji @alexandrfox @dajac @showuon Sorry to disturb you, this problem has been fixed in 2.8, thank you very much! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen removed a comment on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen removed a comment on pull request #10617:
URL: https://github.com/apache/kafka/pull/10617#issuecomment-831393455
> @wenbingshen : It would be useful to be clear what the PR wants to address. To me, if the index is corrupted because of our recovery logic (e.g., not handling cleaning shutdown file properly), we should just fix the cause. It's possible that after a clean restart, the index is corrupted due to storage issues. In that case, validating the index when the index is opened could potentially be useful. If we want to do that, it seems that we should we just do the validation on every index open, instead of just for the active segment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
junrao commented on pull request #10617:
URL: https://github.com/apache/kafka/pull/10617#issuecomment-830451957
@wenbingshen: Thanks for the PR. We recently fixed https://issues.apache.org/jira/browse/KAFKA-10471. Do you think this is the same problem or a different one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen closed pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen closed pull request #10617:
URL: https://github.com/apache/kafka/pull/10617
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen commented on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #10617:
URL: https://github.com/apache/kafka/pull/10617#issuecomment-830153183
@junrao is busy with the work and affairs of the leader. @ijuma @chia7712 @hachikuji @alexandrfox @dajac @showuon Can you help me see this, or can you give some thoughts and suggestions? I have occasionally encountered this kind of problem in the production environment recently. I don't know what method can solve this problem. But I think it might be right to perform sanityCheck on the active segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen closed pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen closed pull request #10617:
URL: https://github.com/apache/kafka/pull/10617
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen commented on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #10617:
URL: https://github.com/apache/kafka/pull/10617#issuecomment-830145333
This is a problem we often encounter before applying LazyIndex. SanityCheck will help us rebuild the index, but after applying LazyIndex, it skips checking all index files, resulting in abnormal index files in the active segment. At this time, appending data to the log will cause niobufferoverflow exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen commented on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #10617:
URL: https://github.com/apache/kafka/pull/10617#issuecomment-831399088
> @wenbingshen : It would be useful to be clear what the PR wants to address. To me, if the index is corrupted because of our recovery logic (e.g., not handling cleaning shutdown file properly), we should just fix the cause. It's possible that after a clean restart, the index is corrupted due to storage issues. In that case, validating the index when the index is opened could potentially be useful. If we want to do that, it seems that we should we just do the validation on every index open, instead of just for the active segment?
@junrao Thank you for your reply. You've got a point there. This PR problem is really due to not handling cleaning shutdown file properly. It's right to fix the root cause.
After a successful restart, the index file is damaged due to storage reasons. If we need to solve this problem, we need other PR to do it. For opening the index file, we may need to check the integrity just like checking the entries in the checkpoint file.
I will close this PR because the problem has been fixed. For the latter situation mentioned by you, I will think about it. If I have any good ideas, I will give you feedback again. Thank you. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen commented on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #10617:
URL: https://github.com/apache/kafka/pull/10617#issuecomment-831393455
> @wenbingshen : It would be useful to be clear what the PR wants to address. To me, if the index is corrupted because of our recovery logic (e.g., not handling cleaning shutdown file properly), we should just fix the cause. It's possible that after a clean restart, the index is corrupted due to storage issues. In that case, validating the index when the index is opened could potentially be useful. If we want to do that, it seems that we should we just do the validation on every index open, instead of just for the active segment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wenbingshen closed pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck
Posted by GitBox <gi...@apache.org>.
wenbingshen closed pull request #10617:
URL: https://github.com/apache/kafka/pull/10617
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org