You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "showuon (via GitHub)" <gi...@apache.org> on 2023/12/08 06:56:33 UTC
[PR] KAFKA-15911: making sure the follower is making progress [kafka]
showuon opened a new pull request, #14971:
URL: https://github.com/apache/kafka/pull/14971
Just because the leader returned a successful response to FETCH and FETCH_SNAPSHOT doesn't mean that the followers were able to handle the response correctly.
For example, imagine the case where the log end offset (LEO) is at 1000 and all of the followers are continuously fetching at offset 0 without ever increasing their fetch offset. This can happen if the followers encounter an error when processing the FETCH or FETCH_SNAPSHOT response.
This PR adds `isFetchOffsetAdvanced` and `isFetchSnapshotPositionAdvanced` to fix this potential issue.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15911: making sure the follower is making progress [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14971:
URL: https://github.com/apache/kafka/pull/14971#issuecomment-2568654983
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15911: making sure the follower is making progress [kafka]
Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on code in PR #14971:
URL: https://github.com/apache/kafka/pull/14971#discussion_r1600190601
##########
raft/src/main/java/org/apache/kafka/raft/LeaderState.java:
##########
@@ -129,16 +135,46 @@ public void updateCheckQuorumForFollowingVoter(int id, long currentTimeMs) {
}
}
- private void updateFetchedVoters(int id) {
+ private void updateFetchedVoters(int id, long offset, long position) {
if (id == localId) {
throw new IllegalArgumentException("Received a FETCH/FETCH_SNAPSHOT request from the leader itself.");
}
- if (isVoter(id)) {
+ if (isVoter(id) && (isFetchOffsetAdvanced(id, offset) || isFetchSnapshotPositionAdvanced(id, position))) {
fetchedVoters.add(id);
}
}
+ /**
+ * We treat fetch offset is advanced when:
+ * 1. fetch offset is greater than or equal to leader end offset
+ * 2. if leader end offset is not set yet, we treat the follower is progressing
+ * 3. fetch offset is greater than previous fetch offset
+ *
+ * @param id the node id
+ * @param offset the fetch offset
+ */
+ private boolean isFetchOffsetAdvanced(int id, long offset) {
+ if (offset >= voterStates.get(localId).endOffset.orElse(new LogOffsetMetadata(-1)).offset ||
+ offset > fetchedOffset.get(id)) {
+ fetchedOffset.put(id, offset);
+ }
+ return false;
+ }
+
+ /**
+ * We treat fetch snapshot position is advanced when fetch snapshot position is greater than previous fetch snapshot position
+ *
+ * @param id the node id
+ * @param position the fetch snapshot position
+ */
+ private boolean isFetchSnapshotPositionAdvanced(int id, long position) {
+ if (position > fetchedPosition.get(id)) {
+ fetchedPosition.put(id, position);
+ }
+ return false;
Review Comment:
Thanks for the PR and excuse the extreme delay. I'll try to take a look this week!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15911: making sure the follower is making progress [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #14971:
URL: https://github.com/apache/kafka/pull/14971#discussion_r1420033396
##########
raft/src/main/java/org/apache/kafka/raft/LeaderState.java:
##########
@@ -129,16 +135,46 @@ public void updateCheckQuorumForFollowingVoter(int id, long currentTimeMs) {
}
}
- private void updateFetchedVoters(int id) {
+ private void updateFetchedVoters(int id, long offset, long position) {
if (id == localId) {
throw new IllegalArgumentException("Received a FETCH/FETCH_SNAPSHOT request from the leader itself.");
}
- if (isVoter(id)) {
+ if (isVoter(id) && (isFetchOffsetAdvanced(id, offset) || isFetchSnapshotPositionAdvanced(id, position))) {
fetchedVoters.add(id);
}
}
+ /**
+ * We treat fetch offset is advanced when:
+ * 1. fetch offset is greater than or equal to leader end offset
+ * 2. if leader end offset is not set yet, we treat the follower is progressing
+ * 3. fetch offset is greater than previous fetch offset
+ *
+ * @param id the node id
+ * @param offset the fetch offset
+ */
+ private boolean isFetchOffsetAdvanced(int id, long offset) {
+ if (offset >= voterStates.get(localId).endOffset.orElse(new LogOffsetMetadata(-1)).offset ||
+ offset > fetchedOffset.get(id)) {
+ fetchedOffset.put(id, offset);
+ }
+ return false;
+ }
+
+ /**
+ * We treat fetch snapshot position is advanced when fetch snapshot position is greater than previous fetch snapshot position
+ *
+ * @param id the node id
+ * @param position the fetch snapshot position
+ */
+ private boolean isFetchSnapshotPositionAdvanced(int id, long position) {
+ if (position > fetchedPosition.get(id)) {
+ fetchedPosition.put(id, position);
+ }
+ return false;
Review Comment:
@jsancio , this is the main change.
For `isFetchOffsetAdvanced`, We treat fetch offset is advanced when:
1. fetch offset is greater than or equal to leader end offset
2. if leader end offset is not set yet, we treat the follower is progressing
3. fetch offset is greater than previous fetch offset
Does this make sense? I'm thinking a corner case that if the follower lose its data (ex: disk failure), and joined again with empty disk, the fetch offset will start from 0. But from the leader's perspective, it doesn't know anything about it. It'll just treat it as not progressing follower.
Another thought is using `lastCaughtUpTimestamp` as a factor to make sure the leader HWM is successfully advancing. But again, if all the followers are slowly catching up with the leader from offset 0, we will resign this leader unexpectedly when fetch timeout expired.
Let's step back, if there is a follower that cannot advance its fetch offset in each fetch request, what it could help if we resign the current leadership? Electing another node as the leader and the follower still fail to catch up? Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15911: making sure the follower is making progress [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14971: KAFKA-15911: making sure the follower is making progress
URL: https://github.com/apache/kafka/pull/14971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15911: making sure the follower is making progress [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14971:
URL: https://github.com/apache/kafka/pull/14971#issuecomment-1984982679
This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch) <p> If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15911: making sure the follower is making progress [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14971:
URL: https://github.com/apache/kafka/pull/14971#issuecomment-2285277003
This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch) <p> If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org