You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "jsancio (via GitHub)" <gi...@apache.org> on 2023/06/09 14:00:31 UTC

[GitHub] [kafka] jsancio opened a new pull request, #13834: KAFKA-15076; KRaft should prefer latest snapshot

jsancio opened a new pull request, #13834:
URL: https://github.com/apache/kafka/pull/13834

   If the KRaft listener is at offset 0, the start of the log, and KRaft has generated a snapshot, it should prefer the latest snapshot instead of having the listener read from the start of the log.
   
   This is implemented by having KafkaRaftClient send a Listener.handleLoadSnapshot event, if the Listener is at offset 0 and the KRaft partition has generated a snapshot.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mumrah commented on a diff in pull request #13834: KAFKA-15076; KRaft should prefer latest snapshot

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #13834:
URL: https://github.com/apache/kafka/pull/13834#discussion_r1224652571


##########
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##########
@@ -308,7 +308,12 @@ private void onUpdateLeaderHighWatermark(
     private void updateListenersProgress(long highWatermark) {
         for (ListenerContext listenerContext : listenerContexts.values()) {
             listenerContext.nextExpectedOffset().ifPresent(nextExpectedOffset -> {
-                if (nextExpectedOffset < log.startOffset() && nextExpectedOffset < highWatermark) {
+                // Send snapshot if the listener is at the beging of the log and there is a snapshot or
+                // the listener is trying to read an offset for which there isn't a segment in the log.
+                if (nextExpectedOffset < highWatermark &&
+                    ((nextExpectedOffset == 0 && latestSnapshot().isPresent()) ||

Review Comment:
   Previously, raft was only sending the listener a snapshot it was reading an offset that was not present in a log segment. The issue we saw was that on startup the listener's next read offset was 0 and if the segment containing 0 was present, then this branch would be skipped and we would not load a snapshot at all.
   
   In other words, if we had a cluster that still had segment `00000000000000000000.log` (either a new cluster, or one with high retention), we would end up replaying the whole log instead of loading from a snapshot.
   
   I think I understand what you're saying about a heuristic. Like, if the listener was behind by some amount and a later snapshot existed that could catch it up -- we could load that snapshot instead of replaying the log. Is that what you're suggesting?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] jsancio commented on a diff in pull request #13834: KAFKA-15076; KRaft should prefer latest snapshot

Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on code in PR #13834:
URL: https://github.com/apache/kafka/pull/13834#discussion_r1224779841


##########
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##########
@@ -308,7 +308,12 @@ private void onUpdateLeaderHighWatermark(
     private void updateListenersProgress(long highWatermark) {
         for (ListenerContext listenerContext : listenerContexts.values()) {
             listenerContext.nextExpectedOffset().ifPresent(nextExpectedOffset -> {
-                if (nextExpectedOffset < log.startOffset() && nextExpectedOffset < highWatermark) {
+                // Send snapshot if the listener is at the beging of the log and there is a snapshot or
+                // the listener is trying to read an offset for which there isn't a segment in the log.
+                if (nextExpectedOffset < highWatermark &&
+                    ((nextExpectedOffset == 0 && latestSnapshot().isPresent()) ||

Review Comment:
   Thanks for the comments @mumrah @hachikuji .
   
   Yes. This PR is only addresses the contract between the `RaftClient.Listener` and `RaftClient`. Before this PR the `KafkaRaftClient` was never calling `Listener.handleLoadSnapshot` if the log segments had offset 0.
   
   Regarding replication between the leader and the followers (voters or observer), I agree that we can have the leader force the follower to send a FETCH_SNAPSHOT request even if the log segments contain the fetched offset. This heuristic could be something like -- if the number of bytes between the fetch offset and the offset of the latest snapshot is greater than the size of the latest snapshot then the leader will response to the FETCH request asking the follower to FETCH_SNAPSHOT instead.
   
   I thought about creating an issue for this but I am not sure how likely this would be. I think the most common case would be the user expanding the cluster by adding a new broker. In that case we can have the leader ask the follower to do a FETCH_SNAPSHOT if the fetch offset is 0 and the leader has generated a snapshot.
   
   Another case would be that the broker or inactive controller was offline for a long time. There the optimization above wouldn't work and we have to have the heuristic we have been discussing. I am just not sure how common this would be for the cluster metadata log partition. We could use the offset index for a rough computation of the number of bytes between the two offsets but I am not sure if it is worth the code complexity.
   
   I created this issue for the case when the follower is fetching offset 0: https://issues.apache.org/jira/browse/KAFKA-15078



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] hachikuji commented on a diff in pull request #13834: KAFKA-15076; KRaft should prefer latest snapshot

Posted by "hachikuji (via GitHub)" <gi...@apache.org>.
hachikuji commented on code in PR #13834:
URL: https://github.com/apache/kafka/pull/13834#discussion_r1224616596


##########
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##########
@@ -308,7 +308,12 @@ private void onUpdateLeaderHighWatermark(
     private void updateListenersProgress(long highWatermark) {
         for (ListenerContext listenerContext : listenerContexts.values()) {
             listenerContext.nextExpectedOffset().ifPresent(nextExpectedOffset -> {
-                if (nextExpectedOffset < log.startOffset() && nextExpectedOffset < highWatermark) {
+                // Send snapshot if the listener is at the beging of the log and there is a snapshot or

Review Comment:
   nit: beginning



##########
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##########
@@ -308,7 +308,12 @@ private void onUpdateLeaderHighWatermark(
     private void updateListenersProgress(long highWatermark) {
         for (ListenerContext listenerContext : listenerContexts.values()) {
             listenerContext.nextExpectedOffset().ifPresent(nextExpectedOffset -> {
-                if (nextExpectedOffset < log.startOffset() && nextExpectedOffset < highWatermark) {
+                // Send snapshot if the listener is at the beging of the log and there is a snapshot or
+                // the listener is trying to read an offset for which there isn't a segment in the log.
+                if (nextExpectedOffset < highWatermark &&
+                    ((nextExpectedOffset == 0 && latestSnapshot().isPresent()) ||

Review Comment:
   Not sure how useful the optimization is if it only covers the case at the beginning of the log. We could potentially send the snapshot any time its offset is higher than `nextExpectedOffset`? Perhaps there is a smarter heuristic to avoid fetching snapshots unnecessarily. Fetching the log is not super efficient I guess because of the healthcheck records.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] jsancio merged pull request #13834: KAFKA-15076; KRaft should prefer latest snapshot

Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio merged PR #13834:
URL: https://github.com/apache/kafka/pull/13834


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] hachikuji commented on a diff in pull request #13834: KAFKA-15076; KRaft should prefer latest snapshot

Posted by "hachikuji (via GitHub)" <gi...@apache.org>.
hachikuji commented on code in PR #13834:
URL: https://github.com/apache/kafka/pull/13834#discussion_r1224733306


##########
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##########
@@ -308,7 +308,12 @@ private void onUpdateLeaderHighWatermark(
     private void updateListenersProgress(long highWatermark) {
         for (ListenerContext listenerContext : listenerContexts.values()) {
             listenerContext.nextExpectedOffset().ifPresent(nextExpectedOffset -> {
-                if (nextExpectedOffset < log.startOffset() && nextExpectedOffset < highWatermark) {
+                // Send snapshot if the listener is at the beging of the log and there is a snapshot or
+                // the listener is trying to read an offset for which there isn't a segment in the log.
+                if (nextExpectedOffset < highWatermark &&
+                    ((nextExpectedOffset == 0 && latestSnapshot().isPresent()) ||

Review Comment:
   Actually I had misunderstood the PR. I was not thinking about the listener at all, but a replica which was trying to catch up. For the listener, the offset always starts at 0, so the patch makes sense. This logic ensures that we always start with the latest snapshot. I'm not sure if listener catch-up is a major case to worry about. At least I think it's fair to say it is not in scope here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] jsancio commented on a diff in pull request #13834: KAFKA-15076; KRaft should prefer latest snapshot

Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on code in PR #13834:
URL: https://github.com/apache/kafka/pull/13834#discussion_r1224787831


##########
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##########
@@ -308,7 +308,12 @@ private void onUpdateLeaderHighWatermark(
     private void updateListenersProgress(long highWatermark) {
         for (ListenerContext listenerContext : listenerContexts.values()) {
             listenerContext.nextExpectedOffset().ifPresent(nextExpectedOffset -> {
-                if (nextExpectedOffset < log.startOffset() && nextExpectedOffset < highWatermark) {
+                // Send snapshot if the listener is at the beging of the log and there is a snapshot or

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org