You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Xu Jianhai <sn...@gmail.com> on 2019/11/01 16:41:46 UTC

question about offsetSync

Hi:
    I am engineer from China, I review kafka mirror latest impl, but I do
not figure out the reason why PartitionState update like that:
```

    // true if we should emit an offset sync
    boolean update(long upstreamOffset, long downstreamOffset) {
        boolean shouldSyncOffsets = false;
        long upstreamStep = upstreamOffset - lastSyncUpstreamOffset;
        long downstreamTargetOffset = lastSyncDownstreamOffset + upstreamStep;
        if (lastSyncDownstreamOffset == -1L
                || downstreamOffset - downstreamTargetOffset >= maxOffsetLag
                || upstreamOffset - previousUpstreamOffset != 1L
                || downstreamOffset < previousDownstreamOffset) {
            lastSyncUpstreamOffset = upstreamOffset;
            lastSyncDownstreamOffset = downstreamOffset;
            shouldSyncOffsets = true;
        }
        previousUpstreamOffset = upstreamOffset;
        previousDownstreamOffset = downstreamOffset;
        return shouldSyncOffsets;
    }
}

```
I can not know why the condition is like that.
1. lastSyncDownstreamOffset == -1L: never sync, so call sync method
2. downstreamOffset - downstreamTargetOffset >= maxOffsetLag: offset is not
accurate, so sync. but why use maxOffsetLag? why not >0: meaning not
accurate
3. upstreamOffset - previousUpstreamOffset != 1L: meaning why?
4. downstreamOffset < previousDownstreamOffset： meaning why?

Re: question about offsetSync

Posted by Xu Jianhai <sn...@gmail.com>.
From my opinion,
condition 4 is downstream data loss because of master down without sync in
time, so downstream data producer get smaller downstreamOffset.
condition 3 meaning upstream broker down without sync in time
condition 1 is init state
so condition 2 may write like this: `downstreamTargetOffset -
lastSyncDownstreamOffset >= maxOffsetLag` meaning not sync offset for long
time, should we rewrite ?

On Sat, Nov 2, 2019 at 12:41 AM Xu Jianhai <sn...@gmail.com> wrote:

> Hi:
>     I am engineer from China, I review kafka mirror latest impl, but I do
> not figure out the reason why PartitionState update like that:
> ```
>
>     // true if we should emit an offset sync
>     boolean update(long upstreamOffset, long downstreamOffset) {
>         boolean shouldSyncOffsets = false;
>         long upstreamStep = upstreamOffset - lastSyncUpstreamOffset;
>         long downstreamTargetOffset = lastSyncDownstreamOffset + upstreamStep;
>         if (lastSyncDownstreamOffset == -1L
>                 || downstreamOffset - downstreamTargetOffset >= maxOffsetLag
>                 || upstreamOffset - previousUpstreamOffset != 1L
>                 || downstreamOffset < previousDownstreamOffset) {
>             lastSyncUpstreamOffset = upstreamOffset;
>             lastSyncDownstreamOffset = downstreamOffset;
>             shouldSyncOffsets = true;
>         }
>         previousUpstreamOffset = upstreamOffset;
>         previousDownstreamOffset = downstreamOffset;
>         return shouldSyncOffsets;
>     }
> }
>
> ```
> I can not know why the condition is like that.
> 1. lastSyncDownstreamOffset == -1L: never sync, so call sync method
> 2. downstreamOffset - downstreamTargetOffset >= maxOffsetLag: offset is
> not accurate, so sync. but why use maxOffsetLag? why not >0: meaning not
> accurate
> 3. upstreamOffset - previousUpstreamOffset != 1L: meaning why?
> 4. downstreamOffset < previousDownstreamOffset： meaning why?
>
>
>
>
>
>
>