[jira] [Commented] (KAFKA-1510) Force offset commits when migrating consumer offsets from zookeeper to kafka

["Joel Koshy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081781#comment-14081781 ] 

Joel Koshy commented on KAFKA-1510:
-----------------------------------

Re: isAutoCommit:
Yes you are right. Can you fix that?

Re: migration steps:
Yes that is correct. "Wait for kafka to be copied on commits" would essentially mean do one rolling bounce after your patch (since the shutdown guarantees that the offsets would have moved over).

Re: applies only when storage==kafka and dual-commit==true.
Yes that is correct.

Re:  why the write to zookeeper the new offsets, only if the write to kafka was ok?
As you observed, avoid redundant writes to ZooKeeper, but also for consistency between ZK and Kafka (although this does not matter too much since while transitioning they are inconsistent to start with).

Re: separate DAOs
We probably don't need that since we plan to eventually remove support for ZooKeeper-based offsets from the server-side - i.e., a consumer can choose to store offsets elsewhere (including ZooKeeper) but support for doing that via the OffsetCommitRequest channel will be removed.

Your patch looks good, but I'm unclear on why you need the "|| forceCommit" on line 318

> Force offset commits when migrating consumer offsets from zookeeper to kafka
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-1510
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1510
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8.2
>            Reporter: Joel Koshy
>            Assignee: Joel Koshy
>              Labels: newbie
>             Fix For: 0.8.2
>
>         Attachments: forceCommitOnShutdownWhenDualCommit.patch
>
>
> When migrating consumer offsets from ZooKeeper to kafka, we have to turn on dual-commit (i.e., the consumers will commit offsets to both zookeeper and kafka) in addition to setting offsets.storage to kafka. However, when we commit offsets we only commit offsets if they have changed (since the last commit). For low-volume topics or for topics that receive data in bursts offsets may not move for a long period of time. Therefore we may want to force the commit (even if offsets have not changed) when migrating (i.e., when dual-commit is enabled) - we can add a minimum interval threshold (say force commit after every 10 auto-commits) as well as on rebalance and shutdown.
> Also, I think it is safe to switch the default for offsets.storage from zookeeper to kafka and set the default to dual-commit (for people who have not migrated yet). We have deployed this to the largest consumers at linkedin and have not seen any issues so far (except for the migration caveat that this jira will resolve).



--
This message was sent by Atlassian JIRA
(v6.2#6252)