[jira] [Comment Edited] (KAFKA-6373) Log end offset of input table changing during restore

["Charles Crain (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16294913#comment-16294913 ] 

Charles Crain edited comment on KAFKA-6373 at 12/18/17 12:40 PM:
-----------------------------------------------------------------

As an update, going back to the 0.11.0.2 client libraries (while changing nothing on the broker side) appears to fix the issue.  Note that it's not apples to apples: I was also using the new 1.0 APIs in the above example so some minor client code change was involved in going back to 0.11.0.2.  I have not tested using the deprecated 0.11 APIs with the 1.0 libraries; I assumed the results would be similar since the plumbing appears the same.


was (Author: ccrain_kuka):
As an update, going back to the 0.11.0.2 client libraries appears to fix the issue.  Note that it's not apples to apples: I was also using the new 1.0 APIs in the above example so some minor client code change was involved in going back to 0.11.0.2.  I have not tested using the deprecated 0.11 APIs with the 1.0 libraries; I assumed the results would be similar since the plumbing appears the same.

> Log end offset of input table changing during restore
> -----------------------------------------------------
>
>                 Key: KAFKA-6373
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6373
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>    Affects Versions: 1.0.0
>         Environment: Client 1.0.0, Brokers 1.0.0 with 1.0.0 message format and inter-broker protocol
>            Reporter: Charles Crain
>
> I am receiving a confusing error from a Kafka Streams application.  Most of the time when I try to bring up just a single replica of the task for the first time, I get this:
> {noformat}
> Detected a task that got migrated to another thread. This implies that this thread missed a rebalance and dropped out of the consumer group. Trying to rejoin the consumer group now.
> org.apache.kafka.streams.errors.TaskMigratedException: Log end offset of [Name of Topic]-36 should not change while restoring: old end offset 37559, current offset 37561
> {noformat}
> The confusing thing is that [Name of Topic] is *not* a change log topic created by the stream app.  Rather it is a topic published from a completely different service.  And since that other service is publishing to that topic actively, of course the end offset is constantly changing.
> Here is a rough view of my stream topology.  I'll call the topic that's showing up in the above error "ExternalTableTopic".
> {noformat}
> externalTable = table(ExternalTableTopic)
> stream(ExternalStreamTopic)
>   .leftJoin(externalTable, joiner)
>   .aggregate(aggregator, SomeQueryableStoreName)
> {noformat}
> ...and that's it.  If I take out the left join this appears not to happen.  Is it illegal to join a table to a stream if that table is being published from somewhere else?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)