[jira] [Commented] (KAFKA-17946) Flaky test DeafultStateUpdaterTest::shouldResumeStandbyTask due to concurrency issue

["Eswarar Siva (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-17946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18090587#comment-18090587 ] 

Eswarar Siva commented on KAFKA-17946:
--------------------------------------

[~lucasbru] I looked at the fix you linked (PR #18253). It stabilizes the test (it waits for the state updater to run the
verified calls, or shuts the updater down before verifying) and changes one log line in DefaultStateUpdater; it does
not touch resumeTask or transitToUpdateStandby. So I think the production path behind this is still open.

resumeTask calls changelogReader.transitToUpdateStandby() with no guard on the reader state (only
if (updatingTasks.size() == 1)), and transitToUpdateStandby() still throws IllegalStateException when the reader is
not in ACTIVE_RESTORING. Under pause/resume churn I can drive the reader into STANDBY_UPDATING and then have
resumeTask call transitToUpdateStandby again, which trips that check and kills the StreamThread.

I reproduced it on 4.1.2 and 4.3.0 with a small standalone Streams app (same reproducer as KAFKA-NNNNN, cold start
churn plus pause/resume, with a Byteman rule that widens the existing transition window so it is deterministic). The
error:

java.lang.IllegalStateException: The changelog reader is not restoring active tasks (is STANDBY_UPDATING) while
trying to transit to update standby tasks: {...}
    at StoreChangelogReader.transitToUpdateStandby(StoreChangelogReader.java:326)
    at DefaultStateUpdater$StateUpdaterThread.resumeTask(DefaultStateUpdater.java:661)
    at DefaultStateUpdater$StateUpdaterThread.resumeTasks(...)
    at DefaultStateUpdater$StateUpdaterThread.runOnce(...)

The resumeTask and transitToUpdateStandby code is unchanged from when this was resolved, so the test was stabilized
but the production transition is still not idempotent. Do you want me to reopen this, or file a separate production
facing ticket? I have a Byteman based reproduction I can attach.

> Flaky test DeafultStateUpdaterTest::shouldResumeStandbyTask due to concurrency issue
> ------------------------------------------------------------------------------------
>
>                 Key: KAFKA-17946
>                 URL: https://issues.apache.org/jira/browse/KAFKA-17946
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: Ao Li
>            Priority: Major
>
> Function `transitToUpdateStandby` may be called after verify statement is executed. 
> See branch: https://github.com/aoli-al/kafka/tree/KAFKA-111 to reproduce the failure. 
> Test command: ./gradlew cleanTest :streams:test --tests org.apache.kafka.streams.processor.internals.DefaultStateUpdaterTest.shouldResumeStandbyTask



--
This message was sent by Atlassian Jira
(v8.20.10#820010)