[jira] [Commented] (FLINK-40109) YARNSessionFIFOSecuredITCase.testDetachedMode is flaky: Kerberos keytab check runs before the YARN application is killed

["Martijn Visser (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-40109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116022#comment-18116022 ] 

Martijn Visser commented on FLINK-40109:
----------------------------------------

This still fails on master after b464b331dbd, in a new shape: the poll added by this ticket runs
to its timeout instead of asserting false immediately.

nightly 2026-09-13 on master, Java 21, module misc, both testDetachedMode and testDetachedModeSecureWithPreInstallKeytab: https://github.com/apache/flink/actions/runs/34732018337
nightly 2026-09-11 on master, Hadoop 3.4.3, module misc: https://github.com/apache/flink/actions/runs/34553043463

{code}
[ERROR] org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.testDetachedMode -- Time elapsed: 86.50 s <<< ERROR!
java.util.concurrent.TimeoutException: Kerberos keytab login [Login successful for user, using keytab file] not found in taskmanager.log for application application_1789094225420_0003 within the timeout; inspect the uploaded container logs.
        at org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:218)
        at org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.verifyResultContainsKerberosKeytab(YARNSessionFIFOSecuredITCase.java:204)
{code}

Two of the sixteen master nightlies in the eight days to 2026-09-16. The failing read is
taskmanager.log in one run and jobmanager.log in the other.

Separately, release-2.3 still has the pre-fix code and fails on the original assertion
(verifyResultContainsKerberosKeytab:201, "Expecting value to be true but was false"):
https://github.com/apache/flink/actions/runs/35046431137/job/104646790138
so this and FLINK-40099 need backporting to release-2.3 as well.

> YARNSessionFIFOSecuredITCase.testDetachedMode is flaky: Kerberos keytab check runs before the YARN application is killed
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-40109
>                 URL: https://issues.apache.org/jira/browse/FLINK-40109
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / YARN, Tests
>    Affects Versions: 2.4.0
>            Reporter: Martijn Visser
>            Assignee: Martijn Visser
>            Priority: Major
>              Labels: pull-request-available, test-stability
>             Fix For: 2.4.0
>
>
> {{YARNSessionFIFOSecuredITCase.testDetachedMode}} fails intermittently on master since FLINK-40099, e.g. GHA run 29005398820 (job 86077406039, module {{misc}}):
> {noformat}
> org.apache.flink.util.FlinkException: Exhausted retry attempts.
>     at org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:175)
>     at org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.verifyKerberosKeytabInLogs(YARNSessionFIFOSecuredITCase.java:209)
>     at org.apache.flink.yarn.YARNSessionFIFOITCase.runDetachedModeTest(YARNSessionFIFOITCase.java:201)
> {noformat}
> Root cause
> FLINK-40099 moved the Kerberos keytab log verification into a callback that runs after the job reaches FINISHED but before the application is killed, and polls the logs with {{waitUntilCondition(condition, 500)}}. Two problems:
> The {{500}} {{int}} literal binds to the {{waitUntilCondition(condition, int retryAttempts)}} overload (500 attempts × 100ms, then {{"Exhausted retry attempts."}}) rather than the intended millisecond interval.
> More fundamentally, reading the JobManager log while the application is still alive is racy. The container {{jobmanager.log}} that is in scope does not reliably contain the keytab login line before teardown: the bootstrap output (which carries the login line) and the running-JobManager output land in different log files, and the pre-kill content is not durably readable. The keytab line is present once the application has been torn down and its logs flushed.
> The original FLINK-17662 concern (a short-lived TaskManager's startup login line being briefly unreadable ~200ms after teardown) is real, but the fix for it should be a bounded post-kill poll, not relocating the check to before the kill.
> Proposed fix
> Revert FLINK-40099's pre-kill callback and read the container logs after the application is killed (output flushed), replacing the original single read with a bounded {{CommonTestUtils.waitUtil(...)}} poll that fails with a message naming the log file and the missing strings.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)