[jira] [Created] (FLINK-40099) YARNSessionFIFOSecuredITCase.testDetachedMode is flaky: Kerberos keytab log check reads a killed TaskManager's log

["Ramin Gharib (Jira)" <ji...@apache.org>]

Ramin Gharib created FLINK-40099:
------------------------------------

             Summary: YARNSessionFIFOSecuredITCase.testDetachedMode is flaky: Kerberos keytab log check reads a killed TaskManager's log 
                 Key: FLINK-40099
                 URL: https://issues.apache.org/jira/browse/FLINK-40099
             Project: Flink
          Issue Type: Bug
            Reporter: Ramin Gharib
            Assignee: Ramin Gharib


{{YARNSessionFIFOSecuredITCase.testDetachedMode}} (and {{{}testDetachedModeSecureWithPreInstallKeytab{}}}) is flaky. It fails at {{verifyResultContainsKerberosKeytab}} with:
{code:java}
assertThat(jobManagerRunsWithKerberos && taskManagerRunsWithKerberos).isTrue();  // was false                                                                                                    Expecting value to be true but was false{code}
h3. Root cause:

verifyResultContainsKerberosKeytab does a single-shot search of jobmanager.log and taskmanager.log for "Login successful for user" and "using keytab file". It runs after runDetachedModeTest has already killed the YARN application.                                                                                                                                                                                       
The TaskManager container is short-lived. In an observed failure it was launched, ran the job to FINISHED, was SIGTERM-killed (exit 143) all within about 3 seconds, and the test read its log about 200ms after the kill. At that instant the TaskManager's startup keytab-login line was not yet durably readable, so the assertion flipped to false. The JobManager log check passed. It is a sub-second race between the TaskManager log flush and the kill plus read.

 

This matches the long-standing intermittent failure tracked as FLINK-17662 (closed Cannot Reproduce), which reported the same assertion.
h3. Fix:

Verify the keytab-login strings while the containers are still alive, before the application is killed, and poll the log files instead of reading them once. The token and ACL checks are unchanged.  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)