[jira] [Created] (FLINK-40078) Lost non-loss-tolerant OperatorEvent to an already-failing task triggers a failover that masks the original failure cause

["Martijn Visser (Jira)" <ji...@apache.org>]

Martijn Visser created FLINK-40078:
--------------------------------------

             Summary: Lost non-loss-tolerant OperatorEvent to an already-failing task triggers a failover that masks the original failure cause
                 Key: FLINK-40078
                 URL: https://issues.apache.org/jira/browse/FLINK-40078
             Project: Flink
          Issue Type: Bug
            Reporter: Martijn Visser


When an OperatorCoordinator sends an event to a subtask that has just failed for an application reason, the delivery fails with {{TaskNotRunningException}} ({{Task.deliverOperatorEvent}}). Because the event is not loss-tolerant, {{SubtaskGatewayImpl.sendEvent}} escalates this into

{code}
org.apache.flink.util.FlinkException: An OperatorEvent from an OperatorCoordinator to a task was lost. Triggering task failover to ensure consistency. Event: '[NoMoreSplitEvent]', ...
{code}

and triggers a task failover. Under {{NoRestartBackoffTimeStrategy}} that lost-event failover can win the race to the scheduler and become the job's reported terminal failure cause, completely replacing the original application error in the cause chain (the coordinator is co-located with the JobMaster, so its failure path can reach the scheduler faster than the failed task's own failure RPC).

The consequence is that a user debugging a failed job sees "An OperatorEvent ... was lost. Triggering task failover to ensure consistency" instead of their actual error. This is general: any coordinator event in flight at the moment a task fails for an application reason can mask that failure. It was observed concretely in FLINK-40077, where a {{NoMoreSplitsEvent}} in flight to a task that intentionally failed on the NOT NULL enforcer replaced the enforcer error as the reported cause (there it was worked around in the test scaffolding; this ticket is about the runtime behavior itself).

The escalation mechanism already has an escape hatch: on {{TaskNotRunningException}}, {{SubtaskGatewayImpl}} returns without failover when {{evt.isLossTolerant()}} is true. {{NoMoreSplitsEvent}} is currently not marked loss-tolerant.

Possible directions (needs input from runtime/coordination owners, as this touches checkpoint/failover consistency semantics):
* Mark {{NoMoreSplitsEvent}} loss-tolerant, since a task that is FAILED or FINISHED has no further use for it.
* More generally, suppress the redundant lost-event failover when the target task is already failing/failed for another reason, so the original cause is preserved.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)