[jira] [Created] (FLINK-37089) Advanced scheduling for derived async state processing

["Zakelly Lan (Jira)" <ji...@apache.org>]

Zakelly Lan created FLINK-37089:
-----------------------------------

             Summary: Advanced scheduling for derived async state processing
                 Key: FLINK-37089
                 URL: https://issues.apache.org/jira/browse/FLINK-37089
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Async State Processing, Runtime / Task
            Reporter: Zakelly Lan
            Assignee: Zakelly Lan


Currently, there is a limit for in-flight records in async state processing. However, we allow developers to initialize a new processing in-middle of another. These derived processings are treated as normal ones for timers or upstreaming records, which is problematic:
* The new derived processing may increase the number of in-flight requests and cause force draining when it reaches the limit. But the draining may require the current processing also finish. This is a deadlock.
* The derived processing will queue behind normal processing, which is not the behavior the user wants. The derived ones should be fired right after the current processing or ASAP.

Thus, I will change the behavior in `AsyncExecutionController` and its related buffer:
* Avoid drain when creating derived processing.
* Provide priority for the queuing processing and make derived one have greater priority.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)