[jira] [Updated] (FLINK-24666) Add job level "table.exec.state-stale.error-handling" option and apply to related stateful stream operators

["Jing Ge (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-24666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jing Ge updated FLINK-24666:
----------------------------
    Fix Version/s: 1.19.0
                       (was: 1.18.0)

> Add job level "table.exec.state-stale.error-handling" option and apply to related stateful stream operators
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-24666
>                 URL: https://issues.apache.org/jira/browse/FLINK-24666
>             Project: Flink
>          Issue Type: Improvement
>          Components: Table SQL / Runtime
>    Affects Versions: 1.14.0, 1.13.3
>            Reporter: lincoln lee
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.19.0
>
>
> In stream processing, records will be deleted when exceed state ttl (if configured), and when the corresponding record's update arrives again,  the operator may not be able to handle it properly, we need a unified error handling mechanism to handle this situation, 
> instead of each stateful operator currently handling its own.
> e.g., currently, there's a lenient option which was not exposed to users on RetractableTopNFunction
> {quote}// flag to skip records with non-exist error instead to fail, true by default.
> private final boolean lenient = true
> {quote}
> So there's no chance to raise the exception when the record(s) unexpectedly cleared by state ttl. Commonly this happens because a too shorter ttl at Line 190 or inconstancy between the two internal state(dataState and treeMap) at other place.
> {quote}List<RowData> inputs = dataState.get(key);
> if (inputs == null)
> Unknown macro: \{ // Skip the data if it's state is cleared because of state ttl. if (lenient)  Unknown macro}
>  else 
> Unknown macro: \{       throw new RuntimeException(STATE_CLEARED_WARN_MSG); }
> }
> {quote}
> We'd better to expose it to users (default value can be true to keep consistent with previous version) and should be unified to cover all the stateful stream operators than specific to RetractableTopNFunction.
> The new option 'table.exec.state-stale.error-handling' has three values:
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)