Re: Feature request: Removing state from operators

[Peter Westermann <no...@genesys.com>]

Renaming operators and migrating the state we still need manually is what we have done in the past. I was just hoping for a more convenient solution.

Peter

From: David Anderson <da...@apache.org>
Date: Friday, October 30, 2020 at 5:55 PM
To: Peter Westermann <no...@genesys.com>, "user@flink.apache.org" <us...@flink.apache.org>
Subject: Re: Feature request: Removing state from operators

It seems like another option here would be to occasionally use the state processor API to purge a savepoint of all unnecessary state.

On Fri, Oct 30, 2020 at 6:57 PM Steven Wu <st...@gmail.com>> wrote:
not a solution, but a potential workaround. Maybe rename the operator uid so that you can continue to leverage allowNonRestoredState?

On Thu, Oct 29, 2020 at 7:58 AM Peter Westermann <no...@genesys.com>> wrote:
Does that actually allow removing a state completely (vs. just modifying the values stored in state)?

Ideally, we would want to just interact with state via KeyedStateStore. Maybe it would be possible to add a couple methods there, e.g. like this:
// List all pre-existing states
<S extends State, T> List<StateDescriptor<S, T>> listStates();
// Completely remove a state
<S extends State, T> void dropState(StateDescriptor<S, T> stateDescriptor);


Thanks,
Peter



From: Congxian Qiu <qc...@gmail.com>>
Date: Thursday, October 29, 2020 at 10:38 AM
To: Robert Metzger <rm...@apache.org>>
Cc: Peter Westermann <no...@genesys.com>>, "user@flink.apache.org<ma...@flink.apache.org>" <us...@flink.apache.org>>
Subject: Re: Feature request: Removing state from operators

Hi Peter
     Can applyToAllKeys[1] in KeyedStateBackend help you here? but currently, this is not exposed to users now.

[1] https://github.com/apache/flink/blob/fada6fb6ac9fd7f6510f1f2d77b6baa06563e222/flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyedStateBackend.java#L65<https://github.com/apache/flink/blob/fada6fb6ac9fd7f6510f1f2d77b6baa06563e222/flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyedStateBackend.java#L65>

Best,
Congxian


Robert Metzger <rm...@apache.org>> 于2020年10月27日周二 下午5:51写道：
Hi Peter,

I'm adding two committers to this thread who can help answering your question.

On Mon, Oct 26, 2020 at 3:22 PM Peter Westermann <no...@genesys.com>> wrote:
We use the feature for removing stateful operators via the allowNonRestoredState relatively often and it works great. However, there doesn’t seem to be anything like that for removing state from an existing operator (that we want to keep).
Say my operator defines a MapState and a ValueState. Later on, the ValueState becomes obsolete. In this case, we can remove the actual data for each key by clearing it out but the state itself is still referenced in savepoints even if it’s not referenced in code anymore – that e.g. means one cannot remove any class that was previously used in state.
Would it be possible to add support for completely removing state from an operator if it’s no longer referenced in code and allowNonRestoredState is set? (Or to add an explicit “drop this state option” in KeyedStateStore and OperatorStateStore?)

Thanks,

Peter