You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alexis Sarda-Espinosa <sa...@gmail.com> on 2025/09/09 14:23:37 UTC

State processor's table API with custom map value

Hello,

I am looking at the new APIs for state processing [1] and I see custom
serde should be supported with a property like
"fields.#.value-type-factory". However, I'm not sure if that applies to
MapState. I have an operator with MapState using String as key but with a
custom type serializer for the values [2], and regardless of how I
define my value type factory, I get an exception stating that the new
serializer must be compatible with the old one, where both the old and new
serializers just mention its the map serializer.

Am I missing something or is this actually not supported yet?

[1]
https://nightlies.apache.org/flink/flink-docs-release-2.1/docs/libs/state_processor_api/#table-api
[2] https://gist.github.com/asardaes/714b8c1db0c4020f5fde9865b95fc398

Regards,
Alexis.

Re: State processor's table API with custom map value

Posted by Alexis Sarda-Espinosa <sa...@gmail.com>.
Hi Gabor,

I can probably do that, in the meantime I thought about how it would look,
and it's pretty clear it can't be supported yet, since the SQL statement
would have to support something like:

MyMapState MAP<STRING, COLLECTION<STRING>>


where the values would have a custom value type factory.

Regards,
Alexis.

Am Di., 9. Sept. 2025 um 16:48 Uhr schrieb Gabor Somogyi <
gabor.g.somogyi@gmail.com>:

> Hi Alexis,
>
> The state table API is relatively new feature and as such actively
> developed based on feedbacks.
> Without having an in-depth look I think this is something what is not yet
> covered.
> If you can create a super simple repro app I can take a detailed look and
> add support for it.
>
> BR,
> G
>
>
> On Tue, Sep 9, 2025 at 4:24 PM Alexis Sarda-Espinosa <
> sarda.espinosa@gmail.com> wrote:
>
>> Hello,
>>
>> I am looking at the new APIs for state processing [1] and I see custom
>> serde should be supported with a property like
>> "fields.#.value-type-factory". However, I'm not sure if that applies to
>> MapState. I have an operator with MapState using String as key but with a
>> custom type serializer for the values [2], and regardless of how I
>> define my value type factory, I get an exception stating that the new
>> serializer must be compatible with the old one, where both the old and new
>> serializers just mention its the map serializer.
>>
>> Am I missing something or is this actually not supported yet?
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-release-2.1/docs/libs/state_processor_api/#table-api
>> [2] https://gist.github.com/asardaes/714b8c1db0c4020f5fde9865b95fc398
>>
>> Regards,
>> Alexis.
>>
>

Re: State processor's table API with custom map value

Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Alexis,

The state table API is relatively new feature and as such actively
developed based on feedbacks.
Without having an in-depth look I think this is something what is not yet
covered.
If you can create a super simple repro app I can take a detailed look and
add support for it.

BR,
G


On Tue, Sep 9, 2025 at 4:24 PM Alexis Sarda-Espinosa <
sarda.espinosa@gmail.com> wrote:

> Hello,
>
> I am looking at the new APIs for state processing [1] and I see custom
> serde should be supported with a property like
> "fields.#.value-type-factory". However, I'm not sure if that applies to
> MapState. I have an operator with MapState using String as key but with a
> custom type serializer for the values [2], and regardless of how I
> define my value type factory, I get an exception stating that the new
> serializer must be compatible with the old one, where both the old and new
> serializers just mention its the map serializer.
>
> Am I missing something or is this actually not supported yet?
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-2.1/docs/libs/state_processor_api/#table-api
> [2] https://gist.github.com/asardaes/714b8c1db0c4020f5fde9865b95fc398
>
> Regards,
> Alexis.
>