You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Juan Gentile <j....@criteo.com> on 2019/07/05 15:07:37 UTC
FlinkKafkaConsumer - Save watermark assigner in the state
Hello,
We are currently facing an issue where we need to store the instance of the watermark and timestamp assigner in the state while consumer from Kafka.
For that purpose we took a look at FlinkKafkaConsumerBase and noticed that since the methods (snapshotState and initializeState from the CheckpointedFunction) are final we can’t override them.
An alternative could be to wrap the class but we don’t like that solution either as it would make it harder to migrate in the future if there are changes there.
We would like to know if it’d be possible to store the ‘assigner’ in the state of the FlinkKafkaConsumerBase. We think it’s not a big change and we are willing to do it if you agree.
Thank you,
Juan G.
Re: FlinkKafkaConsumer - Save watermark assigner in the state
Posted by Konstantin Knauf <ko...@ververica.com>.
Hi Juan,
can you elaborate a bit on why you want to put the WatermarkAssigner itself
into state? It is generally unusual to store a UDF in Managed State.
Cheers,
Konstantin
On Fri, Jul 5, 2019 at 5:07 PM Juan Gentile <j....@criteo.com> wrote:
> Hello,
>
>
>
> We are currently facing an issue where we need to store the instance of
> the watermark and timestamp assigner in the state while consumer from Kafka.
>
> For that purpose we took a look at FlinkKafkaConsumerBase and noticed that
> since the methods (snapshotState and initializeState from the
> CheckpointedFunction) are final we can’t override them.
>
> An alternative could be to wrap the class but we don’t like that solution
> either as it would make it harder to migrate in the future if there are
> changes there.
>
> We would like to know if it’d be possible to store the ‘assigner’ in the
> state of the FlinkKafkaConsumerBase. We think it’s not a big change and we
> are willing to do it if you agree.
>
>
>
> Thank you,
>
> Juan G.
>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
Planned Absences: 10.08.2019 - 31.08.2019, 05.09. - 06.09.2010
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen