You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Marco Villalobos <mv...@kineteque.com> on 2021/06/05 05:54:47 UTC
Is it possible to use OperatorState, when NOT implementing a source
or sink function?
Is it possible to use OperatorState, when NOT implementing a source or sink
function?
If yes, then how?
Re: Re: Is it possible to use OperatorState, when NOT implementing a source or sink function?
Posted by Yun Gao <yu...@aliyun.com>.
Hi Marco,
It seems to me that the imbalance problem and the state is independent for this issue: the data distribution
is only decided by the KeySelector used. The only limitation for state is that the keyed state is bind to the
KeySelector used across the tasks. If the imbalance is the root problem, have you checked
how many keys in total does the job have ?
Best,
Yun
------------------Original Mail ------------------
Sender:Marco Villalobos <mv...@kineteque.com>
Send Date:Sat Jun 5 23:26:09 2021
Recipients:JING ZHANG <be...@gmail.com>
CC:user <us...@flink.apache.org>
Subject:Re: Is it possible to use OperatorState, when NOT implementing a source or sink function?
Oh....that won't work for me either. I needed to use MapState.
Perhaps I should describe my problem. I am using a KeyedState process function, but the workload that it is processing is not distributing well across the cluster. I have four task managers, but the way my data is keyed in this operator, it only goes to one task manager node.
I need state, but I don't really need it keyed.
On Sat, Jun 5, 2021 at 4:56 AM Marco Villalobos <mv...@kineteque.com> wrote:
Does that work in the DataStream API in Batch Execution Mode?
On Sat, Jun 5, 2021 at 12:04 AM JING ZHANG <be...@gmail.com> wrote:
Hi,please use `CheckpointedFunction`, you could initialize your operator state in `initializeState` method by using context.getOperatorStateStore().***
Best regards,
JING ZHANG
Marco Villalobos <mv...@kineteque.com> 于2021年6月5日周六 下午1:55写道：
Is it possible to use OperatorState, when NOT implementing a source or sink function?
If yes, then how?
Re: Is it possible to use OperatorState, when NOT implementing a
source or sink function?
Posted by Marco Villalobos <mv...@kineteque.com>.
Oh....that won't work for me either. I needed to use MapState.
Perhaps I should describe my problem. I am using a KeyedState process
function, but the workload that it is processing is not distributing well
across the cluster. I have four task managers, but the way my data is keyed
in this operator, it only goes to one task manager node.
I need state, but I don't really need it keyed.
On Sat, Jun 5, 2021 at 4:56 AM Marco Villalobos <mv...@kineteque.com>
wrote:
> Does that work in the DataStream API in Batch Execution Mode?
>
> On Sat, Jun 5, 2021 at 12:04 AM JING ZHANG <be...@gmail.com> wrote:
>
>> Hi,
>> please use `CheckpointedFunction`, you could initialize your operator
>> state in `initializeState` method by using
>> context.getOperatorStateStore().***
>>
>> Best regards,
>> JING ZHANG
>>
>>
>> Marco Villalobos <mv...@kineteque.com> 于2021年6月5日周六 下午1:55写道：
>>
>>> Is it possible to use OperatorState, when NOT implementing a source or
>>> sink function?
>>>
>>> If yes, then how?
>>>
>>
Re: Re: Is it possible to use OperatorState, when NOT implementing a source or sink function?
Posted by Yun Gao <yu...@aliyun.com>.
Hi Marco,
I think yes, the operator state could be used in batch mode. Since there
is no checkpoint in batch mode, the operator state would serve as a kind
of ordinary in-memory storage.
Best,
Yun
------------------------------------------------------------------
Sender:Marco Villalobos<mv...@kineteque.com>
Date:2021/06/05 19:56:08
Recipient:JING ZHANG<be...@gmail.com>
Cc:user<us...@flink.apache.org>
Theme:Re: Is it possible to use OperatorState, when NOT implementing a source or sink function?
Does that work in the DataStream API in Batch Execution Mode?
On Sat, Jun 5, 2021 at 12:04 AM JING ZHANG <be...@gmail.com> wrote:
Hi,please use `CheckpointedFunction`, you could initialize your operator state in `initializeState` method by using context.getOperatorStateStore().***
Best regards,
JING ZHANG
Marco Villalobos <mv...@kineteque.com> 于2021年6月5日周六 下午1:55写道：
Is it possible to use OperatorState, when NOT implementing a source or sink function?
If yes, then how?
Re: Is it possible to use OperatorState, when NOT implementing a
source or sink function?
Posted by Marco Villalobos <mv...@kineteque.com>.
Does that work in the DataStream API in Batch Execution Mode?
On Sat, Jun 5, 2021 at 12:04 AM JING ZHANG <be...@gmail.com> wrote:
> Hi,
> please use `CheckpointedFunction`, you could initialize your operator
> state in `initializeState` method by using
> context.getOperatorStateStore().***
>
> Best regards,
> JING ZHANG
>
>
> Marco Villalobos <mv...@kineteque.com> 于2021年6月5日周六 下午1:55写道：
>
>> Is it possible to use OperatorState, when NOT implementing a source or
>> sink function?
>>
>> If yes, then how?
>>
>
Re: Is it possible to use OperatorState, when NOT implementing a
source or sink function?
Posted by JING ZHANG <be...@gmail.com>.
Hi,
please use `CheckpointedFunction`, you could initialize your operator state
in `initializeState` method by using context.getOperatorStateStore().***
Best regards,
JING ZHANG
Marco Villalobos <mv...@kineteque.com> 于2021年6月5日周六 下午1:55写道：
> Is it possible to use OperatorState, when NOT implementing a source or
> sink function?
>
> If yes, then how?
>