Flink Stream assign unique transaction ID

[Tanguy Racinet <ta...@gmail.com>]

Hi,

I am currently working on Flink streaming API and I am trying to assign unique IDs to transactions to track changes and mine Frequent Item Sets.

I identified several potential solutions such as:
 - Check pointing Local variables (snapshotState and restoreState for once only passage)
 - Key Value State Interface (using operatorState and getRuntimeContext but is deprecated)
 - broadcast function
 - Co-Map function 

After trying all of them, the only one I could actually make work for my needs uses the deprecated operatorState.
I tried using co-map function but couldn’t the number from one map and use it in the other mapper to modify the Tid value.

Is there any solution I haven’t thought of in order to assign a unique Long to every one of my transactions within the DataStream and to ensure they are not only unique but also ordered and linearly distributed (transaction1 -> id = 1; transaction2 -> id = 2; etc)
Or maybe some working examples of the solutions I tried but couldn’t obtain what I needed with it ?

Thank you in advance.
Regards,
Tanguy R.

Re: Flink Stream assign unique transaction ID

[Aljoscha Krettek <al...@apache.org>]

Hi,
OperatorState was discarded in favor of ValueState, which you can retrieve using RuntimeContext.getState(…). This provides the same functionality as OperatorState but is the more future proof interface.

Cheers,
Aljoscha
> On 19 Feb 2016, at 15:38, Tanguy Racinet <ta...@gmail.com> wrote:
> 
> Hi,
> 
> I am currently working on Flink streaming API and I am trying to assign unique IDs to transactions to track changes and mine Frequent Item Sets.
> 
> I identified several potential solutions such as:
> - Check pointing Local variables (snapshotState and restoreState for once only passage)
> - Key Value State Interface (using operatorState and getRuntimeContext but is deprecated)
> - broadcast function
> - Co-Map function 
> 
> After trying all of them, the only one I could actually make work for my needs uses the deprecated operatorState.
> I tried using co-map function but couldn’t the number from one map and use it in the other mapper to modify the Tid value.
> 
> Is there any solution I haven’t thought of in order to assign a unique Long to every one of my transactions within the DataStream and to ensure they are not only unique but also ordered and linearly distributed (transaction1 -> id = 1; transaction2 -> id = 2; etc)
> Or maybe some working examples of the solutions I tried but couldn’t obtain what I needed with it ?
> 
> Thank you in advance.
> Regards,
> Tanguy R.