You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Oriol <or...@gmail.com> on 2017/01/26 12:04:54 UTC
CEP and KeyedStreams doubt
Hello everyone,
I'm using the CEP library for event stream processing.
I'm splitting the dataStream into different KeyedStreams using keyBy(). In
the KeyBy, I'm using a tuple of two elements, which means I may have
several millions of KeyedStreams, as I need to monitor all our customer's
users.
Is this the preferred way to use Flink, or should I find a way to reduce
the number of KeyedStreams, for example having one per customer instead of
one per customer's user? (And find a way later to process each user by
itself).
Also, is the bug reported in
https://issues.apache.org/jira/browse/FLINK-5174 related to keys of the
KeyedStreams? I'm not sure what kind of keys it is related to. If so, is it
going to be addressed soon?
Thanks,
Oriol.
Re: CEP and KeyedStreams doubt
Posted by Kostas Kloudas <k....@data-artisans.com>.
Hi Oriol,
The number of keys is related to the number of data-structures (NFAs) Flink is going to create and keep.
Given this, it may make sense to try to reduce your key-space (or your keyedStreams). Other than that, Flink
has not issue handling large numbers of keys.
Now, for the issue you mentioned, we hope to get it fixed soon but there is no concrete horizon yet.
Hope this helps!
Let us know if you have any issues,
Kostas
> On Jan 26, 2017, at 1:04 PM, Oriol <or...@gmail.com> wrote:
>
> Hello everyone,
>
> I'm using the CEP library for event stream processing.
>
> I'm splitting the dataStream into different KeyedStreams using keyBy(). In the KeyBy, I'm using a tuple of two elements, which means I may have several millions of KeyedStreams, as I need to monitor all our customer's users.
>
> Is this the preferred way to use Flink, or should I find a way to reduce the number of KeyedStreams, for example having one per customer instead of one per customer's user? (And find a way later to process each user by itself).
>
> Also, is the bug reported in https://issues.apache.org/jira/browse/FLINK-5174 <https://issues.apache.org/jira/browse/FLINK-5174> related to keys of the KeyedStreams? I'm not sure what kind of keys it is related to. If so, is it going to be addressed soon?
>
> Thanks,
> Oriol.