Question on Pattern Matching

[Basanth Gowda <ba...@gmail.com>]

Hello,

We have a use case where we want to know when a certain pattern doesn't
complete within a given time frame.

For Example A -> B -> C -> D (needs to complete in 10 minutes)

Now with Flink if event D doesn't happen in 10 minutes, the pattern is
discarded and we can get notified. We also want to track how many of them
completed (even if they meet SLA). How do we achieve this with FLINK CEP or
other mechanisms?

thank you,
Basanth

Re: Question on Pattern Matching

[Kostas Kloudas <kk...@gmail.com>]

Hi Basanth,

If I understand your usecase correctly:

1) you want to find all A->B->C->D
2) for all of them you want to know how long it took to complete
3) if one completed within X it is considered ok, if not, it is
considered problematic
4) you want to count each one of them

One way to go is through CEP as Chesnay suggested and count the
resulting matches. This will give you all the requirements apart from
the one about knowing how long it took for the problematic ones to
complete.

One solution to solve this, is to specify a within() clause that is,
for example, 2x or 5x the SLA and then, for all successful (within 5x)
matches, you do the categorization manually (e,g. using a
ProcessFunction to split them into <= SLA and > SLA). After all, I
would assume that if you do not want to keep infinite state your job
has to have a cut off (in the above example 5x), after which a pattern
is considered as failed and you stop tracking it.

Another solution is to go with a ProcessFunction[1] since the
beginning and implement your logic, but if your elements arrive
"out-of-order" e.g. if B may arrive before A, then your code may need
to be pretty complicated. If out-of-orderness is not an issue then the
example in [1] can help, but still the solution will be more
complicated I guess than simply using CEP.

Hope this helps,
Kostas

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#example

On Thu, Jul 23, 2020 at 1:07 PM Basanth Gowda <ba...@gmail.com> wrote:
>
> Yes - I am able to process matched out patterns.
>
> Let's suppose I have an order fulfillment process.
>
> I want to know how many fulfillments have not met SLA and further how late they are and track until they are fulfilled.
>
> From what I tried with samples, once the pattern timeout, it is discarded and events that come after that are ignored (not applied to the pattern).
>
> Is there a better way to do it using Table API? Where I am able to emit an event (alert) when the timeout happens, and it continues to alert me - hey you fulfillment is delayed by 6 hours, 12 hours and so on and also know when it is finally completed.
>
> On Thu, Jul 16, 2020 at 3:08 PM Chesnay Schepler <ch...@apache.org> wrote:
>>
>> Have you read this part of the documentation?
>>
>> From what I understand, it provides you hooks for processing matched/timed out patterns.
>>
>>
>> On 16/07/2020 20:23, Basanth Gowda wrote:
>>
>> Hello,
>>
>> We have a use case where we want to know when a certain pattern doesn't complete within a given time frame.
>>
>> For Example A -> B -> C -> D (needs to complete in 10 minutes)
>>
>> Now with Flink if event D doesn't happen in 10 minutes, the pattern is discarded and we can get notified. We also want to track how many of them completed (even if they meet SLA). How do we achieve this with FLINK CEP or other mechanisms?
>>
>> thank you,
>> Basanth
>>
>>

Re: Question on Pattern Matching

[Basanth Gowda <ba...@gmail.com>]

Yes - I am able to process matched out patterns.

Let's suppose I have an order fulfillment process.

I want to know how many fulfillments have not met SLA and further how late
they are and track until they are fulfilled.

From what I tried with samples, once the pattern timeout, it is discarded
and events that come after that are ignored (not applied to the pattern).

Is there a better way to do it using Table API? Where I am able to emit an
event (alert) when the timeout happens, and it continues to alert me - hey
you fulfillment is delayed by 6 hours, 12 hours and so on and also know
when it is finally completed.

On Thu, Jul 16, 2020 at 3:08 PM Chesnay Schepler <ch...@apache.org> wrote:

> Have you read this part of the documentation
> <https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/libs/cep.html#handling-timed-out-partial-patterns>
> ?
>
> From what I understand, it provides you hooks for processing matched/timed
> out patterns.
>
> On 16/07/2020 20:23, Basanth Gowda wrote:
>
> Hello,
>
> We have a use case where we want to know when a certain pattern doesn't
> complete within a given time frame.
>
> For Example A -> B -> C -> D (needs to complete in 10 minutes)
>
> Now with Flink if event D doesn't happen in 10 minutes, the pattern is
> discarded and we can get notified. We also want to track how many of them
> completed (even if they meet SLA). How do we achieve this with FLINK CEP or
> other mechanisms?
>
> thank you,
> Basanth
>
>
>

Re: Question on Pattern Matching

[Chesnay Schepler <ch...@apache.org>]

Have you read this part of the documentation 
<https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/libs/cep.html#handling-timed-out-partial-patterns>?

 From what I understand, it provides you hooks for processing 
matched/timed out patterns.


On 16/07/2020 20:23, Basanth Gowda wrote:
> Hello,
>
> We have a use case where we want to know when a certain pattern 
> doesn't complete within a given time frame.
>
> For Example A -> B -> C -> D (needs to complete in 10 minutes)
>
> Now with Flink if event D doesn't happen in 10 minutes, the pattern is 
> discarded and we can get notified. We also want to track how many of 
> them completed (even if they meet SLA). How do we achieve this with 
> FLINK CEP or other mechanisms?
>
> thank you,
> Basanth