You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/07/22 14:00:28 UTC
[GitHub] [flink-web] afedulov opened a new pull request #362: Add Blogbost: Advanced Flink Application Patterns: Custom Window Proc…
afedulov opened a new pull request #362:
URL: https://github.com/apache/flink-web/pull/362
Adds third part of the Flink Patterns blogpost based on building a Fraud Detection Engine with Flink.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] afedulov commented on a change in pull request #362: Add Blogbost: Advanced Flink Application Patterns: Custom Window Proc…
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #362:
URL: https://github.com/apache/flink-web/pull/362#discussion_r461477217
##########
File path: _posts/2020-07-23-demo-fraud-detection-3.md
##########
@@ -0,0 +1,655 @@
+---
+layout: post
+title: "Advanced Flink Application Patterns Vol.3: Custom Window Processing"
+date: 2020-07-23T12:00:00.000Z
+authors:
+- alex:
+ name: "Alexander Fedulov"
+ twitter: "alex_fedulov"
+categories: news
+excerpt: In this series of blog posts you will learn about powerful Flink patterns for building streaming applications.
+---
+
+<style type="text/css">
+.tg {border-collapse:collapse;border-spacing:0;}
+.tg td{padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}
+.tg th{padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;background-color:#eff0f1;}
+.tg .tg-wide{padding:10px 30px;}
+.tg .tg-top{vertical-align:top}
+.tg .tg-topcenter{text-align:center;vertical-align:top}
+.tg .tg-center{text-align:center;vertical-align:center}
+</style>
+
+## Introduction
+
+In the previous articles of the series, we described how you can achieve
+flexible stream partitioning based on dynamically-updated configurations
+(a set of fraud-detection rules) and how you can utilize Flink\'s
+Broadcast mechanism to distribute processing configuration at runtime
+among the relevant operators.
+
+Following up directly where we left the discussion of the end-to-end
+solution last time, in this article we will describe how you can use the
+\"Swiss knife\" of Flink - the [*Process Function*](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html) to create an
+implementation that is tailor-made to match your streaming business
+logic requirements. Our discussion will continue in the context of the
+[Fraud Detection engine]({{ site.baseurl }}/news/2020/01/15/demo-fraud-detection.html#fraud-detection-demo). We will also demonstrate how you can
+implement your own **custom replacement for time windows** for cases
+where the out-of-the-box windowing available from the DataStream API
+does not satisfy your requirements. In particular, we will look at the
+trade-offs that you can make when designing a solution which requires
+low-latency reactions to individual events.
+
+This article will describe some high-level concepts that can be applied
+independently, but it is recommended that you review the material in
+[part one]({{ site.baseurl }}/news/2020/01/15/demo-fraud-detection.html) and
+[part two]({{ site.baseurl }}/news/2020/03/24/demo-fraud-detection-2.html) of the series as well as checkout the [code
+base](https://github.com/afedulov/fraud-detection-demo) in order to make
+it easier to follow along.
+
+## ProcessFunction as a "Window"
+
+### Low Latency
+
+Let's start with a reminder of the type of fraud detection rule that we
+would like to support:
+
+*"Whenever the **sum** of **payments** from the same **payer** to the
+same **beneficiary** within **a 24 hour
+period** is **greater** than **200 000 \$** - trigger an alert."*
+
+In other words, given a stream of transactions partitioned by a key that
+combines the payer and the beneficiary fields, we would like to look
+back in time and determine, for each incoming transaction, if the sum of
+all previous payments between the two specific participants exceeds the
+defined threshold. In effect, the computation window is always moved
+along to the position of the last observed event for a particular data
+partitioning key.
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/patterns-blog-3/time-windows.png" width="600px" alt="Figure 1: Time Windows"/>
+<br/>
+<i><small>Figure 1: Time Windows</small></i>
+</center>
+<br/>
+
+
+One of the common key requirements for a fraud detection system is *low
+response time*. The sooner the fraudulent action gets detected, the
+higher the chances that it can be blocked and its negative consequences
+mitigated. This requirement is especially prominent in the financial
+domain, where you have one important constraint - any time spent
+evaluating a fraud detection model is time that a law-abiding user of
+your system will spend waiting for a response. Swiftness of processing
+often becomes a competitive advantage between various payment systems
+and the time limit for producing an alert could lie as low as *300-500
+ms*. This is all the time you get from the moment of ingestion of a
+transaction event into a fraud detection system until an alert has to
+become available to downstream systems.
+
+As you might know, Flink provides a powerful [Window
+API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html)
+that is applicable for a wide range of use cases. However, If you go
+over all of the available types of supported windows, you will realize
+that none of them exactly match our main requirement for this use case -
+the low-latency evaluation of *each* incoming transaction. There is
+no type of window in Flink that can express the *"x minutes/hours/days
+back from the <u>current event</u>"* semantic. In the Window API, events
+fall into windows (as defined by the window
+[assigners](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#window-assigners)),
+but they cannot themselves individually control the creation and
+evaluation of windows\*. As described above, our goal for the fraud
+detection engine is to achieve immediate evaluation of the previous
+relevant data points as soon as the new event is received. This raises
+the question of feasibility of applying the Window API in this case.
Review comment:
I've looked into this approach a while ago and came to the conclusion that it leads to some overly complex implementation, not even taking into account that it won't work with the dynamic rules updates - windows have to be statically defined. Not sure how much value it adds to mention something we do not recommend and that does not match the use case requirements (broadcasted rules).
##########
File path: _posts/2020-07-23-demo-fraud-detection-3.md
##########
@@ -0,0 +1,655 @@
+---
+layout: post
+title: "Advanced Flink Application Patterns Vol.3: Custom Window Processing"
+date: 2020-07-23T12:00:00.000Z
+authors:
+- alex:
+ name: "Alexander Fedulov"
+ twitter: "alex_fedulov"
+categories: news
+excerpt: In this series of blog posts you will learn about powerful Flink patterns for building streaming applications.
+---
+
+<style type="text/css">
+.tg {border-collapse:collapse;border-spacing:0;}
+.tg td{padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}
+.tg th{padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;background-color:#eff0f1;}
+.tg .tg-wide{padding:10px 30px;}
+.tg .tg-top{vertical-align:top}
+.tg .tg-topcenter{text-align:center;vertical-align:top}
+.tg .tg-center{text-align:center;vertical-align:center}
+</style>
+
+## Introduction
+
+In the previous articles of the series, we described how you can achieve
+flexible stream partitioning based on dynamically-updated configurations
+(a set of fraud-detection rules) and how you can utilize Flink\'s
+Broadcast mechanism to distribute processing configuration at runtime
+among the relevant operators.
+
+Following up directly where we left the discussion of the end-to-end
+solution last time, in this article we will describe how you can use the
+\"Swiss knife\" of Flink - the [*Process Function*](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html) to create an
+implementation that is tailor-made to match your streaming business
+logic requirements. Our discussion will continue in the context of the
+[Fraud Detection engine]({{ site.baseurl }}/news/2020/01/15/demo-fraud-detection.html#fraud-detection-demo). We will also demonstrate how you can
+implement your own **custom replacement for time windows** for cases
+where the out-of-the-box windowing available from the DataStream API
+does not satisfy your requirements. In particular, we will look at the
+trade-offs that you can make when designing a solution which requires
+low-latency reactions to individual events.
+
+This article will describe some high-level concepts that can be applied
+independently, but it is recommended that you review the material in
+[part one]({{ site.baseurl }}/news/2020/01/15/demo-fraud-detection.html) and
+[part two]({{ site.baseurl }}/news/2020/03/24/demo-fraud-detection-2.html) of the series as well as checkout the [code
+base](https://github.com/afedulov/fraud-detection-demo) in order to make
+it easier to follow along.
+
+## ProcessFunction as a "Window"
+
+### Low Latency
+
+Let's start with a reminder of the type of fraud detection rule that we
+would like to support:
+
+*"Whenever the **sum** of **payments** from the same **payer** to the
+same **beneficiary** within **a 24 hour
+period** is **greater** than **200 000 \$** - trigger an alert."*
+
+In other words, given a stream of transactions partitioned by a key that
+combines the payer and the beneficiary fields, we would like to look
+back in time and determine, for each incoming transaction, if the sum of
+all previous payments between the two specific participants exceeds the
+defined threshold. In effect, the computation window is always moved
+along to the position of the last observed event for a particular data
+partitioning key.
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/patterns-blog-3/time-windows.png" width="600px" alt="Figure 1: Time Windows"/>
+<br/>
+<i><small>Figure 1: Time Windows</small></i>
+</center>
+<br/>
+
+
+One of the common key requirements for a fraud detection system is *low
+response time*. The sooner the fraudulent action gets detected, the
+higher the chances that it can be blocked and its negative consequences
+mitigated. This requirement is especially prominent in the financial
+domain, where you have one important constraint - any time spent
+evaluating a fraud detection model is time that a law-abiding user of
+your system will spend waiting for a response. Swiftness of processing
+often becomes a competitive advantage between various payment systems
+and the time limit for producing an alert could lie as low as *300-500
+ms*. This is all the time you get from the moment of ingestion of a
+transaction event into a fraud detection system until an alert has to
+become available to downstream systems.
+
+As you might know, Flink provides a powerful [Window
+API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html)
+that is applicable for a wide range of use cases. However, If you go
+over all of the available types of supported windows, you will realize
+that none of them exactly match our main requirement for this use case -
+the low-latency evaluation of *each* incoming transaction. There is
+no type of window in Flink that can express the *"x minutes/hours/days
+back from the <u>current event</u>"* semantic. In the Window API, events
+fall into windows (as defined by the window
+[assigners](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#window-assigners)),
Review comment:
Good idea, changed the links.
##########
File path: _posts/2020-07-23-demo-fraud-detection-3.md
##########
@@ -0,0 +1,655 @@
+---
+layout: post
+title: "Advanced Flink Application Patterns Vol.3: Custom Window Processing"
+date: 2020-07-23T12:00:00.000Z
+authors:
+- alex:
+ name: "Alexander Fedulov"
+ twitter: "alex_fedulov"
+categories: news
+excerpt: In this series of blog posts you will learn about powerful Flink patterns for building streaming applications.
+---
+
+<style type="text/css">
+.tg {border-collapse:collapse;border-spacing:0;}
+.tg td{padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}
+.tg th{padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;background-color:#eff0f1;}
+.tg .tg-wide{padding:10px 30px;}
+.tg .tg-top{vertical-align:top}
+.tg .tg-topcenter{text-align:center;vertical-align:top}
+.tg .tg-center{text-align:center;vertical-align:center}
+</style>
+
+## Introduction
+
+In the previous articles of the series, we described how you can achieve
+flexible stream partitioning based on dynamically-updated configurations
+(a set of fraud-detection rules) and how you can utilize Flink\'s
+Broadcast mechanism to distribute processing configuration at runtime
+among the relevant operators.
+
+Following up directly where we left the discussion of the end-to-end
+solution last time, in this article we will describe how you can use the
+\"Swiss knife\" of Flink - the [*Process Function*](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html) to create an
+implementation that is tailor-made to match your streaming business
+logic requirements. Our discussion will continue in the context of the
+[Fraud Detection engine]({{ site.baseurl }}/news/2020/01/15/demo-fraud-detection.html#fraud-detection-demo). We will also demonstrate how you can
+implement your own **custom replacement for time windows** for cases
+where the out-of-the-box windowing available from the DataStream API
+does not satisfy your requirements. In particular, we will look at the
+trade-offs that you can make when designing a solution which requires
+low-latency reactions to individual events.
+
+This article will describe some high-level concepts that can be applied
+independently, but it is recommended that you review the material in
+[part one]({{ site.baseurl }}/news/2020/01/15/demo-fraud-detection.html) and
+[part two]({{ site.baseurl }}/news/2020/03/24/demo-fraud-detection-2.html) of the series as well as checkout the [code
+base](https://github.com/afedulov/fraud-detection-demo) in order to make
+it easier to follow along.
+
+## ProcessFunction as a "Window"
+
+### Low Latency
+
+Let's start with a reminder of the type of fraud detection rule that we
+would like to support:
+
+*"Whenever the **sum** of **payments** from the same **payer** to the
+same **beneficiary** within **a 24 hour
+period** is **greater** than **200 000 \$** - trigger an alert."*
+
+In other words, given a stream of transactions partitioned by a key that
+combines the payer and the beneficiary fields, we would like to look
+back in time and determine, for each incoming transaction, if the sum of
+all previous payments between the two specific participants exceeds the
+defined threshold. In effect, the computation window is always moved
+along to the position of the last observed event for a particular data
+partitioning key.
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/patterns-blog-3/time-windows.png" width="600px" alt="Figure 1: Time Windows"/>
+<br/>
+<i><small>Figure 1: Time Windows</small></i>
+</center>
+<br/>
+
+
+One of the common key requirements for a fraud detection system is *low
+response time*. The sooner the fraudulent action gets detected, the
+higher the chances that it can be blocked and its negative consequences
+mitigated. This requirement is especially prominent in the financial
+domain, where you have one important constraint - any time spent
+evaluating a fraud detection model is time that a law-abiding user of
+your system will spend waiting for a response. Swiftness of processing
+often becomes a competitive advantage between various payment systems
+and the time limit for producing an alert could lie as low as *300-500
+ms*. This is all the time you get from the moment of ingestion of a
+transaction event into a fraud detection system until an alert has to
+become available to downstream systems.
+
+As you might know, Flink provides a powerful [Window
+API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html)
+that is applicable for a wide range of use cases. However, If you go
+over all of the available types of supported windows, you will realize
+that none of them exactly match our main requirement for this use case -
+the low-latency evaluation of *each* incoming transaction. There is
+no type of window in Flink that can express the *"x minutes/hours/days
+back from the <u>current event</u>"* semantic. In the Window API, events
+fall into windows (as defined by the window
+[assigners](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#window-assigners)),
+but they cannot themselves individually control the creation and
+evaluation of windows\*. As described above, our goal for the fraud
+detection engine is to achieve immediate evaluation of the previous
+relevant data points as soon as the new event is received. This raises
+the question of feasibility of applying the Window API in this case.
Review comment:
Thanks, @NicoK , I think your proposition sounds very good. I am adding it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] NicoK commented on pull request #362: Add Blogbost: Advanced Flink Application Patterns: Custom Window Proc…
Posted by GitBox <gi...@apache.org>.
NicoK commented on pull request #362:
URL: https://github.com/apache/flink-web/pull/362#issuecomment-665077886
I'll merge on Thursday to not overlap with today's post
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] NicoK commented on a change in pull request #362: Add Blogbost: Advanced Flink Application Patterns: Custom Window Proc…
Posted by GitBox <gi...@apache.org>.
NicoK commented on a change in pull request #362:
URL: https://github.com/apache/flink-web/pull/362#discussion_r461605409
##########
File path: _posts/2020-07-23-demo-fraud-detection-3.md
##########
@@ -0,0 +1,655 @@
+---
+layout: post
+title: "Advanced Flink Application Patterns Vol.3: Custom Window Processing"
+date: 2020-07-23T12:00:00.000Z
+authors:
+- alex:
+ name: "Alexander Fedulov"
+ twitter: "alex_fedulov"
+categories: news
+excerpt: In this series of blog posts you will learn about powerful Flink patterns for building streaming applications.
+---
+
+<style type="text/css">
+.tg {border-collapse:collapse;border-spacing:0;}
+.tg td{padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}
+.tg th{padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;background-color:#eff0f1;}
+.tg .tg-wide{padding:10px 30px;}
+.tg .tg-top{vertical-align:top}
+.tg .tg-topcenter{text-align:center;vertical-align:top}
+.tg .tg-center{text-align:center;vertical-align:center}
+</style>
+
+## Introduction
+
+In the previous articles of the series, we described how you can achieve
+flexible stream partitioning based on dynamically-updated configurations
+(a set of fraud-detection rules) and how you can utilize Flink\'s
+Broadcast mechanism to distribute processing configuration at runtime
+among the relevant operators.
+
+Following up directly where we left the discussion of the end-to-end
+solution last time, in this article we will describe how you can use the
+\"Swiss knife\" of Flink - the [*Process Function*](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html) to create an
+implementation that is tailor-made to match your streaming business
+logic requirements. Our discussion will continue in the context of the
+[Fraud Detection engine]({{ site.baseurl }}/news/2020/01/15/demo-fraud-detection.html#fraud-detection-demo). We will also demonstrate how you can
+implement your own **custom replacement for time windows** for cases
+where the out-of-the-box windowing available from the DataStream API
+does not satisfy your requirements. In particular, we will look at the
+trade-offs that you can make when designing a solution which requires
+low-latency reactions to individual events.
+
+This article will describe some high-level concepts that can be applied
+independently, but it is recommended that you review the material in
+[part one]({{ site.baseurl }}/news/2020/01/15/demo-fraud-detection.html) and
+[part two]({{ site.baseurl }}/news/2020/03/24/demo-fraud-detection-2.html) of the series as well as checkout the [code
+base](https://github.com/afedulov/fraud-detection-demo) in order to make
+it easier to follow along.
+
+## ProcessFunction as a "Window"
+
+### Low Latency
+
+Let's start with a reminder of the type of fraud detection rule that we
+would like to support:
+
+*"Whenever the **sum** of **payments** from the same **payer** to the
+same **beneficiary** within **a 24 hour
+period** is **greater** than **200 000 \$** - trigger an alert."*
+
+In other words, given a stream of transactions partitioned by a key that
+combines the payer and the beneficiary fields, we would like to look
+back in time and determine, for each incoming transaction, if the sum of
+all previous payments between the two specific participants exceeds the
+defined threshold. In effect, the computation window is always moved
+along to the position of the last observed event for a particular data
+partitioning key.
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/patterns-blog-3/time-windows.png" width="600px" alt="Figure 1: Time Windows"/>
+<br/>
+<i><small>Figure 1: Time Windows</small></i>
+</center>
+<br/>
+
+
+One of the common key requirements for a fraud detection system is *low
+response time*. The sooner the fraudulent action gets detected, the
+higher the chances that it can be blocked and its negative consequences
+mitigated. This requirement is especially prominent in the financial
+domain, where you have one important constraint - any time spent
+evaluating a fraud detection model is time that a law-abiding user of
+your system will spend waiting for a response. Swiftness of processing
+often becomes a competitive advantage between various payment systems
+and the time limit for producing an alert could lie as low as *300-500
+ms*. This is all the time you get from the moment of ingestion of a
+transaction event into a fraud detection system until an alert has to
+become available to downstream systems.
+
+As you might know, Flink provides a powerful [Window
+API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html)
+that is applicable for a wide range of use cases. However, If you go
+over all of the available types of supported windows, you will realize
+that none of them exactly match our main requirement for this use case -
+the low-latency evaluation of *each* incoming transaction. There is
+no type of window in Flink that can express the *"x minutes/hours/days
+back from the <u>current event</u>"* semantic. In the Window API, events
+fall into windows (as defined by the window
+[assigners](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#window-assigners)),
+but they cannot themselves individually control the creation and
+evaluation of windows\*. As described above, our goal for the fraud
+detection engine is to achieve immediate evaluation of the previous
+relevant data points as soon as the new event is received. This raises
+the question of feasibility of applying the Window API in this case.
Review comment:
I guess, that is exactly what I proposed to add (that there is something but that's not really feasible).
Something around the line of
> The window API offers some customization options for defining custom triggers, evictors, and window assigners which may get to a similar result. However, it is usually difficult to get this right (and easy to break) while at the same time, business logic based on a `ProcessFunction` is easier to understand and maintain and offers access to broadcast state as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] NicoK closed pull request #362: Add Blogbost: Advanced Flink Application Patterns: Custom Window Proc…
Posted by GitBox <gi...@apache.org>.
NicoK closed pull request #362:
URL: https://github.com/apache/flink-web/pull/362
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org