Flink Table API and Rules

[Shahdat Hossain <sh...@gmail.com>]

I am trying to match streaming data against my database table
using Flink Table API. I am able to match the data that exists in my
streaming data and the specific column of the database table.  Now I need
to activate some rules that users have entered in the system (another
database table) and I want to evaluate the rules. What is the best
approach? I have spent a lot of time trying to figure it out within the
Table API but I simply can't seem to connect the dots. Is there another API
that I should use to achieve this?

Example of what I am trying to do: Multiple data from students coming in
via Kafka source. Student A, B, C, D and E data comes in with their grade
of 70, 80, 90, 95 and 98. I have a rule table that has let's say only 3
rows for rule 1, 2 and 3. rule 1 says "if student C gets more than 85 send
a thank you note." Rule 2 says "if student D gets more than 92 send a thank
you note." Rule 3 says "if student E gets more than 96 send out a thank you
note." Now I can use Table API to do a join between the streaming data and
the table to only get C, D and E data. As soon as I get the match for C, I
need to look into the streaming data to get the student's score, compare
that with the rule, and need to execute the rule since in this case student
C did get more than 85.

Your help is much appreciated.

Regards,
Shahdat Hossain

Re: Flink Table API and Rules

[Roman Khachatryan <ro...@apache.org>]

Hi,

Could you clarify whether your question is about
- connecting Table and DataStream APIs? [1]
- matching the records using the Table API?
- or matching in the DataStream API? [3]
...

You should probably look at the temporal joins to match records using
Table API [2]; and connect/join/ or broadcast for the DataStream API
[3]. The choice in the latter case depends on the matching logic.

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/data_stream_api/#converting-between-datastream-and-table
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#event-time-temporal-join
[3]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/

Regards,
Roman

On Tue, Feb 8, 2022 at 8:26 PM Shahdat Hossain <sh...@gmail.com> wrote:
>
> I am trying to match streaming data against my database table using Flink Table API. I am able to match the data that exists in my streaming data and the specific column of the database table.  Now I need to activate some rules that users have entered in the system (another database table) and I want to evaluate the rules. What is the best approach? I have spent a lot of time trying to figure it out within the Table API but I simply can't seem to connect the dots. Is there another API that I should use to achieve this?
>
> Example of what I am trying to do: Multiple data from students coming in via Kafka source. Student A, B, C, D and E data comes in with their grade of 70, 80, 90, 95 and 98. I have a rule table that has let's say only 3 rows for rule 1, 2 and 3. rule 1 says "if student C gets more than 85 send a thank you note." Rule 2 says "if student D gets more than 92 send a thank you note." Rule 3 says "if student E gets more than 96 send out a thank you note." Now I can use Table API to do a join between the streaming data and the table to only get C, D and E data. As soon as I get the match for C, I need to look into the streaming data to get the student's score, compare that with the rule, and need to execute the rule since in this case student C did get more than 85.
>
> Your help is much appreciated.
>
> Regards,
> Shahdat Hossain
>