You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Osada Paranaliyanage <Os...@dialog.lk> on 2021/09/14 12:56:19 UTC
Streaming SQL support for redis streaming connector
Hi All, We are looking to use flink to build a materialized view of a relation db and a document db using cdc streams. For this purpose we would like to use redis for hosting the materialized view. Can we do this in streaming SQL? We have worked through https://github.com/ververica/flink-sql-CDC and can see how this will work with ES as a sink. But can we use redis as the sink? Where do we find the syntax for that?
Thanks,
Osada.
________________________________
This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
RE: Streaming SQL support for redis streaming connector
Posted by Osada Paranaliyanage <Os...@dialog.lk>.
Thanks, will have a look through!
-----Original Message-----
From: Yangze Guo <ka...@gmail.com>
Sent: Wednesday, September 15, 2021 11:25 AM
To: Osada Paranaliyanage <Os...@dialog.lk>
Cc: David Morávek <dm...@apache.org>; user@flink.apache.org
Subject: Re: Streaming SQL support for redis streaming connector
[EXTERNAL EMAIL] This email has been received from an external source – please review before actioning, clicking on links, or opening attachments.
> What about the bahir streaming connectors? Are they considered canonical? Would they be merged in to the main project at some point? Iiuc we can use table API etc to write data to redis using that right?
I think it still can be used with Flink 1.12 and 1.13. However, it has not been updated for 5 months and is not implemented based on the new interfaces introduced in FLIP-95. AFAIK, the community does not plan to merge it.
> What more will be required for us to use SQL? (as in specify the
> connector in the WITH clause as redis)
To use the bahir with SQL, you need to build the project and move the uber jar to the lib directory of your flink dist and fill in necessary properties[1][2].
[1] https://github.com/apache/bahir-flink/blob/master/flink-connector-redis/src/main/java/org/apache/flink/streaming/connectors/redis/RedisTableSinkFactory.java
[2] https://github.com/apache/bahir-flink/blob/master/flink-connector-redis/src/main/java/org/apache/flink/streaming/connectors/redis/descriptor/RedisValidator.java
Best,
Yangze Guo
On Wed, Sep 15, 2021 at 11:34 AM Osada Paranaliyanage <Os...@dialog.lk> wrote:
>
> Hi David,
>
>
>
> What about the bahir streaming connectors? Are they considered
> canonical? Would they be merged in to the main project at some point?
> Iiuc we can use table API etc to write data to redis using that right?
> What more will be required for us to use SQL? (as in specify the
> connector in the WITH clause as redis)
>
>
>
> Thanks,
>
> Osada.
>
>
>
> From: David Morávek <dm...@apache.org>
> Sent: Tuesday, September 14, 2021 7:53 PM
> To: Osada Paranaliyanage <Os...@dialog.lk>
> Cc: user@flink.apache.org
> Subject: Re: Streaming SQL support for redis streaming connector
>
>
>
> [EXTERNAL EMAIL] This email has been received from an external source – please review before actioning, clicking on links, or opening attachments.
>
> Hi Osada,
>
>
>
> in theory building a Redis table from "CDC stream" should definitely be doable. Unfortunately Flink currently doesn't have any official Redis Sink for the Table API and there is currently no on-going effort for adding it, so it would need to be implemented first. The resulting syntax would be pretty much the same to what's outlined in the mentioned example.
>
>
>
> Best,
>
> D.
>
>
>
> On Tue, Sep 14, 2021 at 2:57 PM Osada Paranaliyanage <Os...@dialog.lk> wrote:
>
> Hi All, We are looking to use flink to build a materialized view of a relation db and a document db using cdc streams. For this purpose we would like to use redis for hosting the materialized view. Can we do this in streaming SQL? We have worked through https://github.com/ververica/flink-sql-CDC and can see how this will work with ES as a sink. But can we use redis as the sink? Where do we find the syntax for that?
>
>
>
> Thanks,
>
> Osada.
>
>
>
>
>
> ________________________________
>
>
>
> This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
>
>
> ________________________________
>
>
> This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
>
________________________________
This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
Re: Streaming SQL support for redis streaming connector
Posted by Yangze Guo <ka...@gmail.com>.
> What about the bahir streaming connectors? Are they considered canonical? Would they be merged in to the main project at some point? Iiuc we can use table API etc to write data to redis using that right?
I think it still can be used with Flink 1.12 and 1.13. However, it has
not been updated for 5 months and is not implemented based on the new
interfaces introduced in FLIP-95. AFAIK, the community does not plan
to merge it.
> What more will be required for us to use SQL? (as in specify the connector in the WITH clause as redis)
To use the bahir with SQL, you need to build the project and move the
uber jar to the lib directory of your flink dist and fill in necessary
properties[1][2].
[1] https://github.com/apache/bahir-flink/blob/master/flink-connector-redis/src/main/java/org/apache/flink/streaming/connectors/redis/RedisTableSinkFactory.java
[2] https://github.com/apache/bahir-flink/blob/master/flink-connector-redis/src/main/java/org/apache/flink/streaming/connectors/redis/descriptor/RedisValidator.java
Best,
Yangze Guo
On Wed, Sep 15, 2021 at 11:34 AM Osada Paranaliyanage
<Os...@dialog.lk> wrote:
>
> Hi David,
>
>
>
> What about the bahir streaming connectors? Are they considered canonical? Would they be merged in to the main project at some point? Iiuc we can use table API etc to write data to redis using that right? What more will be required for us to use SQL? (as in specify the connector in the WITH clause as redis)
>
>
>
> Thanks,
>
> Osada.
>
>
>
> From: David Morávek <dm...@apache.org>
> Sent: Tuesday, September 14, 2021 7:53 PM
> To: Osada Paranaliyanage <Os...@dialog.lk>
> Cc: user@flink.apache.org
> Subject: Re: Streaming SQL support for redis streaming connector
>
>
>
> [EXTERNAL EMAIL] This email has been received from an external source – please review before actioning, clicking on links, or opening attachments.
>
> Hi Osada,
>
>
>
> in theory building a Redis table from "CDC stream" should definitely be doable. Unfortunately Flink currently doesn't have any official Redis Sink for the Table API and there is currently no on-going effort for adding it, so it would need to be implemented first. The resulting syntax would be pretty much the same to what's outlined in the mentioned example.
>
>
>
> Best,
>
> D.
>
>
>
> On Tue, Sep 14, 2021 at 2:57 PM Osada Paranaliyanage <Os...@dialog.lk> wrote:
>
> Hi All, We are looking to use flink to build a materialized view of a relation db and a document db using cdc streams. For this purpose we would like to use redis for hosting the materialized view. Can we do this in streaming SQL? We have worked through https://github.com/ververica/flink-sql-CDC and can see how this will work with ES as a sink. But can we use redis as the sink? Where do we find the syntax for that?
>
>
>
> Thanks,
>
> Osada.
>
>
>
>
>
> ________________________________
>
>
>
> This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
>
>
> ________________________________
>
>
> This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
>
RE: Streaming SQL support for redis streaming connector
Posted by Osada Paranaliyanage <Os...@dialog.lk>.
Hi David,
What about the bahir streaming connectors? Are they considered canonical? Would they be merged in to the main project at some point? Iiuc we can use table API etc to write data to redis using that right? What more will be required for us to use SQL? (as in specify the connector in the WITH clause as redis)
Thanks,
Osada.
From: David Morávek <dm...@apache.org>
Sent: Tuesday, September 14, 2021 7:53 PM
To: Osada Paranaliyanage <Os...@dialog.lk>
Cc: user@flink.apache.org
Subject: Re: Streaming SQL support for redis streaming connector
[EXTERNAL EMAIL] This email has been received from an external source – please review before actioning, clicking on links, or opening attachments.
Hi Osada,
in theory building a Redis table from "CDC stream" should definitely be doable. Unfortunately Flink currently doesn't have any official Redis Sink for the Table API and there is currently no on-going effort for adding it, so it would need to be implemented first. The resulting syntax would be pretty much the same to what's outlined in the mentioned example.
Best,
D.
On Tue, Sep 14, 2021 at 2:57 PM Osada Paranaliyanage <Os...@dialog.lk>> wrote:
Hi All, We are looking to use flink to build a materialized view of a relation db and a document db using cdc streams. For this purpose we would like to use redis for hosting the materialized view. Can we do this in streaming SQL? We have worked through https://github.com/ververica/flink-sql-CDC and can see how this will work with ES as a sink. But can we use redis as the sink? Where do we find the syntax for that?
Thanks,
Osada.
________________________________
This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
________________________________
This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
Re: Streaming SQL support for redis streaming connector
Posted by David Morávek <dm...@apache.org>.
Hi Osada,
in theory building a Redis table from "CDC stream" should definitely be
doable. Unfortunately Flink currently doesn't have any official Redis Sink
for the Table API and there is currently no on-going effort for adding it,
so it would need to be implemented first. The resulting syntax would be
pretty much the same to what's outlined in the mentioned example.
Best,
D.
On Tue, Sep 14, 2021 at 2:57 PM Osada Paranaliyanage <
Osada.Paranaliyanage@dialog.lk> wrote:
> Hi All, We are looking to use flink to build a materialized view of a
> relation db and a document db using cdc streams. For this purpose we would
> like to use redis for hosting the materialized view. Can we do this in
> streaming SQL? We have worked through
> https://github.com/ververica/flink-sql-CDC and can see how this will work
> with ES as a sink. But can we use redis as the sink? Where do we find the
> syntax for that?
>
>
>
> Thanks,
>
> Osada.
>
>
>
> ------------------------------
>
>
> This e-mail is confidential. It may also be legally privileged. If you are
> not the intended recipient or have received it in error, please delete it
> and all copies from your system and notify the sender immediately by return
> e-mail. Any unauthorized reading, reproducing, printing or further
> dissemination of this e-mail or its contents is strictly prohibited and may
> be unlawful. Internet communications cannot be guaranteed to be timely,
> secure, error or virus-free. The sender does not accept liability for any
> errors or omissions.
>
>
RE: Streaming SQL support for redis streaming connector
Posted by Osada Paranaliyanage <Os...@dialog.lk>.
Hi Leonard,
That’s awesome news. We are actually using documentdb. Any idea how much work it will be to make it work with documentdb instead?
Thanks,
Osada.
From: Leonard Xu <xb...@gmail.com>
Sent: Wednesday, September 15, 2021 1:08 PM
To: Osada Paranaliyanage <Os...@dialog.lk>
Cc: user@flink.apache.org
Subject: Re: Streaming SQL support for redis streaming connector
[EXTERNAL EMAIL] This email has been received from an external source - please review before actioning, clicking on links, or opening attachments.
Hi, Osada
Just want to offer some material here.The flink-cdc-connectors project [1] maybe also help you, we supports the document db MongoDB[2] recently.
Best,
Leonard
[1] https://github.com/ververica/flink-cdc-connectors
[2] https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mongodb-cdc.html
在 2021年9月14日，20:56，Osada Paranaliyanage <Os...@dialog.lk>> 写道：
Hi All, We are looking to use flink to build a materialized view of a relation db and a document db using cdc streams. For this purpose we would like to use redis for hosting the materialized view. Can we do this in streaming SQL? We have worked through https://github.com/ververica/flink-sql-CDC and can see how this will work with ES as a sink. But can we use redis as the sink? Where do we find the syntax for that?
Thanks,
Osada.
________________________________
This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
________________________________
This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
Re: Streaming SQL support for redis streaming connector
Posted by Leonard Xu <xb...@gmail.com>.
Hi, Osada
Just want to offer some material here.The flink-cdc-connectors project [1] maybe also help you, we supports the document db MongoDB[2] recently.
Best,
Leonard
[1] https://github.com/ververica/flink-cdc-connectors <https://github.com/ververica/flink-cdc-connectors>
[2] https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mongodb-cdc.html <https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mongodb-cdc.html>
> 在 2021年9月14日，20:56，Osada Paranaliyanage <Os...@dialog.lk> 写道：
>
> Hi All, We are looking to use flink to build a materialized view of a relation db and a document db using cdc streams. For this purpose we would like to use redis for hosting the materialized view. Can we do this in streaming SQL? We have worked through https://github.com/ververica/flink-sql-CDC <https://github.com/ververica/flink-sql-CDC> and can see how this will work with ES as a sink. But can we use redis as the sink? Where do we find the syntax for that?
>
> Thanks,
> Osada.
>
>
>
>
> This e-mail is confidential. It may also be legally privileged. If you are not the intended recipient or have received it in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Any unauthorized reading, reproducing, printing or further dissemination of this e-mail or its contents is strictly prohibited and may be unlawful. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.