You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Jinsui Chen <is...@gmail.com> on 2023/12/19 17:24:05 UTC
[DISCUSS] FLIP-404: Create a Redis HyperLogLog Connector for Flink
Hi, there
We would like to start a discussion thread on "FLIP-404: Create a Redis
HyperLogLog Connector for Flink"[1].
There has also been a lot of discussion in the past about Redis connectors,
including creating lookup and sink connectors for generic key-value get and
put, and creating source and sink connectors for streams structures.
Different data structures are used in different ways. I think we can build
generic connectors based on KVs and create special-purpose connectors for
specific data structures.
For example, for list structure, end-user can specify the data to be placed
or get on the left or right side of the list to build the sink or source
connector, and for hyperloglog, end-user needs to write the data to
multiple keys by the content of the table field.
For the differentiated usage of different data structures, a unified
connector may be difficult to maintain due to too much capacity.
Based on above, this FLIP is proposed.
Looking forward to any comment or feedback.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-404%3A+Create+a+Redis+HyperLogLog+Connector+for+Flink
Best regards,
Jinsui
Re: [DISCUSS] FLIP-404: Create a Redis HyperLogLog Connector for Flink
Posted by Jinsui Chen <is...@gmail.com>.
Hi, Martijn
Thank you for your input. I align with your viewpoint and am fully on board
with the idea. I'll continue contributing and provide assistance wherever
possible.
Best regards,
Jinsui
Martijn Visser <ma...@apache.org> 于2023年12月27日周三 23:56写道：
> Hi Jinsui,
>
> Thanks for opening the discussion and creating the FLIP. My main
> concern isn't so much the connector, it's more that we're talking
> about yet another Redis connector implementation but we actually don't
> have a proper Redis connector overall. Ideally it would be great if we
> could get https://github.com/apache/flink-connector-redis-streams/ in
> proper shape (perhaps you could help with some reviews) and then
> moving all Redis connectors into one mono repo, kind of how it's done
> for AWS connectors.
>
> Best regards,
>
> Martijn
>
> On Tue, Dec 19, 2023 at 6:25 PM Jinsui Chen <is...@gmail.com>
> wrote:
> >
> > Hi, there
> >
> > We would like to start a discussion thread on "FLIP-404: Create a Redis
> > HyperLogLog Connector for Flink"[1].
> >
> > There has also been a lot of discussion in the past about Redis
> connectors,
> > including creating lookup and sink connectors for generic key-value get
> and
> > put, and creating source and sink connectors for streams structures.
> >
> > Different data structures are used in different ways. I think we can
> build
> > generic connectors based on KVs and create special-purpose connectors for
> > specific data structures.
> >
> > For example, for list structure, end-user can specify the data to be
> placed
> > or get on the left or right side of the list to build the sink or source
> > connector, and for hyperloglog, end-user needs to write the data to
> > multiple keys by the content of the table field.
> >
> > For the differentiated usage of different data structures, a unified
> > connector may be difficult to maintain due to too much capacity.
> >
> > Based on above, this FLIP is proposed.
> >
> > Looking forward to any comment or feedback.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-404%3A+Create+a+Redis+HyperLogLog+Connector+for+Flink
> >
> > Best regards,
> > Jinsui
>
Re: [DISCUSS] FLIP-404: Create a Redis HyperLogLog Connector for Flink
Posted by Martijn Visser <ma...@apache.org>.
Hi Jinsui,
Thanks for opening the discussion and creating the FLIP. My main
concern isn't so much the connector, it's more that we're talking
about yet another Redis connector implementation but we actually don't
have a proper Redis connector overall. Ideally it would be great if we
could get https://github.com/apache/flink-connector-redis-streams/ in
proper shape (perhaps you could help with some reviews) and then
moving all Redis connectors into one mono repo, kind of how it's done
for AWS connectors.
Best regards,
Martijn
On Tue, Dec 19, 2023 at 6:25 PM Jinsui Chen <is...@gmail.com> wrote:
>
> Hi, there
>
> We would like to start a discussion thread on "FLIP-404: Create a Redis
> HyperLogLog Connector for Flink"[1].
>
> There has also been a lot of discussion in the past about Redis connectors,
> including creating lookup and sink connectors for generic key-value get and
> put, and creating source and sink connectors for streams structures.
>
> Different data structures are used in different ways. I think we can build
> generic connectors based on KVs and create special-purpose connectors for
> specific data structures.
>
> For example, for list structure, end-user can specify the data to be placed
> or get on the left or right side of the list to build the sink or source
> connector, and for hyperloglog, end-user needs to write the data to
> multiple keys by the content of the table field.
>
> For the differentiated usage of different data structures, a unified
> connector may be difficult to maintain due to too much capacity.
>
> Based on above, this FLIP is proposed.
>
> Looking forward to any comment or feedback.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-404%3A+Create+a+Redis+HyperLogLog+Connector+for+Flink
>
> Best regards,
> Jinsui