You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by ConradJam <ja...@gmail.com> on 2023/08/24 05:59:50 UTC

[Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Hi Community

I want to re-initiate the discussion related to FLIP-202. This is a
discussion related to the Clickhouse connector. Some previous discussions
are summarized in [1]

● We need an external code warehouse to store this part of code, as
@Martijn Visser said flink-clickhouse-connector to create
● Discuss the current Flink write or sink and clickhouse need to solve the
common problems of users


Now I want to redesign this part of the function. The relevant FLIP will be
updated after collecting opinions. Welcome to join the discussion

[1] FLINK-26999 <https://issues.apache.org/jira/browse/FLINK-26999> :Introduce
ClickHouse Connector


-- 
Best

ConradJam

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Posted by ConradJam <ja...@gmail.com>.
Thanks to @Martijn @Leonard for raising this question

This is also something I have been worried about, that is, the issue of
code copyright ownership and cooperation. That's why I mark FLIP-202 [1] as
draft.
I have sent a private email inviting the author to participate. If there is
more news, I will synchronize it in time. I believe that through hard work,
we will definitely win opportunities for cooperation.

[1] FLIP-202
https://cwiki.apache.org/confluence/display/FLINK/%5BDRAFT%5D+FLIP-202%3A+Introduce+ClickHouse+Connector

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Posted by Leonard Xu <xb...@gmail.com>.
I share same concern with Matijn, how about invite the author of unofficial flink-clickhouse-connector itinycheng to co-author the FLIP? @ConradJam

And then user would like to use the new official connector from users’ perspective, the maintaining work of unofficial CK connector and the new CK connector could be centralized to official repo(apache/flink-clickhouse-connector) , I feel  this approach should be the more proper  cooperation way in open source community. 

Best,
Leonard



> On Sep 20, 2023, at 8:51 PM, Martijn Visser <ma...@apache.org> wrote:
> 
> Hi ConradJam,
> 
> The FLIP still references the unofficial "flink-clickhouse-connector"
> which I don't really understand: you want to build a new
> implementation for this connector, right?
> 
> Best regards,
> 
> Martijn
> 
> On Thu, Sep 7, 2023 at 12:12 PM ConradJam <ja...@gmail.com> wrote:
>> 
>> Does anyone else have an opinion on this part? If not I will start
>> voting.Comment collection will be open again


Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Posted by Martijn Visser <ma...@apache.org>.
Hi Conrad,

I think we need that confirmation from the original author as part of
this discussion thread.

Best regards,

Martijn

On Tue, Oct 3, 2023 at 4:13 AM ConradJam <ja...@gmail.com> wrote:
>
> After email communication between me and the author of the warehouse, he
> has agreed to donate to Apache Flink and participate in the design of the
> solution.
>
> @Martlin @Leonard
> Yes, it can be said that for clickhouse's connector solution [1] we still
> refer to part of the code and implementation of the existing clickhouse
> connector [2]. To be precise, we should replace some outdated
> implementations. During this process, the author and I hope to make the
> following improvements based on the current warehouse after synchronizing
> information via email:
> ● ClickHouse JDBC is updated to the latest (0.4.6[3] is stable and
> available as of the email deadline)
> ● Support bool type, delete statement, grpc, tcp
> ● Implement new Clickhouse Sink Api (to be confirmed)
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/%5BDRAFT%5D+FLIP-202
> %3A+Introduce+ClickHouse+Connector
> [2] https://github.com/itinycheng/flink-connector-clickhouse
> [3] https://github.com/ClickHouse/clickhouse-java/releases
>
>
>
> Martijn Visser <ma...@apache.org> 于2023年9月20日周三 21:20写道:
>
> > Hi ConradJam,
> >
> > The FLIP still references the unofficial "flink-clickhouse-connector"
> > which I don't really understand: you want to build a new
> > implementation for this connector, right?
> >
> > Best regards,
> >
> > Martijn
> >
> > On Thu, Sep 7, 2023 at 12:12 PM ConradJam <ja...@gmail.com> wrote:
> > >
> > > Does anyone else have an opinion on this part? If not I will start
> > > voting.Comment collection will be open again
> >
>
>
> --
> Best
>
> ConradJam

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Posted by ConradJam <ja...@gmail.com>.
After email communication between me and the author of the warehouse, he
has agreed to donate to Apache Flink and participate in the design of the
solution.

@Martlin @Leonard
Yes, it can be said that for clickhouse's connector solution [1] we still
refer to part of the code and implementation of the existing clickhouse
connector [2]. To be precise, we should replace some outdated
implementations. During this process, the author and I hope to make the
following improvements based on the current warehouse after synchronizing
information via email:
● ClickHouse JDBC is updated to the latest (0.4.6[3] is stable and
available as of the email deadline)
● Support bool type, delete statement, grpc, tcp
● Implement new Clickhouse Sink Api (to be confirmed)

[1] https://cwiki.apache.org/confluence/display/FLINK/%5BDRAFT%5D+FLIP-202
%3A+Introduce+ClickHouse+Connector
[2] https://github.com/itinycheng/flink-connector-clickhouse
[3] https://github.com/ClickHouse/clickhouse-java/releases



Martijn Visser <ma...@apache.org> 于2023年9月20日周三 21:20写道:

> Hi ConradJam,
>
> The FLIP still references the unofficial "flink-clickhouse-connector"
> which I don't really understand: you want to build a new
> implementation for this connector, right?
>
> Best regards,
>
> Martijn
>
> On Thu, Sep 7, 2023 at 12:12 PM ConradJam <ja...@gmail.com> wrote:
> >
> > Does anyone else have an opinion on this part? If not I will start
> > voting.Comment collection will be open again
>


-- 
Best

ConradJam

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Posted by Martijn Visser <ma...@apache.org>.
Hi ConradJam,

The FLIP still references the unofficial "flink-clickhouse-connector"
which I don't really understand: you want to build a new
implementation for this connector, right?

Best regards,

Martijn

On Thu, Sep 7, 2023 at 12:12 PM ConradJam <ja...@gmail.com> wrote:
>
> Does anyone else have an opinion on this part? If not I will start
> voting.Comment collection will be open again

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Posted by ConradJam <ja...@gmail.com>.
Does anyone else have an opinion on this part? If not I will start
voting.Comment collection will be open again

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Posted by ConradJam <ja...@gmail.com>.
RocMarshal mentioned the new connector API in the FLINK-26999 [1] , I
wonder if this refers to Sink V2 Api [2] Or other things?  If someone can
help me troubleshoot this issue it will speed up my design progress


   - [1] FLINK-26999: https://issues.apache.org/jira/browse/FLINK-26999
   - [2] Sink V2:
   https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/connector/sink/Sink.html


Jing Ge <ji...@ververica.com.invalid> 于2023年8月24日周四 14:53写道:

> Hi Conrad,
>
> Thanks for driving it! +1 for starting a new round of discussion based on
> the input you wrote in the comment. Look forward to the new design.
>
> Best regards,
> Jing
>
> On Thu, Aug 24, 2023 at 8:00 AM ConradJam <ja...@gmail.com> wrote:
>
> > Hi Community
> >
> > I want to re-initiate the discussion related to FLIP-202. This is a
> > discussion related to the Clickhouse connector. Some previous discussions
> > are summarized in [1]
> >
> > ● We need an external code warehouse to store this part of code, as
> > @Martijn Visser said flink-clickhouse-connector to create
> > ● Discuss the current Flink write or sink and clickhouse need to solve
> the
> > common problems of users
> >
> >
> > Now I want to redesign this part of the function. The relevant FLIP will
> be
> > updated after collecting opinions. Welcome to join the discussion
> >
> > [1] FLINK-26999 <https://issues.apache.org/jira/browse/FLINK-26999>
> > :Introduce
> > ClickHouse Connector
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>


-- 
Best

ConradJam

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Posted by ConradJam <ja...@gmail.com>.
Thanks for all comments
Now I have updated FLIP-202 design (draft) related design and information

Let me answer the questions mentioned in the previous mailing list and
issuse

@Sergey

1、Async sink If you want to implement this function, it seems that this
part cannot be implemented using the JDBC protocol. I have described it at
the end of FLIP-202 [1]. At present, I am not very clear about this
function. If you can assist in the investigation of the problem, it will be
very grateful

2、Why reject the existing flink-clickhouse-Connecto r[2] to answer at the
end of FLIP-202


[1]
https://cwiki.apache.org/confluence/display/FLINK/%5BDRAFT%5D+FLIP-202%3A+Introduce+ClickHouse+Connector
[2] https://github.com/itinycheng/flink-connector-clickhouse




Sergey Nuyanzin <sn...@gmail.com> 于2023年8月24日周四 15:24写道:

> Hi Conrad,
>
> Thanks for driving it.
>
> In general I'm +1
> however I have a couple of questions
>
> 1. From FLIP's description it is not clear whether this connector will
> support Async sink [1] or not, could you please clarify it?
> 2. Also may be a bit more details in FLIP's page would helpful  why a new
> connector required instead of existing ClickHouse connector [2]
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> [2] https://github.com/itinycheng/flink-connector-clickhouse
>
> On Thu, Aug 24, 2023 at 8:53 AM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > Hi Conrad,
> >
> > Thanks for driving it! +1 for starting a new round of discussion based on
> > the input you wrote in the comment. Look forward to the new design.
> >
> > Best regards,
> > Jing
> >
> > On Thu, Aug 24, 2023 at 8:00 AM ConradJam <ja...@gmail.com> wrote:
> >
> > > Hi Community
> > >
> > > I want to re-initiate the discussion related to FLIP-202. This is a
> > > discussion related to the Clickhouse connector. Some previous
> discussions
> > > are summarized in [1]
> > >
> > > ● We need an external code warehouse to store this part of code, as
> > > @Martijn Visser said flink-clickhouse-connector to create
> > > ● Discuss the current Flink write or sink and clickhouse need to solve
> > the
> > > common problems of users
> > >
> > >
> > > Now I want to redesign this part of the function. The relevant FLIP
> will
> > be
> > > updated after collecting opinions. Welcome to join the discussion
> > >
> > > [1] FLINK-26999 <https://issues.apache.org/jira/browse/FLINK-26999>
> > > :Introduce
> > > ClickHouse Connector
> > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> >
>
>
> --
> Best regards,
> Sergey
>


-- 
Best

ConradJam

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Posted by Sergey Nuyanzin <sn...@gmail.com>.
Hi Conrad,

Thanks for driving it.

In general I'm +1
however I have a couple of questions

1. From FLIP's description it is not clear whether this connector will
support Async sink [1] or not, could you please clarify it?
2. Also may be a bit more details in FLIP's page would helpful  why a new
connector required instead of existing ClickHouse connector [2]

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
[2] https://github.com/itinycheng/flink-connector-clickhouse

On Thu, Aug 24, 2023 at 8:53 AM Jing Ge <ji...@ververica.com.invalid> wrote:

> Hi Conrad,
>
> Thanks for driving it! +1 for starting a new round of discussion based on
> the input you wrote in the comment. Look forward to the new design.
>
> Best regards,
> Jing
>
> On Thu, Aug 24, 2023 at 8:00 AM ConradJam <ja...@gmail.com> wrote:
>
> > Hi Community
> >
> > I want to re-initiate the discussion related to FLIP-202. This is a
> > discussion related to the Clickhouse connector. Some previous discussions
> > are summarized in [1]
> >
> > ● We need an external code warehouse to store this part of code, as
> > @Martijn Visser said flink-clickhouse-connector to create
> > ● Discuss the current Flink write or sink and clickhouse need to solve
> the
> > common problems of users
> >
> >
> > Now I want to redesign this part of the function. The relevant FLIP will
> be
> > updated after collecting opinions. Welcome to join the discussion
> >
> > [1] FLINK-26999 <https://issues.apache.org/jira/browse/FLINK-26999>
> > :Introduce
> > ClickHouse Connector
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>


-- 
Best regards,
Sergey

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Conrad,

Thanks for driving it! +1 for starting a new round of discussion based on
the input you wrote in the comment. Look forward to the new design.

Best regards,
Jing

On Thu, Aug 24, 2023 at 8:00 AM ConradJam <ja...@gmail.com> wrote:

> Hi Community
>
> I want to re-initiate the discussion related to FLIP-202. This is a
> discussion related to the Clickhouse connector. Some previous discussions
> are summarized in [1]
>
> ● We need an external code warehouse to store this part of code, as
> @Martijn Visser said flink-clickhouse-connector to create
> ● Discuss the current Flink write or sink and clickhouse need to solve the
> common problems of users
>
>
> Now I want to redesign this part of the function. The relevant FLIP will be
> updated after collecting opinions. Welcome to join the discussion
>
> [1] FLINK-26999 <https://issues.apache.org/jira/browse/FLINK-26999>
> :Introduce
> ClickHouse Connector
>
>
> --
> Best
>
> ConradJam
>