You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ran Tao <ch...@gmail.com> on 2022/05/07 03:59:16 UTC
[DISCUSS ] HybridSouce Table & Sql api timeline
HybridSource is a good feature, but now release version did not support
table & sql api, i wonder which time it will be ready to end-users.
And i have implemented a inner version of my company and it works well now.
the implementation of table & sql api may involve some core questions, e.g.
bounded & unbounded source start and end offset. child sources's schema is
different from hybrid source ddl schema (batch or streaming is more fields
or lack of some fields) we need process inconsistent filed problem or
default filed value problem (in some child source lack of some fields) etc.
so here we may need a field mapping.
i have some ideas and implementations, if table & sql api work in progress
i'm glad to share or take part in developing.
thanks~
Re: [DISCUSS ] HybridSouce Table & Sql api timeline
Posted by Ran Tao <ch...@gmail.com>.
Thanks, Thomas.
I noticed this issue and asked Jiang for details.
If he doesn't finish this work yet, I will draft the initial FLIP,
otherwise we can wait for Jiang's implementation.
Thomas Weise <th...@apache.org> 于2022年5月11日周三 02:29写道：
> fyi there is a related ticket here:
> https://issues.apache.org/jira/browse/FLINK-22793
>
> On Mon, May 9, 2022 at 11:34 PM Becket Qin <be...@gmail.com> wrote:
> >
> > Cool, I've granted you the permission.
> >
> > Cheers,
> >
> > Jiangjie (Becket) Qin
> >
> > On Tue, May 10, 2022 at 1:14 PM Ran Tao <ch...@gmail.com> wrote:
> >
> > > Hi, Becket. Thanks for your suggestions. My id is: Ran Tao
> > > And i will draft this flip in a few days. thanks~
> > >
> > > Becket Qin <be...@gmail.com> 于2022年5月10日周二 12:40写道：
> > >
> > > > Hi Ran,
> > > >
> > > > The FLIP process can be found here[1].
> > > >
> > > > You don't need to pass the vote, in fact the vote is based on the
> FLIP
> > > > wiki. So drafting the FLIP wiki would be the first step. After that
> you
> > > may
> > > > start a discussion thread in the mailing list so people can have the
> > > > discussion about the feature based on your FLIP wiki. Note that it is
> > > very
> > > > important to follow the structure of the FLIP template so the
> discussion
> > > > would be more efficient.
> > > >
> > > > In case you don't have the permission to add a FLIP page yet, please
> let
> > > me
> > > > know your Apache Confluence ID so I can grant you the permission.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > [1]
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > >
> > > > On Tue, May 10, 2022 at 12:06 PM Ran Tao <ch...@gmail.com>
> wrote:
> > > >
> > > > > Hi, Martijn, Jacky. Thanks for your responding. It indeed need a
> > > designed
> > > > > doc or FLIP to illustrate some details and concrete implementation.
> > > > >
> > > > > And i'm glad to work on this issue. I wonder whether i can create a
> > > FLIP
> > > > > under discussion firstly
> > > > > to write the draft design of the implementation about table & sql
> api
> > > and
> > > > > some details, or we must pass voting?
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Ran Tao
> > >
>
--
Best,
Ran Tao
Re: [DISCUSS ] HybridSouce Table & Sql api timeline
Posted by Thomas Weise <th...@apache.org>.
fyi there is a related ticket here:
https://issues.apache.org/jira/browse/FLINK-22793
On Mon, May 9, 2022 at 11:34 PM Becket Qin <be...@gmail.com> wrote:
>
> Cool, I've granted you the permission.
>
> Cheers,
>
> Jiangjie (Becket) Qin
>
> On Tue, May 10, 2022 at 1:14 PM Ran Tao <ch...@gmail.com> wrote:
>
> > Hi, Becket. Thanks for your suggestions. My id is: Ran Tao
> > And i will draft this flip in a few days. thanks~
> >
> > Becket Qin <be...@gmail.com> 于2022年5月10日周二 12:40写道：
> >
> > > Hi Ran,
> > >
> > > The FLIP process can be found here[1].
> > >
> > > You don't need to pass the vote, in fact the vote is based on the FLIP
> > > wiki. So drafting the FLIP wiki would be the first step. After that you
> > may
> > > start a discussion thread in the mailing list so people can have the
> > > discussion about the feature based on your FLIP wiki. Note that it is
> > very
> > > important to follow the structure of the FLIP template so the discussion
> > > would be more efficient.
> > >
> > > In case you don't have the permission to add a FLIP page yet, please let
> > me
> > > know your Apache Confluence ID so I can grant you the permission.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > [1]
> > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >
> > > On Tue, May 10, 2022 at 12:06 PM Ran Tao <ch...@gmail.com> wrote:
> > >
> > > > Hi, Martijn, Jacky. Thanks for your responding. It indeed need a
> > designed
> > > > doc or FLIP to illustrate some details and concrete implementation.
> > > >
> > > > And i'm glad to work on this issue. I wonder whether i can create a
> > FLIP
> > > > under discussion firstly
> > > > to write the draft design of the implementation about table & sql api
> > and
> > > > some details, or we must pass voting?
> > > >
> > >
> >
> >
> > --
> > Best,
> > Ran Tao
> >
Re: [DISCUSS ] HybridSouce Table & Sql api timeline
Posted by Becket Qin <be...@gmail.com>.
Cool, I've granted you the permission.
Cheers,
Jiangjie (Becket) Qin
On Tue, May 10, 2022 at 1:14 PM Ran Tao <ch...@gmail.com> wrote:
> Hi, Becket. Thanks for your suggestions. My id is: Ran Tao
> And i will draft this flip in a few days. thanks~
>
> Becket Qin <be...@gmail.com> 于2022年5月10日周二 12:40写道：
>
> > Hi Ran,
> >
> > The FLIP process can be found here[1].
> >
> > You don't need to pass the vote, in fact the vote is based on the FLIP
> > wiki. So drafting the FLIP wiki would be the first step. After that you
> may
> > start a discussion thread in the mailing list so people can have the
> > discussion about the feature based on your FLIP wiki. Note that it is
> very
> > important to follow the structure of the FLIP template so the discussion
> > would be more efficient.
> >
> > In case you don't have the permission to add a FLIP page yet, please let
> me
> > know your Apache Confluence ID so I can grant you the permission.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >
> > On Tue, May 10, 2022 at 12:06 PM Ran Tao <ch...@gmail.com> wrote:
> >
> > > Hi, Martijn, Jacky. Thanks for your responding. It indeed need a
> designed
> > > doc or FLIP to illustrate some details and concrete implementation.
> > >
> > > And i'm glad to work on this issue. I wonder whether i can create a
> FLIP
> > > under discussion firstly
> > > to write the draft design of the implementation about table & sql api
> and
> > > some details, or we must pass voting?
> > >
> >
>
>
> --
> Best,
> Ran Tao
>
Re: [DISCUSS ] HybridSouce Table & Sql api timeline
Posted by Ran Tao <ch...@gmail.com>.
Hi, Becket. Thanks for your suggestions. My id is: Ran Tao
And i will draft this flip in a few days. thanks~
Becket Qin <be...@gmail.com> 于2022年5月10日周二 12:40写道：
> Hi Ran,
>
> The FLIP process can be found here[1].
>
> You don't need to pass the vote, in fact the vote is based on the FLIP
> wiki. So drafting the FLIP wiki would be the first step. After that you may
> start a discussion thread in the mailing list so people can have the
> discussion about the feature based on your FLIP wiki. Note that it is very
> important to follow the structure of the FLIP template so the discussion
> would be more efficient.
>
> In case you don't have the permission to add a FLIP page yet, please let me
> know your Apache Confluence ID so I can grant you the permission.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>
> On Tue, May 10, 2022 at 12:06 PM Ran Tao <ch...@gmail.com> wrote:
>
> > Hi, Martijn, Jacky. Thanks for your responding. It indeed need a designed
> > doc or FLIP to illustrate some details and concrete implementation.
> >
> > And i'm glad to work on this issue. I wonder whether i can create a FLIP
> > under discussion firstly
> > to write the draft design of the implementation about table & sql api and
> > some details, or we must pass voting?
> >
>
--
Best,
Ran Tao
Re: [DISCUSS ] HybridSouce Table & Sql api timeline
Posted by Becket Qin <be...@gmail.com>.
Hi Ran,
The FLIP process can be found here[1].
You don't need to pass the vote, in fact the vote is based on the FLIP
wiki. So drafting the FLIP wiki would be the first step. After that you may
start a discussion thread in the mailing list so people can have the
discussion about the feature based on your FLIP wiki. Note that it is very
important to follow the structure of the FLIP template so the discussion
would be more efficient.
In case you don't have the permission to add a FLIP page yet, please let me
know your Apache Confluence ID so I can grant you the permission.
Thanks,
Jiangjie (Becket) Qin
[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
On Tue, May 10, 2022 at 12:06 PM Ran Tao <ch...@gmail.com> wrote:
> Hi, Martijn, Jacky. Thanks for your responding. It indeed need a designed
> doc or FLIP to illustrate some details and concrete implementation.
>
> And i'm glad to work on this issue. I wonder whether i can create a FLIP
> under discussion firstly
> to write the draft design of the implementation about table & sql api and
> some details, or we must pass voting?
>
Re: [DISCUSS ] HybridSouce Table & Sql api timeline
Posted by Ran Tao <ch...@gmail.com>.
Hi, Martijn, Jacky. Thanks for your responding. It indeed need a designed
doc or FLIP to illustrate some details and concrete implementation.
And i'm glad to work on this issue. I wonder whether i can create a FLIP
under discussion firstly
to write the draft design of the implementation about table & sql api and
some details, or we must pass voting?
Re: [DISCUSS ] HybridSouce Table & Sql api timeline
Posted by Martijn Visser <ma...@apache.org>.
Hi,
I would welcome any contribution that gets the HybridSource to become
available for Table API and SQL use cases. I don't think there's anyone
working on this right now.
I do think it requires a FLIP so that we can think how such a connector
could be made available. Do you want to work on that Ran?
Best regards,
Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser
On Sat, 7 May 2022 at 07:02, Jacky Lau <li...@gmail.com> wrote:
> I am interested in this, can you share your DDL first? At the same time,
> there is no standard to describe hybrid Source DDL.
> I also hope that the community can discuss together and share their ideas.
>
> Ran Tao <ch...@gmail.com> 于2022年5月7日周六 11:59写道：
>
> > HybridSource is a good feature, but now release version did not support
> > table & sql api, i wonder which time it will be ready to end-users.
> >
> > And i have implemented a inner version of my company and it works well
> now.
> > the implementation of table & sql api may involve some core questions,
> e.g.
> > bounded & unbounded source start and end offset. child sources's schema
> is
> > different from hybrid source ddl schema (batch or streaming is more
> fields
> > or lack of some fields) we need process inconsistent filed problem or
> > default filed value problem (in some child source lack of some fields)
> etc.
> > so here we may need a field mapping.
> >
> > i have some ideas and implementations, if table & sql api work in
> progress
> > i'm glad to share or take part in developing.
> >
> > thanks~
> >
>
Re: [DISCUSS ] HybridSouce Table & Sql api timeline
Posted by Jacky Lau <li...@gmail.com>.
I am interested in this, can you share your DDL first? At the same time,
there is no standard to describe hybrid Source DDL.
I also hope that the community can discuss together and share their ideas.
Ran Tao <ch...@gmail.com> 于2022年5月7日周六 11:59写道：
> HybridSource is a good feature, but now release version did not support
> table & sql api, i wonder which time it will be ready to end-users.
>
> And i have implemented a inner version of my company and it works well now.
> the implementation of table & sql api may involve some core questions, e.g.
> bounded & unbounded source start and end offset. child sources's schema is
> different from hybrid source ddl schema (batch or streaming is more fields
> or lack of some fields) we need process inconsistent filed problem or
> default filed value problem (in some child source lack of some fields) etc.
> so here we may need a field mapping.
>
> i have some ideas and implementations, if table & sql api work in progress
> i'm glad to share or take part in developing.
>
> thanks~
>