You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by shijinkui <sh...@huawei.com> on 2016/09/27 13:31:08 UTC
答复: 答复: [DISCUSS] add netty tcp/restful pushed source support
It's nice. Will present flink source connector be pushed to bahir-flink?
I can add netty-source to bahir-flink.
Maven repository have no bahir-flink's.
https://mvnrepository.com/artifact/org.apache.bahir
-----邮件原件-----
发件人: Greg Hogan [mailto:code@greghogan.com]
发送时间: 2016年9月27日 20:58
收件人: dev@flink.apache.org
主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support
Apache Bahir's website only suggests support for additional frameworks, but there is a Flink repository at
https://github.com/apache/bahir-flink
On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <sh...@huawei.com> wrote:
> Hey, Stephan Ewen
>
> 1. bahir's target is spark. The contributer are rxin, srowen, tdas,
> mateiz and so on.
> If we want bahir used by flink, we can suggest bahir provide
> streaming connecter interface, such as store(), start(), stop(),
> restart(), receiving(Any)...
> Then same streaming connector can be implemented by spark and
> flink. But I think this is impossible, as bahir depend spark-streaming
> and spark sql.
> 2. About connector fault tolerance. Bahir's mqtt and akka connector
> are themselves' storage. But netty have no persist data feature.
> I think we can append data to a ringbuffer. When SourceContext
> collect() throw error, then write message to inform client to stop
> send message. When flink SourceContext is normal, then write the
> ringbuffer data to flink, inform client to go on.
> Because pushing mode is hard to control the flow throughput,
> the upstream client can.
> This netty connector's purpose is end-to-end streaming,
> minimum time delay.
> 3. Later on, We can provide http protocol ability, put, post, get,
> even websocket or jersey restful based on netty.
>
> -----邮件原件-----
> 发件人: Stephan Ewen [mailto:sewen@apache.org]
> 发送时间: 2016年9月27日 16:54
> 收件人: dev@flink.apache.org
> 主题: Re: [DISCUSS] add netty tcp/restful pushed source support
>
> I think that could be an interesting source. Two quick questions to
> move forward
>
> - To keep the Flink code base from becoming too big (hard to
> maintain and
> test) we started working with Apache Bahir as a project dedicated to
> streaming connectors. Would that be a good target for the connector?
>
> - What are your thoughts on fault tolerance for that connector?
>
> On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <sh...@huawei.com> wrote:
>
> > Hi, all
> >
> > 1. In order to support end-to-end pushed source, I create FLINK-4630<
> > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know
> > whether is this idea worth?
> >
> > -------------------
> > When source stream get start, listen a provided tcp port, receive
> > stream data from user data source.
> > This netty tcp source is keepping alive and end-to-end, that is from
> > business system to flink worker directly.
> >
> > user app push -----> netty server source of Flink
> >
> > describe the source in detail below:
> >
> > 1. source run as a netty tcp server
> > 2. user provide a tcp port, if the port is in used, increace the port
> > number between 1024 to 65535. Source can parallel.
> > 3. callback the provided url to report the real port to listen
> > 4. user push streaming data to netty server, then collect the data to
> > flink
> >
> >
> > Thanks
> >
> > Jinkui Shi
> >
> >
>
Re: 答复: 答复: [DISCUSS] add netty tcp/restful pushed source support
Posted by Stephan Ewen <se...@apache.org>.
+1 to add the source to Bahir
That is an easier way to iterate fast on this and release quickly
On Wed, Oct 12, 2016 at 10:38 AM, Robert Metzger <rm...@apache.org>
wrote:
> Just a quick update on this one: The bahir community started already
> discussing the first bahir-flink release. I expect it to happen soon.
>
> I would really like to see the netty source in Bahir.
>
> On Wed, Sep 28, 2016 at 3:18 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > The Bahir-Flink stuff is fairly new - the first release has not yet
> > happened.
> > Robert is pushing that, will probably happen once he is back.
> >
> > On Tue, Sep 27, 2016 at 3:31 PM, shijinkui <sh...@huawei.com> wrote:
> >
> > > It's nice. Will present flink source connector be pushed to
> bahir-flink?
> > > I can add netty-source to bahir-flink.
> > >
> > > Maven repository have no bahir-flink's.
> > > https://mvnrepository.com/artifact/org.apache.bahir
> > >
> > > -----邮件原件-----
> > > 发件人: Greg Hogan [mailto:code@greghogan.com]
> > > 发送时间: 2016年9月27日 20:58
> > > 收件人: dev@flink.apache.org
> > > 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support
> > >
> > > Apache Bahir's website only suggests support for additional frameworks,
> > > but there is a Flink repository at
> > > https://github.com/apache/bahir-flink
> > >
> > > On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <sh...@huawei.com>
> wrote:
> > >
> > > > Hey, Stephan Ewen
> > > >
> > > > 1. bahir's target is spark. The contributer are rxin, srowen,
> > tdas,
> > > > mateiz and so on.
> > > > If we want bahir used by flink, we can suggest bahir provide
> > > > streaming connecter interface, such as store(), start(), stop(),
> > > > restart(), receiving(Any)...
> > > > Then same streaming connector can be implemented by spark and
> > > > flink. But I think this is impossible, as bahir depend
> spark-streaming
> > > > and spark sql.
> > > > 2. About connector fault tolerance. Bahir's mqtt and akka
> > connector
> > > > are themselves' storage. But netty have no persist data feature.
> > > > I think we can append data to a ringbuffer. When
> SourceContext
> > > > collect() throw error, then write message to inform client to stop
> > > > send message. When flink SourceContext is normal, then write the
> > > > ringbuffer data to flink, inform client to go on.
> > > > Because pushing mode is hard to control the flow throughput,
> > > > the upstream client can.
> > > > This netty connector's purpose is end-to-end streaming,
> > > > minimum time delay.
> > > > 3. Later on, We can provide http protocol ability, put, post,
> get,
> > > > even websocket or jersey restful based on netty.
> > > >
> > > > -----邮件原件-----
> > > > 发件人: Stephan Ewen [mailto:sewen@apache.org]
> > > > 发送时间: 2016年9月27日 16:54
> > > > 收件人: dev@flink.apache.org
> > > > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support
> > > >
> > > > I think that could be an interesting source. Two quick questions to
> > > > move forward
> > > >
> > > > - To keep the Flink code base from becoming too big (hard to
> > > > maintain and
> > > > test) we started working with Apache Bahir as a project dedicated to
> > > > streaming connectors. Would that be a good target for the connector?
> > > >
> > > > - What are your thoughts on fault tolerance for that connector?
> > > >
> > > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <sh...@huawei.com>
> > wrote:
> > > >
> > > > > Hi, all
> > > > >
> > > > > 1. In order to support end-to-end pushed source, I create
> > > FLINK-4630<
> > > > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know
> > > > > whether is this idea worth?
> > > > >
> > > > > -------------------
> > > > > When source stream get start, listen a provided tcp port, receive
> > > > > stream data from user data source.
> > > > > This netty tcp source is keepping alive and end-to-end, that is
> from
> > > > > business system to flink worker directly.
> > > > >
> > > > > user app push -----> netty server source of Flink
> > > > >
> > > > > describe the source in detail below:
> > > > >
> > > > > 1. source run as a netty tcp server
> > > > > 2. user provide a tcp port, if the port is in used, increace the
> > > port
> > > > > number between 1024 to 65535. Source can parallel.
> > > > > 3. callback the provided url to report the real port to listen
> > > > > 4. user push streaming data to netty server, then collect the
> data
> > > to
> > > > > flink
> > > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > > Jinkui Shi
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: 答复: 答复: [DISCUSS] add netty tcp/restful pushed source support
Posted by Robert Metzger <rm...@apache.org>.
Just a quick update on this one: The bahir community started already
discussing the first bahir-flink release. I expect it to happen soon.
I would really like to see the netty source in Bahir.
On Wed, Sep 28, 2016 at 3:18 PM, Stephan Ewen <se...@apache.org> wrote:
> The Bahir-Flink stuff is fairly new - the first release has not yet
> happened.
> Robert is pushing that, will probably happen once he is back.
>
> On Tue, Sep 27, 2016 at 3:31 PM, shijinkui <sh...@huawei.com> wrote:
>
> > It's nice. Will present flink source connector be pushed to bahir-flink?
> > I can add netty-source to bahir-flink.
> >
> > Maven repository have no bahir-flink's.
> > https://mvnrepository.com/artifact/org.apache.bahir
> >
> > -----邮件原件-----
> > 发件人: Greg Hogan [mailto:code@greghogan.com]
> > 发送时间: 2016年9月27日 20:58
> > 收件人: dev@flink.apache.org
> > 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support
> >
> > Apache Bahir's website only suggests support for additional frameworks,
> > but there is a Flink repository at
> > https://github.com/apache/bahir-flink
> >
> > On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <sh...@huawei.com> wrote:
> >
> > > Hey, Stephan Ewen
> > >
> > > 1. bahir's target is spark. The contributer are rxin, srowen,
> tdas,
> > > mateiz and so on.
> > > If we want bahir used by flink, we can suggest bahir provide
> > > streaming connecter interface, such as store(), start(), stop(),
> > > restart(), receiving(Any)...
> > > Then same streaming connector can be implemented by spark and
> > > flink. But I think this is impossible, as bahir depend spark-streaming
> > > and spark sql.
> > > 2. About connector fault tolerance. Bahir's mqtt and akka
> connector
> > > are themselves' storage. But netty have no persist data feature.
> > > I think we can append data to a ringbuffer. When SourceContext
> > > collect() throw error, then write message to inform client to stop
> > > send message. When flink SourceContext is normal, then write the
> > > ringbuffer data to flink, inform client to go on.
> > > Because pushing mode is hard to control the flow throughput,
> > > the upstream client can.
> > > This netty connector's purpose is end-to-end streaming,
> > > minimum time delay.
> > > 3. Later on, We can provide http protocol ability, put, post, get,
> > > even websocket or jersey restful based on netty.
> > >
> > > -----邮件原件-----
> > > 发件人: Stephan Ewen [mailto:sewen@apache.org]
> > > 发送时间: 2016年9月27日 16:54
> > > 收件人: dev@flink.apache.org
> > > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support
> > >
> > > I think that could be an interesting source. Two quick questions to
> > > move forward
> > >
> > > - To keep the Flink code base from becoming too big (hard to
> > > maintain and
> > > test) we started working with Apache Bahir as a project dedicated to
> > > streaming connectors. Would that be a good target for the connector?
> > >
> > > - What are your thoughts on fault tolerance for that connector?
> > >
> > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <sh...@huawei.com>
> wrote:
> > >
> > > > Hi, all
> > > >
> > > > 1. In order to support end-to-end pushed source, I create
> > FLINK-4630<
> > > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know
> > > > whether is this idea worth?
> > > >
> > > > -------------------
> > > > When source stream get start, listen a provided tcp port, receive
> > > > stream data from user data source.
> > > > This netty tcp source is keepping alive and end-to-end, that is from
> > > > business system to flink worker directly.
> > > >
> > > > user app push -----> netty server source of Flink
> > > >
> > > > describe the source in detail below:
> > > >
> > > > 1. source run as a netty tcp server
> > > > 2. user provide a tcp port, if the port is in used, increace the
> > port
> > > > number between 1024 to 65535. Source can parallel.
> > > > 3. callback the provided url to report the real port to listen
> > > > 4. user push streaming data to netty server, then collect the data
> > to
> > > > flink
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Jinkui Shi
> > > >
> > > >
> > >
> >
>
Re: 答复: 答复: [DISCUSS] add netty tcp/restful pushed source support
Posted by Stephan Ewen <se...@apache.org>.
The Bahir-Flink stuff is fairly new - the first release has not yet
happened.
Robert is pushing that, will probably happen once he is back.
On Tue, Sep 27, 2016 at 3:31 PM, shijinkui <sh...@huawei.com> wrote:
> It's nice. Will present flink source connector be pushed to bahir-flink?
> I can add netty-source to bahir-flink.
>
> Maven repository have no bahir-flink's.
> https://mvnrepository.com/artifact/org.apache.bahir
>
> -----邮件原件-----
> 发件人: Greg Hogan [mailto:code@greghogan.com]
> 发送时间: 2016年9月27日 20:58
> 收件人: dev@flink.apache.org
> 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support
>
> Apache Bahir's website only suggests support for additional frameworks,
> but there is a Flink repository at
> https://github.com/apache/bahir-flink
>
> On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <sh...@huawei.com> wrote:
>
> > Hey, Stephan Ewen
> >
> > 1. bahir's target is spark. The contributer are rxin, srowen, tdas,
> > mateiz and so on.
> > If we want bahir used by flink, we can suggest bahir provide
> > streaming connecter interface, such as store(), start(), stop(),
> > restart(), receiving(Any)...
> > Then same streaming connector can be implemented by spark and
> > flink. But I think this is impossible, as bahir depend spark-streaming
> > and spark sql.
> > 2. About connector fault tolerance. Bahir's mqtt and akka connector
> > are themselves' storage. But netty have no persist data feature.
> > I think we can append data to a ringbuffer. When SourceContext
> > collect() throw error, then write message to inform client to stop
> > send message. When flink SourceContext is normal, then write the
> > ringbuffer data to flink, inform client to go on.
> > Because pushing mode is hard to control the flow throughput,
> > the upstream client can.
> > This netty connector's purpose is end-to-end streaming,
> > minimum time delay.
> > 3. Later on, We can provide http protocol ability, put, post, get,
> > even websocket or jersey restful based on netty.
> >
> > -----邮件原件-----
> > 发件人: Stephan Ewen [mailto:sewen@apache.org]
> > 发送时间: 2016年9月27日 16:54
> > 收件人: dev@flink.apache.org
> > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support
> >
> > I think that could be an interesting source. Two quick questions to
> > move forward
> >
> > - To keep the Flink code base from becoming too big (hard to
> > maintain and
> > test) we started working with Apache Bahir as a project dedicated to
> > streaming connectors. Would that be a good target for the connector?
> >
> > - What are your thoughts on fault tolerance for that connector?
> >
> > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <sh...@huawei.com> wrote:
> >
> > > Hi, all
> > >
> > > 1. In order to support end-to-end pushed source, I create
> FLINK-4630<
> > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know
> > > whether is this idea worth?
> > >
> > > -------------------
> > > When source stream get start, listen a provided tcp port, receive
> > > stream data from user data source.
> > > This netty tcp source is keepping alive and end-to-end, that is from
> > > business system to flink worker directly.
> > >
> > > user app push -----> netty server source of Flink
> > >
> > > describe the source in detail below:
> > >
> > > 1. source run as a netty tcp server
> > > 2. user provide a tcp port, if the port is in used, increace the
> port
> > > number between 1024 to 65535. Source can parallel.
> > > 3. callback the provided url to report the real port to listen
> > > 4. user push streaming data to netty server, then collect the data
> to
> > > flink
> > >
> > >
> > > Thanks
> > >
> > > Jinkui Shi
> > >
> > >
> >
>