You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Steve Yurong Su <ro...@apache.org> on 2022/07/13 11:55:00 UTC
[REQUEST] Edit Permissions for FLIP
Hi folks!
My name is Steve Su, and I am currently a third-year master’s student
at Tsinghua University.
I am now participating in a Flink-related student project in OSPP
(Open Source Promotion Plan[1]). The project aims to design and
implement the generic rate limiter for the FLIP-27 source and apply
the rate-limiting feature to the Flink CDC project. @Leonard Xu is my
mentor in this project.
We already had many discussions on source rate-limiting on the mailing
list, such as [2], [3], and so on. Leonard and I had an offline
discussion, and we thought that it would be great to have a generic
rate limiter, so I’d like to raise a new FLIP to illustrate the design
further.
Please grant my account FLIP edit permissions. My apache confluence
account id is `steveyurongsu`. :D
Thanks,
Steve Su
---
[1] https://summer-ospp.ac.cn/homepagy
[2]https://lists.apache.org/thread/ff6mcos8g4otnhjrp030lrcrf3omgfmf
[3] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
Re: [REQUEST] Edit Permissions for FLIP
Posted by Alexander Fedulov <al...@ververica.com>.
Hi Steve,
If your intended implementation involves changes to the Source API, it
should certainly be a separate FLIP.
Looking forward to your proposals.
Best,
Alexandere Fedulov
On Thu, Jul 14, 2022 at 7:18 AM Steve Yurong Su <ro...@apache.org> wrote:
> Hi Alexander,
>
> The reference [3] mentioned in my previous email is your discussion
> thread on FLIP-238. We have noticed your idea of providing a new
> rate-limiter, which is great and matches our starting point. I was
> just about to chat with you on slack about this feature :)
>
> I tend to propose a separate FLIP for the following reasons:
> 1. Source rate limiting is an independent feature and involves changes
> to the Source API.
> 2. Source rate limiting is a much more important feature. Projects
> such as Flink CDC can directly benefit from this feature, and a
> separate FLIP will help its implementation and application.
> 3. The function of the source rate-limiter may go further, it can
> provide not only limits on throughput but also limits on the number of
> requests and so on (perhaps your implementation is enough for now),
> which can better serve different types of sources.
>
> What do you think? Is it necessary to raise a FLIP separately? If
> necessary, you can also write the first version of the proposal for
> source rate-limiting, and then I will add some of my ideas :)
>
> Thanks,
> Steve
>
> Alexander Fedulov <al...@ververica.com> 于2022年7月14日周四 00:47写道：
> >
> > Hi Steve,
> >
> > in the FLIP-238 [1] I propose to also introduce a rate limiter with a
> > simple API at the level of SourceReader:
> > RateLimiter.java
> > <
> https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/common/io/ratelimiting/RateLimiter.java
> >
> > GuavaRateLimiter.java
> > <
> https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/common/io/ratelimiting/GuavaRateLimiter.java
> >
> > RateLimitedSourceReader.java
> > <
> https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/util/RateLimitedSourceReader.java
> >
> > DataGeneratorSourceV3.java#L140
> > <
> https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/DataGeneratorSourceV3.java#L140
> >
> >
> > Did you plan to go further than this?
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-238%3A+Introduce+FLIP-27-based+Data+Generator+Source
> >
> > Best,
> > Alexander Fedulov
> >
> > On Wed, Jul 13, 2022 at 6:38 PM Steve Yurong Su <ro...@apache.org> wrote:
> >
> > > Hi Jark,
> > >
> > > Thanks a lot :)
> > >
> > > Steve
> > >
> > > Jark Wu <im...@gmail.com> 于2022年7月13日周三 22:48写道：
> > > >
> > > > Hi Steve,
> > > >
> > > > Welcome to the Flink community. I have granted the edit permission to
> > > you.
> > > > Looking forward to your FLIP.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Wed, 13 Jul 2022 at 19:56, Steve Yurong Su <ro...@apache.org>
> wrote:
> > > >
> > > > > Hi folks!
> > > > >
> > > > > My name is Steve Su, and I am currently a third-year master’s
> student
> > > > > at Tsinghua University.
> > > > >
> > > > > I am now participating in a Flink-related student project in OSPP
> > > > > (Open Source Promotion Plan[1]). The project aims to design and
> > > > > implement the generic rate limiter for the FLIP-27 source and apply
> > > > > the rate-limiting feature to the Flink CDC project. @Leonard Xu is
> my
> > > > > mentor in this project.
> > > > >
> > > > > We already had many discussions on source rate-limiting on the
> mailing
> > > > > list, such as [2], [3], and so on. Leonard and I had an offline
> > > > > discussion, and we thought that it would be great to have a generic
> > > > > rate limiter, so I’d like to raise a new FLIP to illustrate the
> design
> > > > > further.
> > > > >
> > > > > Please grant my account FLIP edit permissions. My apache confluence
> > > > > account id is `steveyurongsu`. :D
> > > > >
> > > > > Thanks,
> > > > > Steve Su
> > > > > ---
> > > > > [1] https://summer-ospp.ac.cn/homepagy
> > > > > [2]
> https://lists.apache.org/thread/ff6mcos8g4otnhjrp030lrcrf3omgfmf
> > > > > [3]
> https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
> > > > >
> > >
>
Re: [REQUEST] Edit Permissions for FLIP
Posted by Steve Yurong Su <ro...@apache.org>.
Hi Alexander,
The reference [3] mentioned in my previous email is your discussion
thread on FLIP-238. We have noticed your idea of providing a new
rate-limiter, which is great and matches our starting point. I was
just about to chat with you on slack about this feature :)
I tend to propose a separate FLIP for the following reasons:
1. Source rate limiting is an independent feature and involves changes
to the Source API.
2. Source rate limiting is a much more important feature. Projects
such as Flink CDC can directly benefit from this feature, and a
separate FLIP will help its implementation and application.
3. The function of the source rate-limiter may go further, it can
provide not only limits on throughput but also limits on the number of
requests and so on (perhaps your implementation is enough for now),
which can better serve different types of sources.
What do you think? Is it necessary to raise a FLIP separately? If
necessary, you can also write the first version of the proposal for
source rate-limiting, and then I will add some of my ideas :)
Thanks,
Steve
Alexander Fedulov <al...@ververica.com> 于2022年7月14日周四 00:47写道：
>
> Hi Steve,
>
> in the FLIP-238 [1] I propose to also introduce a rate limiter with a
> simple API at the level of SourceReader:
> RateLimiter.java
> <https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/common/io/ratelimiting/RateLimiter.java>
> GuavaRateLimiter.java
> <https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/common/io/ratelimiting/GuavaRateLimiter.java>
> RateLimitedSourceReader.java
> <https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/util/RateLimitedSourceReader.java>
> DataGeneratorSourceV3.java#L140
> <https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/DataGeneratorSourceV3.java#L140>
>
> Did you plan to go further than this?
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-238%3A+Introduce+FLIP-27-based+Data+Generator+Source
>
> Best,
> Alexander Fedulov
>
> On Wed, Jul 13, 2022 at 6:38 PM Steve Yurong Su <ro...@apache.org> wrote:
>
> > Hi Jark,
> >
> > Thanks a lot :)
> >
> > Steve
> >
> > Jark Wu <im...@gmail.com> 于2022年7月13日周三 22:48写道：
> > >
> > > Hi Steve,
> > >
> > > Welcome to the Flink community. I have granted the edit permission to
> > you.
> > > Looking forward to your FLIP.
> > >
> > > Best,
> > > Jark
> > >
> > > On Wed, 13 Jul 2022 at 19:56, Steve Yurong Su <ro...@apache.org> wrote:
> > >
> > > > Hi folks!
> > > >
> > > > My name is Steve Su, and I am currently a third-year master’s student
> > > > at Tsinghua University.
> > > >
> > > > I am now participating in a Flink-related student project in OSPP
> > > > (Open Source Promotion Plan[1]). The project aims to design and
> > > > implement the generic rate limiter for the FLIP-27 source and apply
> > > > the rate-limiting feature to the Flink CDC project. @Leonard Xu is my
> > > > mentor in this project.
> > > >
> > > > We already had many discussions on source rate-limiting on the mailing
> > > > list, such as [2], [3], and so on. Leonard and I had an offline
> > > > discussion, and we thought that it would be great to have a generic
> > > > rate limiter, so I’d like to raise a new FLIP to illustrate the design
> > > > further.
> > > >
> > > > Please grant my account FLIP edit permissions. My apache confluence
> > > > account id is `steveyurongsu`. :D
> > > >
> > > > Thanks,
> > > > Steve Su
> > > > ---
> > > > [1] https://summer-ospp.ac.cn/homepagy
> > > > [2]https://lists.apache.org/thread/ff6mcos8g4otnhjrp030lrcrf3omgfmf
> > > > [3] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
> > > >
> >
Re: [REQUEST] Edit Permissions for FLIP
Posted by Alexander Fedulov <al...@ververica.com>.
Hi Steve,
in the FLIP-238 [1] I propose to also introduce a rate limiter with a
simple API at the level of SourceReader:
RateLimiter.java
<https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/common/io/ratelimiting/RateLimiter.java>
GuavaRateLimiter.java
<https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/common/io/ratelimiting/GuavaRateLimiter.java>
RateLimitedSourceReader.java
<https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/util/RateLimitedSourceReader.java>
DataGeneratorSourceV3.java#L140
<https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/DataGeneratorSourceV3.java#L140>
Did you plan to go further than this?
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-238%3A+Introduce+FLIP-27-based+Data+Generator+Source
Best,
Alexander Fedulov
On Wed, Jul 13, 2022 at 6:38 PM Steve Yurong Su <ro...@apache.org> wrote:
> Hi Jark,
>
> Thanks a lot :)
>
> Steve
>
> Jark Wu <im...@gmail.com> 于2022年7月13日周三 22:48写道：
> >
> > Hi Steve,
> >
> > Welcome to the Flink community. I have granted the edit permission to
> you.
> > Looking forward to your FLIP.
> >
> > Best,
> > Jark
> >
> > On Wed, 13 Jul 2022 at 19:56, Steve Yurong Su <ro...@apache.org> wrote:
> >
> > > Hi folks!
> > >
> > > My name is Steve Su, and I am currently a third-year master’s student
> > > at Tsinghua University.
> > >
> > > I am now participating in a Flink-related student project in OSPP
> > > (Open Source Promotion Plan[1]). The project aims to design and
> > > implement the generic rate limiter for the FLIP-27 source and apply
> > > the rate-limiting feature to the Flink CDC project. @Leonard Xu is my
> > > mentor in this project.
> > >
> > > We already had many discussions on source rate-limiting on the mailing
> > > list, such as [2], [3], and so on. Leonard and I had an offline
> > > discussion, and we thought that it would be great to have a generic
> > > rate limiter, so I’d like to raise a new FLIP to illustrate the design
> > > further.
> > >
> > > Please grant my account FLIP edit permissions. My apache confluence
> > > account id is `steveyurongsu`. :D
> > >
> > > Thanks,
> > > Steve Su
> > > ---
> > > [1] https://summer-ospp.ac.cn/homepagy
> > > [2]https://lists.apache.org/thread/ff6mcos8g4otnhjrp030lrcrf3omgfmf
> > > [3] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
> > >
>
Re: [REQUEST] Edit Permissions for FLIP
Posted by Steve Yurong Su <ro...@apache.org>.
Hi Jark,
Thanks a lot :)
Steve
Jark Wu <im...@gmail.com> 于2022年7月13日周三 22:48写道：
>
> Hi Steve,
>
> Welcome to the Flink community. I have granted the edit permission to you.
> Looking forward to your FLIP.
>
> Best,
> Jark
>
> On Wed, 13 Jul 2022 at 19:56, Steve Yurong Su <ro...@apache.org> wrote:
>
> > Hi folks!
> >
> > My name is Steve Su, and I am currently a third-year master’s student
> > at Tsinghua University.
> >
> > I am now participating in a Flink-related student project in OSPP
> > (Open Source Promotion Plan[1]). The project aims to design and
> > implement the generic rate limiter for the FLIP-27 source and apply
> > the rate-limiting feature to the Flink CDC project. @Leonard Xu is my
> > mentor in this project.
> >
> > We already had many discussions on source rate-limiting on the mailing
> > list, such as [2], [3], and so on. Leonard and I had an offline
> > discussion, and we thought that it would be great to have a generic
> > rate limiter, so I’d like to raise a new FLIP to illustrate the design
> > further.
> >
> > Please grant my account FLIP edit permissions. My apache confluence
> > account id is `steveyurongsu`. :D
> >
> > Thanks,
> > Steve Su
> > ---
> > [1] https://summer-ospp.ac.cn/homepagy
> > [2]https://lists.apache.org/thread/ff6mcos8g4otnhjrp030lrcrf3omgfmf
> > [3] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
> >
Re: [REQUEST] Edit Permissions for FLIP
Posted by Jark Wu <im...@gmail.com>.
Hi Steve,
Welcome to the Flink community. I have granted the edit permission to you.
Looking forward to your FLIP.
Best,
Jark
On Wed, 13 Jul 2022 at 19:56, Steve Yurong Su <ro...@apache.org> wrote:
> Hi folks!
>
> My name is Steve Su, and I am currently a third-year master’s student
> at Tsinghua University.
>
> I am now participating in a Flink-related student project in OSPP
> (Open Source Promotion Plan[1]). The project aims to design and
> implement the generic rate limiter for the FLIP-27 source and apply
> the rate-limiting feature to the Flink CDC project. @Leonard Xu is my
> mentor in this project.
>
> We already had many discussions on source rate-limiting on the mailing
> list, such as [2], [3], and so on. Leonard and I had an offline
> discussion, and we thought that it would be great to have a generic
> rate limiter, so I’d like to raise a new FLIP to illustrate the design
> further.
>
> Please grant my account FLIP edit permissions. My apache confluence
> account id is `steveyurongsu`. :D
>
> Thanks,
> Steve Su
> ---
> [1] https://summer-ospp.ac.cn/homepagy
> [2]https://lists.apache.org/thread/ff6mcos8g4otnhjrp030lrcrf3omgfmf
> [3] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
>