throttled stream

[Chen Bekor <ch...@gmail.com>]

is there a way to consume a kafka stream using flink with  a predefined
rate limit (eg 5 events per second)

we need this because we need to control some 3rd party api rate limitations
so,  even if we have a much larger throughput potential, we must control
the consumption rate in order not to overflow the API channel.

Re: throttled stream

[Robert Metzger <rm...@apache.org>]

Hi,
I would also go for Niels approach. If the mapper has the same parallelism
as the source and its right after it, it'll be chained to the source. The
throttling then happens with almost no overhead.

Regarding the ThrottledIterator: Afaik there is no iterator involved when
reading data out of the Kafka connector, so there is no way to plug it in
anywhere.


On Sun, Apr 17, 2016 at 8:53 AM, Márton Balassi <ba...@gmail.com>
wrote:

> There is a utility in flink-streaming-examples that might be useful, but
> is generally the same idea that Niels suggests. [1]
>
> [1]
> https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java
>
> On Sun, Apr 17, 2016 at 8:42 AM, Niels Basjes <ni...@basj.es> wrote:
>
>> Simple idea: create a map function that only does "sleep 1/5 second" and
>> put that in your pipeline somewhere.
>>
>> Niels
>> On 16 Apr 2016 22:38, "Chen Bekor" <ch...@gmail.com> wrote:
>>
>>> is there a way to consume a kafka stream using flink with  a predefined
>>> rate limit (eg 5 events per second)
>>>
>>> we need this because we need to control some 3rd party api rate
>>> limitations so,  even if we have a much larger throughput potential, we
>>> must control the consumption rate in order not to overflow the API channel.
>>>
>>
>

Re: throttled stream

[Márton Balassi <ba...@gmail.com>]

There is a utility in flink-streaming-examples that might be useful, but is
generally the same idea that Niels suggests. [1]

[1]
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java

On Sun, Apr 17, 2016 at 8:42 AM, Niels Basjes <ni...@basj.es> wrote:

> Simple idea: create a map function that only does "sleep 1/5 second" and
> put that in your pipeline somewhere.
>
> Niels
> On 16 Apr 2016 22:38, "Chen Bekor" <ch...@gmail.com> wrote:
>
>> is there a way to consume a kafka stream using flink with  a predefined
>> rate limit (eg 5 events per second)
>>
>> we need this because we need to control some 3rd party api rate
>> limitations so,  even if we have a much larger throughput potential, we
>> must control the consumption rate in order not to overflow the API channel.
>>
>

Re: throttled stream

[Niels Basjes <ni...@basj.es>]

Simple idea: create a map function that only does "sleep 1/5 second" and
put that in your pipeline somewhere.

Niels
On 16 Apr 2016 22:38, "Chen Bekor" <ch...@gmail.com> wrote:

> is there a way to consume a kafka stream using flink with  a predefined
> rate limit (eg 5 events per second)
>
> we need this because we need to control some 3rd party api rate
> limitations so,  even if we have a much larger throughput potential, we
> must control the consumption rate in order not to overflow the API channel.
>