You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Zhijiang(wangzhijiang999)" <wa...@aliyun.com> on 2018/07/06 03:16:04 UTC
回复：Limiting in flight data
Hi Vishal,
Before Flink-1.5.0, the sender tries best to send data on the network until the wire is filled with data. From Flink-1.5.0 the network flow control is improved by credit-based idea. That means the sender transfers data based on how many buffers avaiable on receiver side, so there will be no data accumulated on the wire. From this point, the in-flighting data is less than before.
Also you can further limit the in-flighting data by controling the number of credits on receiver side, and the related parameters are taskmanager.network.memory.buffers-per-channel and taskmanager.network.memory.floating-buffers-per-gate.
If you have other questions about them, let me know then i can explain for you.
Zhijiang
------------------------------------------------------------------
发件人：Vishal Santoshi <vi...@gmail.com>
发送时间：2018年7月5日(星期四) 22:28
收件人：user <us...@flink.apache.org>
主 题：Limiting in flight data
"Yes, Flink 1.5.0 will come with better tools to handle this problem. Namely you will be able to limit the “in flight” data, by controlling the number of assigned credits per channel/input gate. Even without any configuring Flink 1.5.0 will out of the box buffer less data, thus mitigating the problem."
I read this in another email chain. The docs ( may be you can point me to them ) are not very clear on how to do the above. Any pointers will be appreciated.
Thanks much.
回复：Limiting in flight data
Posted by "Zhijiang(wangzhijiang999)" <wa...@aliyun.com>.
The config you mentioned is not operator level, but can be setted at job level currently I think. The operator level needs the API support but seems more reasonable.
There exists "inPoolUsage" and "outPoolUsage" metrcis to indicate backpreesure to some extent. If the percentages of these metrics are both 100% between producer and consumer, the producer will be blocked (backpressure) by the consumer for a while.
Also there exists latency marker from source to sink in the whole topology to sample latency. Maybe you can resort these metrics for some helps.
------------------------------------------------------------------
发件人：Vishal Santoshi <vi...@gmail.com>
发送时间：2018年7月6日(星期五) 22:05
收件人：Zhijiang(wangzhijiang999) <wa...@aliyun.com>
抄 送：user <us...@flink.apache.org>
主 题：Re: Limiting in flight data
Further if there is are metrics that allows us to chart delays per pipe on n/w buffers, that would be immensely helpful.
On Fri, Jul 6, 2018 at 10:02 AM, Vishal Santoshi <vi...@gmail.com> wrote:
Awesome, thank you for pointing that out. We have seen stability on pipes where previously throttling the source ( rateLimiter ) was the only way out.
https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java#L291
This though seems to be a cluster wide setting. Is it possible to do this at an operator level ? Does this work with the pipe level configuration per job ( or has that been deprecated )
On Thu, Jul 5, 2018 at 11:16 PM, Zhijiang(wangzhijiang999) <wa...@aliyun.com> wrote:
Hi Vishal,
Before Flink-1.5.0, the sender tries best to send data on the network until the wire is filled with data. From Flink-1.5.0 the network flow control is improved by credit-based idea. That means the sender transfers data based on how many buffers avaiable on receiver side, so there will be no data accumulated on the wire. From this point, the in-flighting data is less than before.
Also you can further limit the in-flighting data by controling the number of credits on receiver side, and the related parameters are taskmanager.network.memory.buffers-per-channel and taskmanager.network.memory.floating-buffers-per-gate.
If you have other questions about them, let me know then i can explain for you.
Zhijiang
------------------------------------------------------------------
发件人：Vishal Santoshi <vi...@gmail.com>
发送时间：2018年7月5日(星期四) 22:28
收件人：user <us...@flink.apache.org>
主 题：Limiting in flight data
"Yes, Flink 1.5.0 will come with better tools to handle this problem. Namely you will be able to limit the “in flight” data, by controlling the number of assigned credits per channel/input gate. Even without any configuring Flink 1.5.0 will out of the box buffer less data, thus mitigating the problem."
I read this in another email chain. The docs ( may be you can point me to them ) are not very clear on how to do the above. Any pointers will be appreciated.
Thanks much.
Re: Limiting in flight data
Posted by Vishal Santoshi <vi...@gmail.com>.
Further if there is are metrics that allows us to chart delays per pipe on
n/w buffers, that would be immensely helpful.
On Fri, Jul 6, 2018 at 10:02 AM, Vishal Santoshi <vi...@gmail.com>
wrote:
> Awesome, thank you for pointing that out. We have seen stability on pipes
> where previously throttling the source ( rateLimiter ) was the only way out.
>
> https://github.com/apache/flink/blob/master/flink-core/
> src/main/java/org/apache/flink/configuration/TaskManagerOptions.java#L291
>
> This though seems to be a cluster wide setting. Is it possible to do this
> at an operator level ? Does this work with the pipe level configuration
> per job ( or has that been deprecated )
>
> On Thu, Jul 5, 2018 at 11:16 PM, Zhijiang(wangzhijiang999) <
> wangzhijiang999@aliyun.com> wrote:
>
>> Hi Vishal,
>>
>> Before Flink-1.5.0, the sender tries best to send data on the network
>> until the wire is filled with data. From Flink-1.5.0
>> the network flow control is improved by credit-based idea. That mea
>> ns the sender transfers data based on how many buffers
>> avaiable on receiver side, so there will be no data accumulated on the wire. From
>> this point, the in-flighting data is less than before.
>>
>> Also you can further limit the in-flighting data by controling the number
>> of credits on receiver side, and the related parameters are
>> taskmanager.network.memory.buffers-per-channel and
>> taskmanager.network.memory.floating-buffers-per-gate.
>>
>> If you have other questions about them, let me know then i can explain
>> for you.
>>
>> Zhijiang
>>
>> ------------------------------------------------------------------
>> 发件人：Vishal Santoshi <vi...@gmail.com>
>> 发送时间：2018年7月5日(星期四) 22:28
>> 收件人：user <us...@flink.apache.org>
>> 主 题：Limiting in flight data
>>
>> "Yes, Flink 1.5.0 will come with better tools to handle this problem.
>> Namely you will be able to limit the “in flight” data, by controlling the
>> number of assigned credits per channel/input gate. Even without any
>> configuring Flink 1.5.0 will out of the box buffer less data, thus
>> mitigating the problem."
>>
>> I read this in another email chain. The docs ( may be you can point me to
>> them ) are not very clear on how to do the above. Any pointers will be
>> appreciated.
>>
>> Thanks much.
>>
>>
>>
>
Re: Limiting in flight data
Posted by Vishal Santoshi <vi...@gmail.com>.
Awesome, thank you for pointing that out. We have seen stability on pipes
where previously throttling the source ( rateLimiter ) was the only way out.
https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java#L291
This though seems to be a cluster wide setting. Is it possible to do this
at an operator level ? Does this work with the pipe level configuration
per job ( or has that been deprecated )
On Thu, Jul 5, 2018 at 11:16 PM, Zhijiang(wangzhijiang999) <
wangzhijiang999@aliyun.com> wrote:
> Hi Vishal,
>
> Before Flink-1.5.0, the sender tries best to send data on the network
> until the wire is filled with data. From Flink-1.5.0
> the network flow control is improved by credit-based idea. That
> means the sender transfers data based on how many
> buffers avaiable on receiver side, so there will be no
> data accumulated on the wire. From this point, the in-flighting data is
> less than before.
>
> Also you can further limit the in-flighting data by controling the number
> of credits on receiver side, and the related parameters are
> taskmanager.network.memory.buffers-per-channel and
> taskmanager.network.memory.floating-buffers-per-gate.
>
> If you have other questions about them, let me know then i can explain for
> you.
>
> Zhijiang
>
> ------------------------------------------------------------------
> 发件人：Vishal Santoshi <vi...@gmail.com>
> 发送时间：2018年7月5日(星期四) 22:28
> 收件人：user <us...@flink.apache.org>
> 主 题：Limiting in flight data
>
> "Yes, Flink 1.5.0 will come with better tools to handle this problem.
> Namely you will be able to limit the “in flight” data, by controlling the
> number of assigned credits per channel/input gate. Even without any
> configuring Flink 1.5.0 will out of the box buffer less data, thus
> mitigating the problem."
>
> I read this in another email chain. The docs ( may be you can point me to
> them ) are not very clear on how to do the above. Any pointers will be
> appreciated.
>
> Thanks much.
>
>
>