You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Piotr Nowojski <pi...@ververica.com> on 2019/10/02 15:01:33 UTC
[SURVEY] Dropping non Credit-based Flow Control
Hi,
In Flink 1.5 we have introduced Credit-based Flow Control [1] in the network stack. Back then we were aware about potential downsides of it [2] and we decided to keep the old model in the code base ( configurable by setting `taskmanager.network.credit-model: false` ). Now, that we are about to modify internals of the network stack again [3], it might be a good time to clean up the code and remove the older code paths.
Is anyone still using the non default non Credit-based model ( `taskmanager.network.credit-model: false`)? If so, why?
Piotrek
[1] https://flink.apache.org/2019/06/05/flink-network-stack.html <https://flink.apache.org/2019/06/05/flink-network-stack.html>
[2] https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch <https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch>
[3] https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E <https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E>
Re: [SURVEY] Dropping non Credit-based Flow Control
Posted by Piotr Nowojski <pi...@ververica.com>.
Hi,
Thank you all for the feedback. I’ve created a ticket [1] to remove the non Credit-based Flow Control code paths.
Piotrek
[1] https://issues.apache.org/jira/browse/FLINK-14516 <https://issues.apache.org/jira/browse/FLINK-14516>
> On 23 Oct 2019, at 10:59, Nico Kruber <ni...@ververica.com> wrote:
>
> +1
>
> I have not heard of a real-world use-case that suffered more than it
> gained and also think it is time to remove the old paths.
>
> There are, however, still improvements to be made in credit-based flow
> control (like [1]) but that should not stop us from removing the old
> paths if no-one is really using them anyway
>
>
> Nico
>
> [1] https://issues.apache.org/jira/browse/FLINK-10742
>
> On 21/10/2019 03:14, SHI Xiaogang wrote:
>> +1
>>
>> Credit-based flow control has long been used in our production environment
>> as well. It works fine and there seems no reason to use non credit-based
>> implementation.
>>
>> Regards,
>> Xiaogang
>>
>> Zhu Zhu <re...@gmail.com> 于2019年10月19日周六 下午3:01写道：
>>
>>> +1 to drop the non credit-based flow control.
>>> We have turned to credit-based flow control for long in production. It has
>>> been good for all our cases.
>>> The non credit-based flow control code has been a burden when we are trying
>>> to change the network stack code for new features.
>>>
>>> Thanks,
>>> Zhu Zhu
>>>
>>>
>>> Biao Liu <mm...@gmail.com> 于2019年10月10日周四 下午5:45写道：
>>>
>>>> Thanks for start this survey, Piotr.
>>>>
>>>> We have benefitted from credit-based flow control a lot. I can't figure
>>> out
>>>> a reason to use non credit-based model.
>>>> I think we have kept the older code paths long enough (1.5 -> 1.9).
>>> That's
>>>> a big burden to maintain. Especially there are a lot duplicated codes
>>>> between credit-based and non credit-based model.
>>>>
>>>> So +1 to do the cleanup.
>>>>
>>>> Thanks,
>>>> Biao /'bɪ.aʊ/
>>>>
>>>>
>>>>
>>>> On Thu, 10 Oct 2019 at 11:15, zhijiang <wangzhijiang999@aliyun.com
>>>> .invalid>
>>>> wrote:
>>>>
>>>>> Thanks for bringing this survey Piotr.
>>>>>
>>>>> Actually I was also trying to dropping the non credit-based code path
>>>> from
>>>>> release-1.9, and now I think it is the proper time to do it motivated
>>> by
>>>>> [3].
>>>>> The credit-based mode is as default from Flink 1.5 and it has been
>>>>> verified to be stable and reliable in many versions. In Alibaba we are
>>>>> always using the default credit-based mode in all products.
>>>>> It can reduce much overhead of maintaining non credit-based code path,
>>> so
>>>>> +1 from my side to drop it.
>>>>>
>>>>> Best,
>>>>> Zhijiang
>>>>> ------------------------------------------------------------------
>>>>> From:Piotr Nowojski <pi...@ververica.com>
>>>>> Send Time:2019年10月2日(星期三) 17:01
>>>>> To:dev <de...@flink.apache.org>
>>>>> Subject:[SURVEY] Dropping non Credit-based Flow Control
>>>>>
>>>>> Hi,
>>>>>
>>>>> In Flink 1.5 we have introduced Credit-based Flow Control [1] in the
>>>>> network stack. Back then we were aware about potential downsides of it
>>>> [2]
>>>>> and we decided to keep the old model in the code base ( configurable by
>>>>> setting `taskmanager.network.credit-model: false` ). Now, that we are
>>>>> about to modify internals of the network stack again [3], it might be a
>>>>> good time to clean up the code and remove the older code paths.
>>>>>
>>>>> Is anyone still using the non default non Credit-based model (
>>>>> `taskmanager.network.credit-model: false`)? If so, why?
>>>>>
>>>>> Piotrek
>>>>>
>>>>> [1] https://flink.apache.org/2019/06/05/flink-network-stack.html <
>>>>> https://flink.apache.org/2019/06/05/flink-network-stack.html>
>>>>> [2]
>>>>>
>>>>
>>> https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch
>>>>> <
>>>>>
>>>>
>>> https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch
>>>>>>
>>>>> [3]
>>>>>
>>>>
>>> https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E
>>>>> <
>>>>>
>>>>
>>> https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
> --
> Nico Kruber | Solutions Architect
>
> Follow us @VervericaData Ververica
> --
> Join Flink Forward - The Apache Flink Conference
> Stream Processing | Event Driven | Real Time
> --
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Tony) Cheng
>
Re: [SURVEY] Dropping non Credit-based Flow Control
Posted by Nico Kruber <ni...@ververica.com>.
+1
I have not heard of a real-world use-case that suffered more than it
gained and also think it is time to remove the old paths.
There are, however, still improvements to be made in credit-based flow
control (like [1]) but that should not stop us from removing the old
paths if no-one is really using them anyway
Nico
[1] https://issues.apache.org/jira/browse/FLINK-10742
On 21/10/2019 03:14, SHI Xiaogang wrote:
> +1
>
> Credit-based flow control has long been used in our production environment
> as well. It works fine and there seems no reason to use non credit-based
> implementation.
>
> Regards,
> Xiaogang
>
> Zhu Zhu <re...@gmail.com> 于2019年10月19日周六 下午3:01写道：
>
>> +1 to drop the non credit-based flow control.
>> We have turned to credit-based flow control for long in production. It has
>> been good for all our cases.
>> The non credit-based flow control code has been a burden when we are trying
>> to change the network stack code for new features.
>>
>> Thanks,
>> Zhu Zhu
>>
>>
>> Biao Liu <mm...@gmail.com> 于2019年10月10日周四 下午5:45写道：
>>
>>> Thanks for start this survey, Piotr.
>>>
>>> We have benefitted from credit-based flow control a lot. I can't figure
>> out
>>> a reason to use non credit-based model.
>>> I think we have kept the older code paths long enough (1.5 -> 1.9).
>> That's
>>> a big burden to maintain. Especially there are a lot duplicated codes
>>> between credit-based and non credit-based model.
>>>
>>> So +1 to do the cleanup.
>>>
>>> Thanks,
>>> Biao /'bɪ.aʊ/
>>>
>>>
>>>
>>> On Thu, 10 Oct 2019 at 11:15, zhijiang <wangzhijiang999@aliyun.com
>>> .invalid>
>>> wrote:
>>>
>>>> Thanks for bringing this survey Piotr.
>>>>
>>>> Actually I was also trying to dropping the non credit-based code path
>>> from
>>>> release-1.9, and now I think it is the proper time to do it motivated
>> by
>>>> [3].
>>>> The credit-based mode is as default from Flink 1.5 and it has been
>>>> verified to be stable and reliable in many versions. In Alibaba we are
>>>> always using the default credit-based mode in all products.
>>>> It can reduce much overhead of maintaining non credit-based code path,
>> so
>>>> +1 from my side to drop it.
>>>>
>>>> Best,
>>>> Zhijiang
>>>> ------------------------------------------------------------------
>>>> From:Piotr Nowojski <pi...@ververica.com>
>>>> Send Time:2019年10月2日(星期三) 17:01
>>>> To:dev <de...@flink.apache.org>
>>>> Subject:[SURVEY] Dropping non Credit-based Flow Control
>>>>
>>>> Hi,
>>>>
>>>> In Flink 1.5 we have introduced Credit-based Flow Control [1] in the
>>>> network stack. Back then we were aware about potential downsides of it
>>> [2]
>>>> and we decided to keep the old model in the code base ( configurable by
>>>> setting `taskmanager.network.credit-model: false` ). Now, that we are
>>>> about to modify internals of the network stack again [3], it might be a
>>>> good time to clean up the code and remove the older code paths.
>>>>
>>>> Is anyone still using the non default non Credit-based model (
>>>> `taskmanager.network.credit-model: false`)? If so, why?
>>>>
>>>> Piotrek
>>>>
>>>> [1] https://flink.apache.org/2019/06/05/flink-network-stack.html <
>>>> https://flink.apache.org/2019/06/05/flink-network-stack.html>
>>>> [2]
>>>>
>>>
>> https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch
>>>> <
>>>>
>>>
>> https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch
>>>>>
>>>> [3]
>>>>
>>>
>> https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E
>>>> <
>>>>
>>>
>> https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E
>>>>>
>>>>
>>>>
>>>
>>
>
--
Nico Kruber | Solutions Architect
Follow us @VervericaData Ververica
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Tony) Cheng
Re: [SURVEY] Dropping non Credit-based Flow Control
Posted by SHI Xiaogang <sh...@gmail.com>.
+1
Credit-based flow control has long been used in our production environment
as well. It works fine and there seems no reason to use non credit-based
implementation.
Regards,
Xiaogang
Zhu Zhu <re...@gmail.com> 于2019年10月19日周六 下午3:01写道：
> +1 to drop the non credit-based flow control.
> We have turned to credit-based flow control for long in production. It has
> been good for all our cases.
> The non credit-based flow control code has been a burden when we are trying
> to change the network stack code for new features.
>
> Thanks,
> Zhu Zhu
>
>
> Biao Liu <mm...@gmail.com> 于2019年10月10日周四 下午5:45写道：
>
> > Thanks for start this survey, Piotr.
> >
> > We have benefitted from credit-based flow control a lot. I can't figure
> out
> > a reason to use non credit-based model.
> > I think we have kept the older code paths long enough (1.5 -> 1.9).
> That's
> > a big burden to maintain. Especially there are a lot duplicated codes
> > between credit-based and non credit-based model.
> >
> > So +1 to do the cleanup.
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Thu, 10 Oct 2019 at 11:15, zhijiang <wangzhijiang999@aliyun.com
> > .invalid>
> > wrote:
> >
> > > Thanks for bringing this survey Piotr.
> > >
> > > Actually I was also trying to dropping the non credit-based code path
> > from
> > > release-1.9, and now I think it is the proper time to do it motivated
> by
> > > [3].
> > > The credit-based mode is as default from Flink 1.5 and it has been
> > > verified to be stable and reliable in many versions. In Alibaba we are
> > > always using the default credit-based mode in all products.
> > > It can reduce much overhead of maintaining non credit-based code path,
> so
> > > +1 from my side to drop it.
> > >
> > > Best,
> > > Zhijiang
> > > ------------------------------------------------------------------
> > > From:Piotr Nowojski <pi...@ververica.com>
> > > Send Time:2019年10月2日(星期三) 17:01
> > > To:dev <de...@flink.apache.org>
> > > Subject:[SURVEY] Dropping non Credit-based Flow Control
> > >
> > > Hi,
> > >
> > > In Flink 1.5 we have introduced Credit-based Flow Control [1] in the
> > > network stack. Back then we were aware about potential downsides of it
> > [2]
> > > and we decided to keep the old model in the code base ( configurable by
> > > setting `taskmanager.network.credit-model: false` ). Now, that we are
> > > about to modify internals of the network stack again [3], it might be a
> > > good time to clean up the code and remove the older code paths.
> > >
> > > Is anyone still using the non default non Credit-based model (
> > > `taskmanager.network.credit-model: false`)? If so, why?
> > >
> > > Piotrek
> > >
> > > [1] https://flink.apache.org/2019/06/05/flink-network-stack.html <
> > > https://flink.apache.org/2019/06/05/flink-network-stack.html>
> > > [2]
> > >
> >
> https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch
> > > <
> > >
> >
> https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch
> > > >
> > > [3]
> > >
> >
> https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E
> > > <
> > >
> >
> https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E
> > > >
> > >
> > >
> >
>
Re: [SURVEY] Dropping non Credit-based Flow Control
Posted by Zhu Zhu <re...@gmail.com>.
+1 to drop the non credit-based flow control.
We have turned to credit-based flow control for long in production. It has
been good for all our cases.
The non credit-based flow control code has been a burden when we are trying
to change the network stack code for new features.
Thanks,
Zhu Zhu
Biao Liu <mm...@gmail.com> 于2019年10月10日周四 下午5:45写道：
> Thanks for start this survey, Piotr.
>
> We have benefitted from credit-based flow control a lot. I can't figure out
> a reason to use non credit-based model.
> I think we have kept the older code paths long enough (1.5 -> 1.9). That's
> a big burden to maintain. Especially there are a lot duplicated codes
> between credit-based and non credit-based model.
>
> So +1 to do the cleanup.
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Thu, 10 Oct 2019 at 11:15, zhijiang <wangzhijiang999@aliyun.com
> .invalid>
> wrote:
>
> > Thanks for bringing this survey Piotr.
> >
> > Actually I was also trying to dropping the non credit-based code path
> from
> > release-1.9, and now I think it is the proper time to do it motivated by
> > [3].
> > The credit-based mode is as default from Flink 1.5 and it has been
> > verified to be stable and reliable in many versions. In Alibaba we are
> > always using the default credit-based mode in all products.
> > It can reduce much overhead of maintaining non credit-based code path, so
> > +1 from my side to drop it.
> >
> > Best,
> > Zhijiang
> > ------------------------------------------------------------------
> > From:Piotr Nowojski <pi...@ververica.com>
> > Send Time:2019年10月2日(星期三) 17:01
> > To:dev <de...@flink.apache.org>
> > Subject:[SURVEY] Dropping non Credit-based Flow Control
> >
> > Hi,
> >
> > In Flink 1.5 we have introduced Credit-based Flow Control [1] in the
> > network stack. Back then we were aware about potential downsides of it
> [2]
> > and we decided to keep the old model in the code base ( configurable by
> > setting `taskmanager.network.credit-model: false` ). Now, that we are
> > about to modify internals of the network stack again [3], it might be a
> > good time to clean up the code and remove the older code paths.
> >
> > Is anyone still using the non default non Credit-based model (
> > `taskmanager.network.credit-model: false`)? If so, why?
> >
> > Piotrek
> >
> > [1] https://flink.apache.org/2019/06/05/flink-network-stack.html <
> > https://flink.apache.org/2019/06/05/flink-network-stack.html>
> > [2]
> >
> https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch
> > <
> >
> https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch
> > >
> > [3]
> >
> https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E
> > <
> >
> https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E
> > >
> >
> >
>
Re: [SURVEY] Dropping non Credit-based Flow Control
Posted by Biao Liu <mm...@gmail.com>.
Thanks for start this survey, Piotr.
We have benefitted from credit-based flow control a lot. I can't figure out
a reason to use non credit-based model.
I think we have kept the older code paths long enough (1.5 -> 1.9). That's
a big burden to maintain. Especially there are a lot duplicated codes
between credit-based and non credit-based model.
So +1 to do the cleanup.
Thanks,
Biao /'bɪ.aʊ/
On Thu, 10 Oct 2019 at 11:15, zhijiang <wa...@aliyun.com.invalid>
wrote:
> Thanks for bringing this survey Piotr.
>
> Actually I was also trying to dropping the non credit-based code path from
> release-1.9, and now I think it is the proper time to do it motivated by
> [3].
> The credit-based mode is as default from Flink 1.5 and it has been
> verified to be stable and reliable in many versions. In Alibaba we are
> always using the default credit-based mode in all products.
> It can reduce much overhead of maintaining non credit-based code path, so
> +1 from my side to drop it.
>
> Best,
> Zhijiang
> ------------------------------------------------------------------
> From:Piotr Nowojski <pi...@ververica.com>
> Send Time:2019年10月2日(星期三) 17:01
> To:dev <de...@flink.apache.org>
> Subject:[SURVEY] Dropping non Credit-based Flow Control
>
> Hi,
>
> In Flink 1.5 we have introduced Credit-based Flow Control [1] in the
> network stack. Back then we were aware about potential downsides of it [2]
> and we decided to keep the old model in the code base ( configurable by
> setting `taskmanager.network.credit-model: false` ). Now, that we are
> about to modify internals of the network stack again [3], it might be a
> good time to clean up the code and remove the older code paths.
>
> Is anyone still using the non default non Credit-based model (
> `taskmanager.network.credit-model: false`)? If so, why?
>
> Piotrek
>
> [1] https://flink.apache.org/2019/06/05/flink-network-stack.html <
> https://flink.apache.org/2019/06/05/flink-network-stack.html>
> [2]
> https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch
> <
> https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch
> >
> [3]
> https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E
> <
> https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E
> >
>
>
Re: [SURVEY] Dropping non Credit-based Flow Control
Posted by zhijiang <wa...@aliyun.com.INVALID>.
Thanks for bringing this survey Piotr.
Actually I was also trying to dropping the non credit-based code path from release-1.9, and now I think it is the proper time to do it motivated by [3].
The credit-based mode is as default from Flink 1.5 and it has been verified to be stable and reliable in many versions. In Alibaba we are always using the default credit-based mode in all products.
It can reduce much overhead of maintaining non credit-based code path, so +1 from my side to drop it.
Best,
Zhijiang
------------------------------------------------------------------
From:Piotr Nowojski <pi...@ververica.com>
Send Time:2019年10月2日(星期三) 17:01
To:dev <de...@flink.apache.org>
Subject:[SURVEY] Dropping non Credit-based Flow Control
Hi,
In Flink 1.5 we have introduced Credit-based Flow Control [1] in the network stack. Back then we were aware about potential downsides of it [2] and we decided to keep the old model in the code base ( configurable by setting `taskmanager.network.credit-model: false` ). Now, that we are about to modify internals of the network stack again [3], it might be a good time to clean up the code and remove the older code paths.
Is anyone still using the non default non Credit-based model ( `taskmanager.network.credit-model: false`)? If so, why?
Piotrek
[1] https://flink.apache.org/2019/06/05/flink-network-stack.html <https://flink.apache.org/2019/06/05/flink-network-stack.html>
[2] https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch <https://flink.apache.org/2019/06/05/flink-network-stack.html#what-do-we-gain-where-is-the-catch>
[3] https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E <https://lists.apache.org/thread.html/a2b58b7b2b24b9bd4814b2aa51d2fc44b08a919eddbb5b1256be5b6a@%3Cdev.flink.apache.org%3E>