You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Abdul Qadeer <qu...@gmail.com> on 2018/05/30 21:26:38 UTC
Multiple Task Slots support in Flink 1.5
Hi!
I came across the following point in release notes
<https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html>
of 1.5 version:
"The allocation of TaskManagers with multiple slots is not fully supported
yet."
Does this mean the support for it will come as a patch for 1.5? or will it
be in the next stable release?
If I use legacy mode, will that support multiple slots per TaskManager?, or
is it only the deployment change that will get affected?
Re: Multiple Task Slots support in Flink 1.5
Posted by Till Rohrmann <tr...@apache.org>.
For standalone mode it works for the legacy and the new mode. Only when
using Yarn or Mesos, it can happen that you temporarily allocate too many
container. The unused container will, however, be freed if they idle too
long.
Cheers,
Till
On Fri, Jun 1, 2018, 21:52 Abdul Qadeer <qu...@gmail.com> wrote:
> Thank you for creating this Till, so as I understand this won't be
> supported (for standalone cluster mode) even if legacy mode is enabled?
>
> On Fri, Jun 1, 2018 at 6:21 AM, Till Rohrmann <tr...@apache.org>
> wrote:
>
>> FYI: https://issues.apache.org/jira/browse/FLINK-9455 tracks the problem
>> with the multi slot support for the SlotManager.
>>
>> Cheers,
>> Till
>>
>> On Fri, Jun 1, 2018 at 10:10 AM, Abdul Qadeer <qu...@gmail.com>
>> wrote:
>>
>>> Hi Kien!
>>>
>>> Thank you for sharing your experience. Do you have Flink deployed in
>>> Standalone Cluster mode or K8S/Docker mode?
>>>
>>> On Fri, 1 Jun 2018 at 00:55, Fabian Hueske <fh...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> The release notes state that "multiple slots are not *fully* supported".
>>>>
>>>> In Flink 1.5.0, the configured number of slots is ignored when
>>>> requesting containers for TaskManagers from a resource manager, i.e., Flink
>>>> assumes TMs with 1 slot.
>>>> Hence, Flink request too many containers and starts too many TMs, but
>>>> each TM is started with the correct number of slots.
>>>> All unused containers are returned after a configurable time out.
>>>>
>>>> The problem can be prevented by configuring 1 slot per TM.
>>>>
>>>> Best, Fabian
>>>>
>>>> 2018-05-31 14:12 GMT+02:00 Kien Truong <du...@gmail.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> We're using multiple slots per TaskManager with legacy mode, and
>>>>> everything works fine.
>>>>>
>>>>> For the new default mode, it also seems to works for us, so I'm not
>>>>> sure what is not supported. May be someone from Flink team could clarify.
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Kien
>>>>>
>>>>>
>>>>> On 5/31/2018 4:26 AM, Abdul Qadeer wrote:
>>>>>
>>>>> Hi!
>>>>>
>>>>> I came across the following point in release notes
>>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html>
>>>>> of 1.5 version:
>>>>>
>>>>> "The allocation of TaskManagers with multiple slots is not fully
>>>>> supported yet."
>>>>>
>>>>> Does this mean the support for it will come as a patch for 1.5? or
>>>>> will it be in the next stable release?
>>>>> If I use legacy mode, will that support multiple slots per
>>>>> TaskManager?, or is it only the deployment change that will get affected?
>>>>>
>>>>>
>>>>
>>
>
Re: Multiple Task Slots support in Flink 1.5
Posted by Abdul Qadeer <qu...@gmail.com>.
Thank you for creating this Till, so as I understand this won't be
supported (for standalone cluster mode) even if legacy mode is enabled?
On Fri, Jun 1, 2018 at 6:21 AM, Till Rohrmann <tr...@apache.org> wrote:
> FYI: https://issues.apache.org/jira/browse/FLINK-9455 tracks the problem
> with the multi slot support for the SlotManager.
>
> Cheers,
> Till
>
> On Fri, Jun 1, 2018 at 10:10 AM, Abdul Qadeer <qu...@gmail.com>
> wrote:
>
>> Hi Kien!
>>
>> Thank you for sharing your experience. Do you have Flink deployed in
>> Standalone Cluster mode or K8S/Docker mode?
>>
>> On Fri, 1 Jun 2018 at 00:55, Fabian Hueske <fh...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> The release notes state that "multiple slots are not *fully* supported".
>>>
>>> In Flink 1.5.0, the configured number of slots is ignored when
>>> requesting containers for TaskManagers from a resource manager, i.e., Flink
>>> assumes TMs with 1 slot.
>>> Hence, Flink request too many containers and starts too many TMs, but
>>> each TM is started with the correct number of slots.
>>> All unused containers are returned after a configurable time out.
>>>
>>> The problem can be prevented by configuring 1 slot per TM.
>>>
>>> Best, Fabian
>>>
>>> 2018-05-31 14:12 GMT+02:00 Kien Truong <du...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> We're using multiple slots per TaskManager with legacy mode, and
>>>> everything works fine.
>>>>
>>>> For the new default mode, it also seems to works for us, so I'm not
>>>> sure what is not supported. May be someone from Flink team could clarify.
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Kien
>>>>
>>>>
>>>> On 5/31/2018 4:26 AM, Abdul Qadeer wrote:
>>>>
>>>> Hi!
>>>>
>>>> I came across the following point in release notes
>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html>
>>>> of 1.5 version:
>>>>
>>>> "The allocation of TaskManagers with multiple slots is not fully
>>>> supported yet."
>>>>
>>>> Does this mean the support for it will come as a patch for 1.5? or will
>>>> it be in the next stable release?
>>>> If I use legacy mode, will that support multiple slots per
>>>> TaskManager?, or is it only the deployment change that will get affected?
>>>>
>>>>
>>>
>
Re: Multiple Task Slots support in Flink 1.5
Posted by Till Rohrmann <tr...@apache.org>.
FYI: https://issues.apache.org/jira/browse/FLINK-9455 tracks the problem
with the multi slot support for the SlotManager.
Cheers,
Till
On Fri, Jun 1, 2018 at 10:10 AM, Abdul Qadeer <qu...@gmail.com> wrote:
> Hi Kien!
>
> Thank you for sharing your experience. Do you have Flink deployed in
> Standalone Cluster mode or K8S/Docker mode?
>
> On Fri, 1 Jun 2018 at 00:55, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi,
>>
>> The release notes state that "multiple slots are not *fully* supported".
>>
>> In Flink 1.5.0, the configured number of slots is ignored when requesting
>> containers for TaskManagers from a resource manager, i.e., Flink assumes
>> TMs with 1 slot.
>> Hence, Flink request too many containers and starts too many TMs, but
>> each TM is started with the correct number of slots.
>> All unused containers are returned after a configurable time out.
>>
>> The problem can be prevented by configuring 1 slot per TM.
>>
>> Best, Fabian
>>
>> 2018-05-31 14:12 GMT+02:00 Kien Truong <du...@gmail.com>:
>>
>>> Hi,
>>>
>>> We're using multiple slots per TaskManager with legacy mode, and
>>> everything works fine.
>>>
>>> For the new default mode, it also seems to works for us, so I'm not sure
>>> what is not supported. May be someone from Flink team could clarify.
>>>
>>>
>>> Best regards,
>>>
>>> Kien
>>>
>>>
>>> On 5/31/2018 4:26 AM, Abdul Qadeer wrote:
>>>
>>> Hi!
>>>
>>> I came across the following point in release notes
>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html>
>>> of 1.5 version:
>>>
>>> "The allocation of TaskManagers with multiple slots is not fully
>>> supported yet."
>>>
>>> Does this mean the support for it will come as a patch for 1.5? or will
>>> it be in the next stable release?
>>> If I use legacy mode, will that support multiple slots per TaskManager?,
>>> or is it only the deployment change that will get affected?
>>>
>>>
>>
Re: Multiple Task Slots support in Flink 1.5
Posted by Abdul Qadeer <qu...@gmail.com>.
Hi Kien!
Thank you for sharing your experience. Do you have Flink deployed in
Standalone Cluster mode or K8S/Docker mode?
On Fri, 1 Jun 2018 at 00:55, Fabian Hueske <fh...@gmail.com> wrote:
> Hi,
>
> The release notes state that "multiple slots are not *fully* supported".
>
> In Flink 1.5.0, the configured number of slots is ignored when requesting
> containers for TaskManagers from a resource manager, i.e., Flink assumes
> TMs with 1 slot.
> Hence, Flink request too many containers and starts too many TMs, but each
> TM is started with the correct number of slots.
> All unused containers are returned after a configurable time out.
>
> The problem can be prevented by configuring 1 slot per TM.
>
> Best, Fabian
>
> 2018-05-31 14:12 GMT+02:00 Kien Truong <du...@gmail.com>:
>
>> Hi,
>>
>> We're using multiple slots per TaskManager with legacy mode, and
>> everything works fine.
>>
>> For the new default mode, it also seems to works for us, so I'm not sure
>> what is not supported. May be someone from Flink team could clarify.
>>
>>
>> Best regards,
>>
>> Kien
>>
>>
>> On 5/31/2018 4:26 AM, Abdul Qadeer wrote:
>>
>> Hi!
>>
>> I came across the following point in release notes
>> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html>
>> of 1.5 version:
>>
>> "The allocation of TaskManagers with multiple slots is not fully
>> supported yet."
>>
>> Does this mean the support for it will come as a patch for 1.5? or will
>> it be in the next stable release?
>> If I use legacy mode, will that support multiple slots per TaskManager?,
>> or is it only the deployment change that will get affected?
>>
>>
>
Re: Multiple Task Slots support in Flink 1.5
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
The release notes state that "multiple slots are not *fully* supported".
In Flink 1.5.0, the configured number of slots is ignored when requesting
containers for TaskManagers from a resource manager, i.e., Flink assumes
TMs with 1 slot.
Hence, Flink request too many containers and starts too many TMs, but each
TM is started with the correct number of slots.
All unused containers are returned after a configurable time out.
The problem can be prevented by configuring 1 slot per TM.
Best, Fabian
2018-05-31 14:12 GMT+02:00 Kien Truong <du...@gmail.com>:
> Hi,
>
> We're using multiple slots per TaskManager with legacy mode, and
> everything works fine.
>
> For the new default mode, it also seems to works for us, so I'm not sure
> what is not supported. May be someone from Flink team could clarify.
>
>
> Best regards,
>
> Kien
>
>
> On 5/31/2018 4:26 AM, Abdul Qadeer wrote:
>
> Hi!
>
> I came across the following point in release notes
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html>
> of 1.5 version:
>
> "The allocation of TaskManagers with multiple slots is not fully supported
> yet."
>
> Does this mean the support for it will come as a patch for 1.5? or will it
> be in the next stable release?
> If I use legacy mode, will that support multiple slots per TaskManager?,
> or is it only the deployment change that will get affected?
>
>
Re: Multiple Task Slots support in Flink 1.5
Posted by Kien Truong <du...@gmail.com>.
Hi,
We're using multiple slots per TaskManager with legacy mode, and
everything works fine.
For the new default mode, it also seems to works for us, so I'm not sure
what is not supported. May be someone from Flink team could clarify.
Best regards,
Kien
On 5/31/2018 4:26 AM, Abdul Qadeer wrote:
> Hi!
>
> I came across the following point in release notes
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html>
> of 1.5 version:
>
> "The allocation of TaskManagers with multiple slots is not fully
> supported yet."
>
> Does this mean the support for it will come as a patch for 1.5? or
> will it be in the next stable release?
> If I use legacy mode, will that support multiple slots per
> TaskManager?, or is it only the deployment change that will get affected?