You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Gwenhael Pasquiers <gw...@ericsson.com> on 2016/02/04 10:56:32 UTC
Internal buffers supervision and yarn vCPUs
Hi,
I’ve got two more questions on different topic…
First one :
Is there a way to monitor the buffers status. In order to find bottleneck in our application we though it could be usefull to be able to have a look at the different exchange buffers’ status. To know if they are full (or as an example if a mapper had to wait before being able to push it’s data into the buffer). That way we can know where the bottleneck is.
Second one :
On type of resources on yarn is vCPU. In flink 0.8 there was a “-tmc” argument that allowed to specify the number of vCPU per task manager. We cannot find it anymore. Was it removed ? Is there another way to set the number of vCPU. Or did it became useless ?
Thanks in advance.
Gwen’
Re: Internal buffers supervision and yarn vCPUs
Posted by Ufuk Celebi <uc...@apache.org>.
> On 04 Feb 2016, at 12:02, Gwenhael Pasquiers <gw...@ericsson.com> wrote:
>
> Ok thanks !
>
> All that’s left is to wait then.
If you have spare time and are working with the current snapshot version, it would be great to get some feedback on the pull request. :-)
– Ufuk
RE: Internal buffers supervision and yarn vCPUs
Posted by Gwenhael Pasquiers <gw...@ericsson.com>.
Ok thanks !
All that’s left is to wait then.
B.R.
From: ewenstephan@gmail.com [mailto:ewenstephan@gmail.com] On Behalf Of Stephan Ewen
Sent: jeudi 4 février 2016 11:19
To: user@flink.apache.org
Subject: Re: Internal buffers supervision and yarn vCPUs
Concerning the first question:
What you are looking for is backpressure monitoring. If a task cannot push its data to the next task, it is backpressured.
This pull request adds a first version of backpressure monitoring: https://github.com/apache/flink/pull/1578
We will try and get it merged soon!
On Thu, Feb 4, 2016 at 11:03 AM, Robert Metzger <rm...@apache.org>> wrote:
Hi Gwen,
let me answer the second question: There is a JIRA to reintroduce the configuration parameter: https://issues.apache.org/jira/browse/FLINK-2213. I will try to get a fix for this into the 1.0 release.
I think I removed back then because users were unable to define the number of vcores independently of the number of slots ... and too many users were running into issues with the yarn scheduler (containers were not started because there were no CPU resources available anymore).
On Thu, Feb 4, 2016 at 10:56 AM, Gwenhael Pasquiers <gw...@ericsson.com>> wrote:
Hi,
I’ve got two more questions on different topic…
First one :
Is there a way to monitor the buffers status. In order to find bottleneck in our application we though it could be usefull to be able to have a look at the different exchange buffers’ status. To know if they are full (or as an example if a mapper had to wait before being able to push it’s data into the buffer). That way we can know where the bottleneck is.
Second one :
On type of resources on yarn is vCPU. In flink 0.8 there was a “-tmc” argument that allowed to specify the number of vCPU per task manager. We cannot find it anymore. Was it removed ? Is there another way to set the number of vCPU. Or did it became useless ?
Thanks in advance.
Gwen’
Re: Internal buffers supervision and yarn vCPUs
Posted by Stephan Ewen <se...@apache.org>.
Concerning the first question:
What you are looking for is backpressure monitoring. If a task cannot push
its data to the next task, it is backpressured.
This pull request adds a first version of backpressure monitoring:
https://github.com/apache/flink/pull/1578
We will try and get it merged soon!
On Thu, Feb 4, 2016 at 11:03 AM, Robert Metzger <rm...@apache.org> wrote:
> Hi Gwen,
>
> let me answer the second question: There is a JIRA to reintroduce the
> configuration parameter: https://issues.apache.org/jira/browse/FLINK-2213.
> I will try to get a fix for this into the 1.0 release.
>
> I think I removed back then because users were unable to define the number
> of vcores independently of the number of slots ... and too many users were
> running into issues with the yarn scheduler (containers were not started
> because there were no CPU resources available anymore).
>
>
>
> On Thu, Feb 4, 2016 at 10:56 AM, Gwenhael Pasquiers <
> gwenhael.pasquiers@ericsson.com> wrote:
>
>> Hi,
>>
>>
>>
>> I’ve got two more questions on different topic…
>>
>>
>>
>> First one :
>>
>> Is there a way to monitor the buffers status. In order to find
>> bottleneck in our application we though it could be usefull to be able to
>> have a look at the different exchange buffers’ status. To know if they are
>> full (or as an example if a mapper had to wait before being able to push
>> it’s data into the buffer). That way we can know where the bottleneck is.
>>
>>
>>
>> Second one :
>>
>> On type of resources on yarn is vCPU. In flink 0.8 there was a “-tmc”
>> argument that allowed to specify the number of vCPU per task manager. We
>> cannot find it anymore. Was it removed ? Is there another way to set the
>> number of vCPU. Or did it became useless ?
>>
>>
>>
>> Thanks in advance.
>>
>>
>>
>> Gwen’
>>
>
>
Re: Internal buffers supervision and yarn vCPUs
Posted by Robert Metzger <rm...@apache.org>.
Hi Gwen,
let me answer the second question: There is a JIRA to reintroduce the
configuration parameter: https://issues.apache.org/jira/browse/FLINK-2213.
I will try to get a fix for this into the 1.0 release.
I think I removed back then because users were unable to define the number
of vcores independently of the number of slots ... and too many users were
running into issues with the yarn scheduler (containers were not started
because there were no CPU resources available anymore).
On Thu, Feb 4, 2016 at 10:56 AM, Gwenhael Pasquiers <
gwenhael.pasquiers@ericsson.com> wrote:
> Hi,
>
>
>
> I’ve got two more questions on different topic…
>
>
>
> First one :
>
> Is there a way to monitor the buffers status. In order to find bottleneck
> in our application we though it could be usefull to be able to have a look
> at the different exchange buffers’ status. To know if they are full (or as
> an example if a mapper had to wait before being able to push it’s data into
> the buffer). That way we can know where the bottleneck is.
>
>
>
> Second one :
>
> On type of resources on yarn is vCPU. In flink 0.8 there was a “-tmc”
> argument that allowed to specify the number of vCPU per task manager. We
> cannot find it anymore. Was it removed ? Is there another way to set the
> number of vCPU. Or did it became useless ?
>
>
>
> Thanks in advance.
>
>
>
> Gwen’
>