You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Stefan Bunk <st...@googlemail.com> on 2014/10/17 20:29:01 UTC
Measure Operator Runtime in Local Execution Environment
Hey Flinkers!
After updating to 0.7-SNAPSHOT I noticed that the runtime of my program
when running in the local environment went up by 25%.
That might be due to the refactoring I had to do.
Is there some way to get a report of the runtime of each operator, so that
I can narrow down the issue?
Greetings
Stefan
Re: Measure Operator Runtime in Local Execution Environment
Posted by Robert Metzger <rm...@apache.org>.
Hi,
were the reference jobs implemented using the old Scala API? Maybe it is
faster than the new Scala API?
The webinterface of Flink shows (in the history view) a breakdown of the
operator runtimes. You can also set the log level to "INFO" and calculate
the operator runtimes yourself.
Remember that operators often run in parallel, which makes it harder to
understand the performance behavior.
Please share with us what you found out in your analysis.
Robert
On Fri, Oct 17, 2014 at 8:29 PM, Stefan Bunk <st...@googlemail.com>
wrote:
> Hey Flinkers!
>
> After updating to 0.7-SNAPSHOT I noticed that the runtime of my program
> when running in the local environment went up by 25%.
> That might be due to the refactoring I had to do.
> Is there some way to get a report of the runtime of each operator, so that
> I can narrow down the issue?
>
> Greetings
> Stefan
>
Re: Measure Operator Runtime in Local Execution Environment
Posted by Stefan Bunk <st...@googlemail.com>.
See here:
https://www.dropbox.com/sh/r041egk3dt9eev3/AADlkAyJBlPXSANY4rwOjCBBa?dl=0
On Fri, Oct 24, 2014 at 4:30 PM, Stephan Ewen <se...@apache.org> wrote:
> Hey Stefan!
>
> The Apache mailing lists do not take attachments. Can you upload them
> somewhere (e.g. public dropbox folder or so?)
>
> Stephan
>
>
> On Fri, Oct 24, 2014 at 3:18 PM, Stefan Bunk <st...@googlemail.com>
> wrote:
>
>> Hi,
>>
>> yes, I used the old Scala API, and now ported to the new one.
>> One thing I noticed is the different memory behaviour. 0.7 seems to
>> allocate more memory (but when I look at the documentation for 0.6 and 0.7
>> the value of taskmanager.memory.fraction did not change), and the garbage
>> collection (therefore?) runs more often.
>> Please find attached two profiling screenshots of the same flink program,
>> in 0.6 and 0.7.
>> Is there some way to set the taskmanager.memory.fraction for the local
>> execution?
>>
>> Stefan
>>
>>
>> On Sat, Oct 18, 2014 at 8:52 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> Hi!
>>>
>>> This may be due to many aspects: different startup behavior of the local
>>> environment. Different deployment behavior, different shutdown behavior.
>>>
>>> Do you know where the time is spent?
>>>
>>> Stephan
>>>
>>
>>
>
Re: Measure Operator Runtime in Local Execution Environment
Posted by Stephan Ewen <se...@apache.org>.
Hey Stefan!
The Apache mailing lists do not take attachments. Can you upload them
somewhere (e.g. public dropbox folder or so?)
Stephan
On Fri, Oct 24, 2014 at 3:18 PM, Stefan Bunk <st...@googlemail.com>
wrote:
> Hi,
>
> yes, I used the old Scala API, and now ported to the new one.
> One thing I noticed is the different memory behaviour. 0.7 seems to
> allocate more memory (but when I look at the documentation for 0.6 and 0.7
> the value of taskmanager.memory.fraction did not change), and the garbage
> collection (therefore?) runs more often.
> Please find attached two profiling screenshots of the same flink program,
> in 0.6 and 0.7.
> Is there some way to set the taskmanager.memory.fraction for the local
> execution?
>
> Stefan
>
>
> On Sat, Oct 18, 2014 at 8:52 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi!
>>
>> This may be due to many aspects: different startup behavior of the local
>> environment. Different deployment behavior, different shutdown behavior.
>>
>> Do you know where the time is spent?
>>
>> Stephan
>>
>
>
Re: Measure Operator Runtime in Local Execution Environment
Posted by Stefan Bunk <st...@googlemail.com>.
Hi,
yes, I used the old Scala API, and now ported to the new one.
One thing I noticed is the different memory behaviour. 0.7 seems to
allocate more memory (but when I look at the documentation for 0.6 and 0.7
the value of taskmanager.memory.fraction did not change), and the garbage
collection (therefore?) runs more often.
Please find attached two profiling screenshots of the same flink program,
in 0.6 and 0.7.
Is there some way to set the taskmanager.memory.fraction for the local
execution?
Stefan
On Sat, Oct 18, 2014 at 8:52 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi!
>
> This may be due to many aspects: different startup behavior of the local
> environment. Different deployment behavior, different shutdown behavior.
>
> Do you know where the time is spent?
>
> Stephan
>
Re: Measure Operator Runtime in Local Execution Environment
Posted by Stephan Ewen <se...@apache.org>.
Hi!
This may be due to many aspects: different startup behavior of the local
environment. Different deployment behavior, different shutdown behavior.
Do you know where the time is spent?
Stephan