You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by narasimha <sw...@gmail.com> on 2020/08/10 11:06:03 UTC
Tools for Flink Job performance testing
Hi,
I'm new to the streaming world, checking on Performance testing tools. Are
there any recommended Performance testing tools for Flink?
--
A.Narasimha Swamy
Re: Tools for Flink Job performance testing
Posted by narasimha <sw...@gmail.com>.
Thanks, Arvid.
The guide was helpful in how to start working with Flink. I'm currently
exploring SQL/Table API.
Will surely come back for queries on it.
On Thu, Aug 13, 2020 at 1:25 PM Arvid Heise <ar...@ververica.com> wrote:
> Hi,
>
> performance testing is quite vague. Usually you start by writing a small
> first version of your pipeline and check how the well computation scales on
> your data. Flink's web UI [1] already helps quite well for the first time.
> Usually you'd also add some metric system and look for advanced metrics in
> there [2].
>
> Now, you might be satisfied with your current solution and just start to
> extend it. Or you feel that it's not fast enough, scaling well enough. Then
> you can tweak your pipeline and perform smaller performance tests on your
> user code. Doing performance tests on the whole pipeline [3] would probably
> be hard if you just started as you also need to understand internals of
> Flink.
>
> I also recommend to use Table API / SQL instead of DataStream if your
> application can be expressed well in relational operations. Table API
> already applies a wide range of optimizations that are much harder to
> implement manually in DataStream API. Table API will also bring your
> noticeable performance improvements over time when you update to a newer
> Flink version.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/back_pressure.html
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html
> [3]
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/application_profiling.html
>
> On Mon, Aug 10, 2020 at 1:06 PM narasimha <sw...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm new to the streaming world, checking on Performance testing tools.
>> Are there any recommended Performance testing tools for Flink?
>>
>> --
>> A.Narasimha Swamy
>>
>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> <https://www.ververica.com/>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>
--
A.Narasimha Swamy
Re: Tools for Flink Job performance testing
Posted by Arvid Heise <ar...@ververica.com>.
Hi,
performance testing is quite vague. Usually you start by writing a small
first version of your pipeline and check how the well computation scales on
your data. Flink's web UI [1] already helps quite well for the first time.
Usually you'd also add some metric system and look for advanced metrics in
there [2].
Now, you might be satisfied with your current solution and just start to
extend it. Or you feel that it's not fast enough, scaling well enough. Then
you can tweak your pipeline and perform smaller performance tests on your
user code. Doing performance tests on the whole pipeline [3] would probably
be hard if you just started as you also need to understand internals of
Flink.
I also recommend to use Table API / SQL instead of DataStream if your
application can be expressed well in relational operations. Table API
already applies a wide range of optimizations that are much harder to
implement manually in DataStream API. Table API will also bring your
noticeable performance improvements over time when you update to a newer
Flink version.
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/back_pressure.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html
[3]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/application_profiling.html
On Mon, Aug 10, 2020 at 1:06 PM narasimha <sw...@gmail.com> wrote:
> Hi,
>
> I'm new to the streaming world, checking on Performance testing tools.
> Are there any recommended Performance testing tools for Flink?
>
> --
> A.Narasimha Swamy
>
--
Arvid Heise | Senior Java Developer
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng