You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Yu Chen <yu...@gmail.com> on 2023/10/09 09:24:17 UTC
[DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Hi all,
Yun Tang and I are opening this thread to discuss our proposal to integrate
async-profiler's capabilities for profiling taskmananger (e.g., generating
flame graphs) in the Flink Web [1].
Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
sampling task threads. The results generated in such way missing the
relevant stack of java threads and system calls. The async-profiler[2] is a
low-overhead sampling profiler for Java, but the steps to use it in the
production environment are cumbersome and suffer from permissions and
security risks.
Therefore, we propose adding rest APIs to provide the capability to invoke
async-profiler on multiple platforms through JNI, which can be easily
operated on Web UI. This enhancement will improve the efficiency and
experience of Flink users in identifying performance bottlenecks.
Please refer to the FLIP document for more details about the proposed design
and implementation. We welcome any feedback and opinions on this proposal.
[1] FLIP-375: Built-in cross-platform powerful java profiler on
taskmanagers - Apache Flink - Apache Software Foundation
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers>
[2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP profiler
for Java featuring AsyncGetCallTrace + perf_events
<https://github.com/async-profiler/async-profiler>
Best regards,
Yun Tang and Yu Chen
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Yun,
3% extra cost sounds good! It could be useful, if the reference could be
added into the FLIP. Thanks for answering my questions.
Best regards,
Jing
On Tue, Oct 10, 2023 at 9:15 AM Yun Tang <my...@live.com> wrote:
> Hi Jing,
>
> First, developers would accept the little overhead when debugging the
> performance issues. Secondly, according to the async-profiler's report, it
> should only have less than 3% extra cost. I also verified it with a
> CPU-intensive ETL Flink job with this profiler, it does not show any
> obvious performance regression during profiling.
>
>
>
> [1] https://github.com/async-profiler/async-profiler/issues/14
>
> Best
> Yun Tang
>
> ________________________________
> From: Jing Ge <ji...@ververica.com.INVALID>
> Sent: Tuesday, October 10, 2023 12:05
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> profiler on taskmanagers
>
> Thanks Yun for your clarification. Especially thanks Rui for your
> informative elaboration. Since we will have two flame graphs, I would
> suggest updating Flink documentation to help users understand it and know
> when to use which one. The content provided by Rui is already a really good
> starting point. Like it!
>
> Yun
>
> Theoretically yeah, I am with you. Practically, it will help get more
> attraction and attention, if you can provide the real (test) metrics of how
> "extremely light" means, e.g. engineering mindset. Otherwise, each serious
> user will have to evaluate the performance on her/his own before using it
> in production. WDYT?
>
> Best regards,
> Jing
>
> On Tue, Oct 10, 2023 at 5:29 AM Yun Tang <my...@live.com> wrote:
>
> > Hi Jing,
> >
> > I will answer current questions.
> >
> > > 1. will it replace the current flame graph, i.e. the current flame
> graph
> > will be deprecated and removed?
> >
> > Although I think the new java profiler introduced in FLIP-375 is more
> > powerful, just as Rui has replied, I don't think it could replace current
> > flame graph totally.
> >
> >
> > > 2.does it make sense to provide the performance difference between
> enable
> > and disable it?
> >
> > The new java profiler would not introduce any performance impact after we
> > enable it, it will only start work when we trigger the profiling. And
> from
> > our experiences, the overhead of profiling is extremely light.
> >
> >
> >
> > For Rui's question:
> >
> > > Are all process-level flamegraphs stored at BlobStore? Are they
> > maintained by JobManager after sampling? Is there cleanup strategy? Or
> > max-save-count strategy?
> >
> > Yes, we use blobstore to store the process-level flamegraph-files and
> > maintained on taskmanager side. They flamegraph-files will be cleanup
> > automatically once reached to rest.profiling.history-size.
> >
> > Best
> > Yun Tang
> >
> >
> >
> > ________________________________
> > From: Rui Fan <19...@gmail.com>
> > Sent: Tuesday, October 10, 2023 10:10
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> > profiler on taskmanagers
> >
> > Hi Jing,
> >
> > > 1. will it replace the current flame graph, i.e. the current flame
> graph
> > will be deprecated and removed?
> >
> > I think the current flame graph cannot be removed.
> >
> > As a core contributor to the current flame graph, and I use it almost
> > every week. I would like to clarify the difference between the current
> > flame graph and the flame graph proposed by FLIP-375.
> >
> > @The current flame graph
> >
> > The current flame graph is the operator level or task level, when one
> > operator is the bottleneck of current job. We can see the current
> > flamegraph to check what the operator is doing.
> >
> > It includes three types: On-CPU, Off-CPU and Mixed-Type. The Mixed-Type
> > is very useful, it can detect why operator is slow even if the operator
> > doesn't use CPU. For example, the operator is blocked on querying hbase.
> >
> > It just support the task thread, it means it cannot detect the cpu usage
> of
> > other threads, such as: RocksDB Flush or compaction. This's the
> > limitation of current flamegraph.
> >
> > @The flame graph proposed by FLIP-375.
> >
> > The flamegraph proposed by FLIP-375 works on process level, such as
> > JobManager or TaskManager, so it can monitor all threads. Such as:
> > rocksdb background threads.
> >
> > When the CPU usage of one TM is high, and all tasks are not busy.
> > The new flamegraph will be useful.
> >
> > Back to the question: It includes task or operator thread,
> > why the current flamegraph is still needed?
> >
> > 1. The flamegraph of process level cannot easily distinguish tasks.
> > Especially if there are multiple slots in a TM, and different subtasks of
> > the
> > same task running in multiple slots, their stacks are very similar.
> >
> > 2. The Mixed-Type of current flamegraph may not be replaced by the
> > process-level flame graph.
> >
> > Please correct me if anything is wrong, thanks~
> >
> > Hi Yu,
> >
> > > Jobmanager allows the user to download the results of the corresponding
> > files on taskmanager with the blob service.
> >
> > Are all process-level flamegraphs stored at BlobStore?
> > Are they maintained by JobManager after sampling?
> > Is there cleanup strategy? Or max-save-count strategy?
> >
> > Best,
> > Rui
> >
> >
> > On Tue, Oct 10, 2023 at 1:24 AM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> >
> > > Hi Yu, Hi Yun,
> > >
> > > Brilliant idea! People are keen to use it. Thanks for your proposal! I
> > was
> > > wondering:
> > >
> > > 1. will it replace the current flame graph, i.e. the current flame
> graph
> > > will be deprecated and removed?
> > > 2. does it make sense to provide the performance difference between
> > enable
> > > and disable it?
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Mon, Oct 9, 2023 at 1:50 PM Yu Chen <yu...@gmail.com> wrote:
> > >
> > > > Hi zhanghao,
> > > >
> > > > Yes, agree with you. We'll take Jobmanager into consideration and
> > update
> > > > the FLIP later!
> > > >
> > > > Best,
> > > > Yu Chen
> > > >
> > > > Zhanghao Chen <zh...@outlook.com> 于2023年10月9日周一 19:22写道：
> > > >
> > > > > Hi Yun and Yu,
> > > > >
> > > > > Thanks for driving this. This would definitely help users identify
> > > > > performance bottlenecks, especially for the cases where the
> > bottleneck
> > > > lies
> > > > > in the system stack (e.g. GC), and big +1 for the downloadable
> > > flamegraph
> > > > > to ease sharing. I'm wondering if we could add this for the job
> > manager
> > > > as
> > > > > well. In the OLAP scenario and sometimes in the streaming scenario
> > > (when
> > > > > there're some heavy operations during execution plan generation or
> in
> > > > > operator coordinators), the JM can have bottleneck as well.
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > ________________________________
> > > > > From: Yu Chen <yu...@gmail.com>
> > > > > Sent: Monday, October 9, 2023 17:24
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> > > > > profiler on taskmanagers
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Yun Tang and I are opening this thread to discuss our proposal to
> > > > integrate
> > > > > async-profiler's capabilities for profiling taskmananger (e.g.,
> > > > generating
> > > > > flame graphs) in the Flink Web [1].
> > > > >
> > > > >
> > > > > Currently, Flink provides ThreadDump and Operator-Level Flame
> Graphs
> > by
> > > > > sampling task threads. The results generated in such way missing
> the
> > > > > relevant stack of java threads and system calls. The
> > async-profiler[2]
> > > > is a
> > > > > low-overhead sampling profiler for Java, but the steps to use it in
> > the
> > > > > production environment are cumbersome and suffer from permissions
> and
> > > > > security risks.
> > > > >
> > > > > Therefore, we propose adding rest APIs to provide the capability to
> > > > invoke
> > > > > async-profiler on multiple platforms through JNI, which can be
> easily
> > > > > operated on Web UI. This enhancement will improve the efficiency
> and
> > > > > experience of Flink users in identifying performance bottlenecks.
> > > > >
> > > > >
> > > > >
> > > > > Please refer to the FLIP document for more details about the
> proposed
> > > > > design
> > > > > and implementation. We welcome any feedback and opinions on this
> > > > proposal.
> > > > >
> > > > >
> > > > >
> > > > > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > > > > taskmanagers - Apache Flink - Apache Software Foundation
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> > > > > >
> > > > >
> > > > > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> > > > profiler
> > > > > for Java featuring AsyncGetCallTrace + perf_events
> > > > > <https://github.com/async-profiler/async-profiler>
> > > > >
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Yun Tang and Yu Chen
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Yun Tang <my...@live.com>.
Hi Jing,
First, developers would accept the little overhead when debugging the performance issues. Secondly, according to the async-profiler's report, it should only have less than 3% extra cost. I also verified it with a CPU-intensive ETL Flink job with this profiler, it does not show any obvious performance regression during profiling.
[1] https://github.com/async-profiler/async-profiler/issues/14
Best
Yun Tang
________________________________
From: Jing Ge <ji...@ververica.com.INVALID>
Sent: Tuesday, October 10, 2023 12:05
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Thanks Yun for your clarification. Especially thanks Rui for your
informative elaboration. Since we will have two flame graphs, I would
suggest updating Flink documentation to help users understand it and know
when to use which one. The content provided by Rui is already a really good
starting point. Like it!
Yun
Theoretically yeah, I am with you. Practically, it will help get more
attraction and attention, if you can provide the real (test) metrics of how
"extremely light" means, e.g. engineering mindset. Otherwise, each serious
user will have to evaluate the performance on her/his own before using it
in production. WDYT?
Best regards,
Jing
On Tue, Oct 10, 2023 at 5:29 AM Yun Tang <my...@live.com> wrote:
> Hi Jing,
>
> I will answer current questions.
>
> > 1. will it replace the current flame graph, i.e. the current flame graph
> will be deprecated and removed?
>
> Although I think the new java profiler introduced in FLIP-375 is more
> powerful, just as Rui has replied, I don't think it could replace current
> flame graph totally.
>
>
> > 2.does it make sense to provide the performance difference between enable
> and disable it?
>
> The new java profiler would not introduce any performance impact after we
> enable it, it will only start work when we trigger the profiling. And from
> our experiences, the overhead of profiling is extremely light.
>
>
>
> For Rui's question:
>
> > Are all process-level flamegraphs stored at BlobStore? Are they
> maintained by JobManager after sampling? Is there cleanup strategy? Or
> max-save-count strategy?
>
> Yes, we use blobstore to store the process-level flamegraph-files and
> maintained on taskmanager side. They flamegraph-files will be cleanup
> automatically once reached to rest.profiling.history-size.
>
> Best
> Yun Tang
>
>
>
> ________________________________
> From: Rui Fan <19...@gmail.com>
> Sent: Tuesday, October 10, 2023 10:10
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> profiler on taskmanagers
>
> Hi Jing,
>
> > 1. will it replace the current flame graph, i.e. the current flame graph
> will be deprecated and removed?
>
> I think the current flame graph cannot be removed.
>
> As a core contributor to the current flame graph, and I use it almost
> every week. I would like to clarify the difference between the current
> flame graph and the flame graph proposed by FLIP-375.
>
> @The current flame graph
>
> The current flame graph is the operator level or task level, when one
> operator is the bottleneck of current job. We can see the current
> flamegraph to check what the operator is doing.
>
> It includes three types: On-CPU, Off-CPU and Mixed-Type. The Mixed-Type
> is very useful, it can detect why operator is slow even if the operator
> doesn't use CPU. For example, the operator is blocked on querying hbase.
>
> It just support the task thread, it means it cannot detect the cpu usage of
> other threads, such as: RocksDB Flush or compaction. This's the
> limitation of current flamegraph.
>
> @The flame graph proposed by FLIP-375.
>
> The flamegraph proposed by FLIP-375 works on process level, such as
> JobManager or TaskManager, so it can monitor all threads. Such as:
> rocksdb background threads.
>
> When the CPU usage of one TM is high, and all tasks are not busy.
> The new flamegraph will be useful.
>
> Back to the question: It includes task or operator thread,
> why the current flamegraph is still needed?
>
> 1. The flamegraph of process level cannot easily distinguish tasks.
> Especially if there are multiple slots in a TM, and different subtasks of
> the
> same task running in multiple slots, their stacks are very similar.
>
> 2. The Mixed-Type of current flamegraph may not be replaced by the
> process-level flame graph.
>
> Please correct me if anything is wrong, thanks~
>
> Hi Yu,
>
> > Jobmanager allows the user to download the results of the corresponding
> files on taskmanager with the blob service.
>
> Are all process-level flamegraphs stored at BlobStore?
> Are they maintained by JobManager after sampling?
> Is there cleanup strategy? Or max-save-count strategy?
>
> Best,
> Rui
>
>
> On Tue, Oct 10, 2023 at 1:24 AM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > Hi Yu, Hi Yun,
> >
> > Brilliant idea! People are keen to use it. Thanks for your proposal! I
> was
> > wondering:
> >
> > 1. will it replace the current flame graph, i.e. the current flame graph
> > will be deprecated and removed?
> > 2. does it make sense to provide the performance difference between
> enable
> > and disable it?
> >
> > Best regards,
> > Jing
> >
> > On Mon, Oct 9, 2023 at 1:50 PM Yu Chen <yu...@gmail.com> wrote:
> >
> > > Hi zhanghao,
> > >
> > > Yes, agree with you. We'll take Jobmanager into consideration and
> update
> > > the FLIP later!
> > >
> > > Best,
> > > Yu Chen
> > >
> > > Zhanghao Chen <zh...@outlook.com> 于2023年10月9日周一 19:22写道：
> > >
> > > > Hi Yun and Yu,
> > > >
> > > > Thanks for driving this. This would definitely help users identify
> > > > performance bottlenecks, especially for the cases where the
> bottleneck
> > > lies
> > > > in the system stack (e.g. GC), and big +1 for the downloadable
> > flamegraph
> > > > to ease sharing. I'm wondering if we could add this for the job
> manager
> > > as
> > > > well. In the OLAP scenario and sometimes in the streaming scenario
> > (when
> > > > there're some heavy operations during execution plan generation or in
> > > > operator coordinators), the JM can have bottleneck as well.
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > ________________________________
> > > > From: Yu Chen <yu...@gmail.com>
> > > > Sent: Monday, October 9, 2023 17:24
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> > > > profiler on taskmanagers
> > > >
> > > > Hi all,
> > > >
> > > > Yun Tang and I are opening this thread to discuss our proposal to
> > > integrate
> > > > async-profiler's capabilities for profiling taskmananger (e.g.,
> > > generating
> > > > flame graphs) in the Flink Web [1].
> > > >
> > > >
> > > > Currently, Flink provides ThreadDump and Operator-Level Flame Graphs
> by
> > > > sampling task threads. The results generated in such way missing the
> > > > relevant stack of java threads and system calls. The
> async-profiler[2]
> > > is a
> > > > low-overhead sampling profiler for Java, but the steps to use it in
> the
> > > > production environment are cumbersome and suffer from permissions and
> > > > security risks.
> > > >
> > > > Therefore, we propose adding rest APIs to provide the capability to
> > > invoke
> > > > async-profiler on multiple platforms through JNI, which can be easily
> > > > operated on Web UI. This enhancement will improve the efficiency and
> > > > experience of Flink users in identifying performance bottlenecks.
> > > >
> > > >
> > > >
> > > > Please refer to the FLIP document for more details about the proposed
> > > > design
> > > > and implementation. We welcome any feedback and opinions on this
> > > proposal.
> > > >
> > > >
> > > >
> > > > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > > > taskmanagers - Apache Flink - Apache Software Foundation
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> > > > >
> > > >
> > > > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> > > profiler
> > > > for Java featuring AsyncGetCallTrace + perf_events
> > > > <https://github.com/async-profiler/async-profiler>
> > > >
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > Yun Tang and Yu Chen
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Thanks Yun for your clarification. Especially thanks Rui for your
informative elaboration. Since we will have two flame graphs, I would
suggest updating Flink documentation to help users understand it and know
when to use which one. The content provided by Rui is already a really good
starting point. Like it!
Yun
Theoretically yeah, I am with you. Practically, it will help get more
attraction and attention, if you can provide the real (test) metrics of how
"extremely light" means, e.g. engineering mindset. Otherwise, each serious
user will have to evaluate the performance on her/his own before using it
in production. WDYT?
Best regards,
Jing
On Tue, Oct 10, 2023 at 5:29 AM Yun Tang <my...@live.com> wrote:
> Hi Jing,
>
> I will answer current questions.
>
> > 1. will it replace the current flame graph, i.e. the current flame graph
> will be deprecated and removed?
>
> Although I think the new java profiler introduced in FLIP-375 is more
> powerful, just as Rui has replied, I don't think it could replace current
> flame graph totally.
>
>
> > 2.does it make sense to provide the performance difference between enable
> and disable it?
>
> The new java profiler would not introduce any performance impact after we
> enable it, it will only start work when we trigger the profiling. And from
> our experiences, the overhead of profiling is extremely light.
>
>
>
> For Rui's question:
>
> > Are all process-level flamegraphs stored at BlobStore? Are they
> maintained by JobManager after sampling? Is there cleanup strategy? Or
> max-save-count strategy?
>
> Yes, we use blobstore to store the process-level flamegraph-files and
> maintained on taskmanager side. They flamegraph-files will be cleanup
> automatically once reached to rest.profiling.history-size.
>
> Best
> Yun Tang
>
>
>
> ________________________________
> From: Rui Fan <19...@gmail.com>
> Sent: Tuesday, October 10, 2023 10:10
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> profiler on taskmanagers
>
> Hi Jing,
>
> > 1. will it replace the current flame graph, i.e. the current flame graph
> will be deprecated and removed?
>
> I think the current flame graph cannot be removed.
>
> As a core contributor to the current flame graph, and I use it almost
> every week. I would like to clarify the difference between the current
> flame graph and the flame graph proposed by FLIP-375.
>
> @The current flame graph
>
> The current flame graph is the operator level or task level, when one
> operator is the bottleneck of current job. We can see the current
> flamegraph to check what the operator is doing.
>
> It includes three types: On-CPU, Off-CPU and Mixed-Type. The Mixed-Type
> is very useful, it can detect why operator is slow even if the operator
> doesn't use CPU. For example, the operator is blocked on querying hbase.
>
> It just support the task thread, it means it cannot detect the cpu usage of
> other threads, such as: RocksDB Flush or compaction. This's the
> limitation of current flamegraph.
>
> @The flame graph proposed by FLIP-375.
>
> The flamegraph proposed by FLIP-375 works on process level, such as
> JobManager or TaskManager, so it can monitor all threads. Such as:
> rocksdb background threads.
>
> When the CPU usage of one TM is high, and all tasks are not busy.
> The new flamegraph will be useful.
>
> Back to the question: It includes task or operator thread,
> why the current flamegraph is still needed?
>
> 1. The flamegraph of process level cannot easily distinguish tasks.
> Especially if there are multiple slots in a TM, and different subtasks of
> the
> same task running in multiple slots, their stacks are very similar.
>
> 2. The Mixed-Type of current flamegraph may not be replaced by the
> process-level flame graph.
>
> Please correct me if anything is wrong, thanks~
>
> Hi Yu,
>
> > Jobmanager allows the user to download the results of the corresponding
> files on taskmanager with the blob service.
>
> Are all process-level flamegraphs stored at BlobStore?
> Are they maintained by JobManager after sampling?
> Is there cleanup strategy? Or max-save-count strategy?
>
> Best,
> Rui
>
>
> On Tue, Oct 10, 2023 at 1:24 AM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > Hi Yu, Hi Yun,
> >
> > Brilliant idea! People are keen to use it. Thanks for your proposal! I
> was
> > wondering:
> >
> > 1. will it replace the current flame graph, i.e. the current flame graph
> > will be deprecated and removed?
> > 2. does it make sense to provide the performance difference between
> enable
> > and disable it?
> >
> > Best regards,
> > Jing
> >
> > On Mon, Oct 9, 2023 at 1:50 PM Yu Chen <yu...@gmail.com> wrote:
> >
> > > Hi zhanghao,
> > >
> > > Yes, agree with you. We'll take Jobmanager into consideration and
> update
> > > the FLIP later!
> > >
> > > Best,
> > > Yu Chen
> > >
> > > Zhanghao Chen <zh...@outlook.com> 于2023年10月9日周一 19:22写道：
> > >
> > > > Hi Yun and Yu,
> > > >
> > > > Thanks for driving this. This would definitely help users identify
> > > > performance bottlenecks, especially for the cases where the
> bottleneck
> > > lies
> > > > in the system stack (e.g. GC), and big +1 for the downloadable
> > flamegraph
> > > > to ease sharing. I'm wondering if we could add this for the job
> manager
> > > as
> > > > well. In the OLAP scenario and sometimes in the streaming scenario
> > (when
> > > > there're some heavy operations during execution plan generation or in
> > > > operator coordinators), the JM can have bottleneck as well.
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > ________________________________
> > > > From: Yu Chen <yu...@gmail.com>
> > > > Sent: Monday, October 9, 2023 17:24
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> > > > profiler on taskmanagers
> > > >
> > > > Hi all,
> > > >
> > > > Yun Tang and I are opening this thread to discuss our proposal to
> > > integrate
> > > > async-profiler's capabilities for profiling taskmananger (e.g.,
> > > generating
> > > > flame graphs) in the Flink Web [1].
> > > >
> > > >
> > > > Currently, Flink provides ThreadDump and Operator-Level Flame Graphs
> by
> > > > sampling task threads. The results generated in such way missing the
> > > > relevant stack of java threads and system calls. The
> async-profiler[2]
> > > is a
> > > > low-overhead sampling profiler for Java, but the steps to use it in
> the
> > > > production environment are cumbersome and suffer from permissions and
> > > > security risks.
> > > >
> > > > Therefore, we propose adding rest APIs to provide the capability to
> > > invoke
> > > > async-profiler on multiple platforms through JNI, which can be easily
> > > > operated on Web UI. This enhancement will improve the efficiency and
> > > > experience of Flink users in identifying performance bottlenecks.
> > > >
> > > >
> > > >
> > > > Please refer to the FLIP document for more details about the proposed
> > > > design
> > > > and implementation. We welcome any feedback and opinions on this
> > > proposal.
> > > >
> > > >
> > > >
> > > > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > > > taskmanagers - Apache Flink - Apache Software Foundation
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> > > > >
> > > >
> > > > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> > > profiler
> > > > for Java featuring AsyncGetCallTrace + perf_events
> > > > <https://github.com/async-profiler/async-profiler>
> > > >
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > Yun Tang and Yu Chen
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Yun Tang <my...@live.com>.
Hi Jing,
I will answer current questions.
> 1. will it replace the current flame graph, i.e. the current flame graph
will be deprecated and removed?
Although I think the new java profiler introduced in FLIP-375 is more powerful, just as Rui has replied, I don't think it could replace current flame graph totally.
> 2.does it make sense to provide the performance difference between enable
and disable it?
The new java profiler would not introduce any performance impact after we enable it, it will only start work when we trigger the profiling. And from our experiences, the overhead of profiling is extremely light.
For Rui's question:
> Are all process-level flamegraphs stored at BlobStore? Are they maintained by JobManager after sampling? Is there cleanup strategy? Or max-save-count strategy?
Yes, we use blobstore to store the process-level flamegraph-files and maintained on taskmanager side. They flamegraph-files will be cleanup automatically once reached to rest.profiling.history-size.
Best
Yun Tang
________________________________
From: Rui Fan <19...@gmail.com>
Sent: Tuesday, October 10, 2023 10:10
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Hi Jing,
> 1. will it replace the current flame graph, i.e. the current flame graph
will be deprecated and removed?
I think the current flame graph cannot be removed.
As a core contributor to the current flame graph, and I use it almost
every week. I would like to clarify the difference between the current
flame graph and the flame graph proposed by FLIP-375.
@The current flame graph
The current flame graph is the operator level or task level, when one
operator is the bottleneck of current job. We can see the current
flamegraph to check what the operator is doing.
It includes three types: On-CPU, Off-CPU and Mixed-Type. The Mixed-Type
is very useful, it can detect why operator is slow even if the operator
doesn't use CPU. For example, the operator is blocked on querying hbase.
It just support the task thread, it means it cannot detect the cpu usage of
other threads, such as: RocksDB Flush or compaction. This's the
limitation of current flamegraph.
@The flame graph proposed by FLIP-375.
The flamegraph proposed by FLIP-375 works on process level, such as
JobManager or TaskManager, so it can monitor all threads. Such as:
rocksdb background threads.
When the CPU usage of one TM is high, and all tasks are not busy.
The new flamegraph will be useful.
Back to the question: It includes task or operator thread,
why the current flamegraph is still needed?
1. The flamegraph of process level cannot easily distinguish tasks.
Especially if there are multiple slots in a TM, and different subtasks of
the
same task running in multiple slots, their stacks are very similar.
2. The Mixed-Type of current flamegraph may not be replaced by the
process-level flame graph.
Please correct me if anything is wrong, thanks~
Hi Yu,
> Jobmanager allows the user to download the results of the corresponding
files on taskmanager with the blob service.
Are all process-level flamegraphs stored at BlobStore?
Are they maintained by JobManager after sampling?
Is there cleanup strategy? Or max-save-count strategy?
Best,
Rui
On Tue, Oct 10, 2023 at 1:24 AM Jing Ge <ji...@ververica.com.invalid> wrote:
> Hi Yu, Hi Yun,
>
> Brilliant idea! People are keen to use it. Thanks for your proposal! I was
> wondering:
>
> 1. will it replace the current flame graph, i.e. the current flame graph
> will be deprecated and removed?
> 2. does it make sense to provide the performance difference between enable
> and disable it?
>
> Best regards,
> Jing
>
> On Mon, Oct 9, 2023 at 1:50 PM Yu Chen <yu...@gmail.com> wrote:
>
> > Hi zhanghao,
> >
> > Yes, agree with you. We'll take Jobmanager into consideration and update
> > the FLIP later!
> >
> > Best,
> > Yu Chen
> >
> > Zhanghao Chen <zh...@outlook.com> 于2023年10月9日周一 19:22写道：
> >
> > > Hi Yun and Yu,
> > >
> > > Thanks for driving this. This would definitely help users identify
> > > performance bottlenecks, especially for the cases where the bottleneck
> > lies
> > > in the system stack (e.g. GC), and big +1 for the downloadable
> flamegraph
> > > to ease sharing. I'm wondering if we could add this for the job manager
> > as
> > > well. In the OLAP scenario and sometimes in the streaming scenario
> (when
> > > there're some heavy operations during execution plan generation or in
> > > operator coordinators), the JM can have bottleneck as well.
> > >
> > > Best,
> > > Zhanghao Chen
> > > ________________________________
> > > From: Yu Chen <yu...@gmail.com>
> > > Sent: Monday, October 9, 2023 17:24
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> > > profiler on taskmanagers
> > >
> > > Hi all,
> > >
> > > Yun Tang and I are opening this thread to discuss our proposal to
> > integrate
> > > async-profiler's capabilities for profiling taskmananger (e.g.,
> > generating
> > > flame graphs) in the Flink Web [1].
> > >
> > >
> > > Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
> > > sampling task threads. The results generated in such way missing the
> > > relevant stack of java threads and system calls. The async-profiler[2]
> > is a
> > > low-overhead sampling profiler for Java, but the steps to use it in the
> > > production environment are cumbersome and suffer from permissions and
> > > security risks.
> > >
> > > Therefore, we propose adding rest APIs to provide the capability to
> > invoke
> > > async-profiler on multiple platforms through JNI, which can be easily
> > > operated on Web UI. This enhancement will improve the efficiency and
> > > experience of Flink users in identifying performance bottlenecks.
> > >
> > >
> > >
> > > Please refer to the FLIP document for more details about the proposed
> > > design
> > > and implementation. We welcome any feedback and opinions on this
> > proposal.
> > >
> > >
> > >
> > > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > > taskmanagers - Apache Flink - Apache Software Foundation
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> > > >
> > >
> > > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> > profiler
> > > for Java featuring AsyncGetCallTrace + perf_events
> > > <https://github.com/async-profiler/async-profiler>
> > >
> > >
> > >
> > > Best regards,
> > >
> > > Yun Tang and Yu Chen
> > >
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Rui Fan <19...@gmail.com>.
Hi Jing,
> 1. will it replace the current flame graph, i.e. the current flame graph
will be deprecated and removed?
I think the current flame graph cannot be removed.
As a core contributor to the current flame graph, and I use it almost
every week. I would like to clarify the difference between the current
flame graph and the flame graph proposed by FLIP-375.
@The current flame graph
The current flame graph is the operator level or task level, when one
operator is the bottleneck of current job. We can see the current
flamegraph to check what the operator is doing.
It includes three types: On-CPU, Off-CPU and Mixed-Type. The Mixed-Type
is very useful, it can detect why operator is slow even if the operator
doesn't use CPU. For example, the operator is blocked on querying hbase.
It just support the task thread, it means it cannot detect the cpu usage of
other threads, such as: RocksDB Flush or compaction. This's the
limitation of current flamegraph.
@The flame graph proposed by FLIP-375.
The flamegraph proposed by FLIP-375 works on process level, such as
JobManager or TaskManager, so it can monitor all threads. Such as:
rocksdb background threads.
When the CPU usage of one TM is high, and all tasks are not busy.
The new flamegraph will be useful.
Back to the question: It includes task or operator thread,
why the current flamegraph is still needed?
1. The flamegraph of process level cannot easily distinguish tasks.
Especially if there are multiple slots in a TM, and different subtasks of
the
same task running in multiple slots, their stacks are very similar.
2. The Mixed-Type of current flamegraph may not be replaced by the
process-level flame graph.
Please correct me if anything is wrong, thanks~
Hi Yu,
> Jobmanager allows the user to download the results of the corresponding
files on taskmanager with the blob service.
Are all process-level flamegraphs stored at BlobStore?
Are they maintained by JobManager after sampling?
Is there cleanup strategy? Or max-save-count strategy?
Best,
Rui
On Tue, Oct 10, 2023 at 1:24 AM Jing Ge <ji...@ververica.com.invalid> wrote:
> Hi Yu, Hi Yun,
>
> Brilliant idea! People are keen to use it. Thanks for your proposal! I was
> wondering:
>
> 1. will it replace the current flame graph, i.e. the current flame graph
> will be deprecated and removed?
> 2. does it make sense to provide the performance difference between enable
> and disable it?
>
> Best regards,
> Jing
>
> On Mon, Oct 9, 2023 at 1:50 PM Yu Chen <yu...@gmail.com> wrote:
>
> > Hi zhanghao,
> >
> > Yes, agree with you. We'll take Jobmanager into consideration and update
> > the FLIP later!
> >
> > Best,
> > Yu Chen
> >
> > Zhanghao Chen <zh...@outlook.com> 于2023年10月9日周一 19:22写道：
> >
> > > Hi Yun and Yu,
> > >
> > > Thanks for driving this. This would definitely help users identify
> > > performance bottlenecks, especially for the cases where the bottleneck
> > lies
> > > in the system stack (e.g. GC), and big +1 for the downloadable
> flamegraph
> > > to ease sharing. I'm wondering if we could add this for the job manager
> > as
> > > well. In the OLAP scenario and sometimes in the streaming scenario
> (when
> > > there're some heavy operations during execution plan generation or in
> > > operator coordinators), the JM can have bottleneck as well.
> > >
> > > Best,
> > > Zhanghao Chen
> > > ________________________________
> > > From: Yu Chen <yu...@gmail.com>
> > > Sent: Monday, October 9, 2023 17:24
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> > > profiler on taskmanagers
> > >
> > > Hi all,
> > >
> > > Yun Tang and I are opening this thread to discuss our proposal to
> > integrate
> > > async-profiler's capabilities for profiling taskmananger (e.g.,
> > generating
> > > flame graphs) in the Flink Web [1].
> > >
> > >
> > > Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
> > > sampling task threads. The results generated in such way missing the
> > > relevant stack of java threads and system calls. The async-profiler[2]
> > is a
> > > low-overhead sampling profiler for Java, but the steps to use it in the
> > > production environment are cumbersome and suffer from permissions and
> > > security risks.
> > >
> > > Therefore, we propose adding rest APIs to provide the capability to
> > invoke
> > > async-profiler on multiple platforms through JNI, which can be easily
> > > operated on Web UI. This enhancement will improve the efficiency and
> > > experience of Flink users in identifying performance bottlenecks.
> > >
> > >
> > >
> > > Please refer to the FLIP document for more details about the proposed
> > > design
> > > and implementation. We welcome any feedback and opinions on this
> > proposal.
> > >
> > >
> > >
> > > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > > taskmanagers - Apache Flink - Apache Software Foundation
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> > > >
> > >
> > > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> > profiler
> > > for Java featuring AsyncGetCallTrace + perf_events
> > > <https://github.com/async-profiler/async-profiler>
> > >
> > >
> > >
> > > Best regards,
> > >
> > > Yun Tang and Yu Chen
> > >
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Yu, Hi Yun,
Brilliant idea! People are keen to use it. Thanks for your proposal! I was
wondering:
1. will it replace the current flame graph, i.e. the current flame graph
will be deprecated and removed?
2. does it make sense to provide the performance difference between enable
and disable it?
Best regards,
Jing
On Mon, Oct 9, 2023 at 1:50 PM Yu Chen <yu...@gmail.com> wrote:
> Hi zhanghao,
>
> Yes, agree with you. We'll take Jobmanager into consideration and update
> the FLIP later!
>
> Best,
> Yu Chen
>
> Zhanghao Chen <zh...@outlook.com> 于2023年10月9日周一 19:22写道：
>
> > Hi Yun and Yu,
> >
> > Thanks for driving this. This would definitely help users identify
> > performance bottlenecks, especially for the cases where the bottleneck
> lies
> > in the system stack (e.g. GC), and big +1 for the downloadable flamegraph
> > to ease sharing. I'm wondering if we could add this for the job manager
> as
> > well. In the OLAP scenario and sometimes in the streaming scenario (when
> > there're some heavy operations during execution plan generation or in
> > operator coordinators), the JM can have bottleneck as well.
> >
> > Best,
> > Zhanghao Chen
> > ________________________________
> > From: Yu Chen <yu...@gmail.com>
> > Sent: Monday, October 9, 2023 17:24
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> > profiler on taskmanagers
> >
> > Hi all,
> >
> > Yun Tang and I are opening this thread to discuss our proposal to
> integrate
> > async-profiler's capabilities for profiling taskmananger (e.g.,
> generating
> > flame graphs) in the Flink Web [1].
> >
> >
> > Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
> > sampling task threads. The results generated in such way missing the
> > relevant stack of java threads and system calls. The async-profiler[2]
> is a
> > low-overhead sampling profiler for Java, but the steps to use it in the
> > production environment are cumbersome and suffer from permissions and
> > security risks.
> >
> > Therefore, we propose adding rest APIs to provide the capability to
> invoke
> > async-profiler on multiple platforms through JNI, which can be easily
> > operated on Web UI. This enhancement will improve the efficiency and
> > experience of Flink users in identifying performance bottlenecks.
> >
> >
> >
> > Please refer to the FLIP document for more details about the proposed
> > design
> > and implementation. We welcome any feedback and opinions on this
> proposal.
> >
> >
> >
> > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > taskmanagers - Apache Flink - Apache Software Foundation
> > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> > >
> >
> > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> profiler
> > for Java featuring AsyncGetCallTrace + perf_events
> > <https://github.com/async-profiler/async-profiler>
> >
> >
> >
> > Best regards,
> >
> > Yun Tang and Yu Chen
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Yu Chen <yu...@gmail.com>.
Hi zhanghao,
Yes, agree with you. We'll take Jobmanager into consideration and update
the FLIP later!
Best,
Yu Chen
Zhanghao Chen <zh...@outlook.com> 于2023年10月9日周一 19:22写道：
> Hi Yun and Yu,
>
> Thanks for driving this. This would definitely help users identify
> performance bottlenecks, especially for the cases where the bottleneck lies
> in the system stack (e.g. GC), and big +1 for the downloadable flamegraph
> to ease sharing. I'm wondering if we could add this for the job manager as
> well. In the OLAP scenario and sometimes in the streaming scenario (when
> there're some heavy operations during execution plan generation or in
> operator coordinators), the JM can have bottleneck as well.
>
> Best,
> Zhanghao Chen
> ________________________________
> From: Yu Chen <yu...@gmail.com>
> Sent: Monday, October 9, 2023 17:24
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> profiler on taskmanagers
>
> Hi all,
>
> Yun Tang and I are opening this thread to discuss our proposal to integrate
> async-profiler's capabilities for profiling taskmananger (e.g., generating
> flame graphs) in the Flink Web [1].
>
>
> Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
> sampling task threads. The results generated in such way missing the
> relevant stack of java threads and system calls. The async-profiler[2] is a
> low-overhead sampling profiler for Java, but the steps to use it in the
> production environment are cumbersome and suffer from permissions and
> security risks.
>
> Therefore, we propose adding rest APIs to provide the capability to invoke
> async-profiler on multiple platforms through JNI, which can be easily
> operated on Web UI. This enhancement will improve the efficiency and
> experience of Flink users in identifying performance bottlenecks.
>
>
>
> Please refer to the FLIP document for more details about the proposed
> design
> and implementation. We welcome any feedback and opinions on this proposal.
>
>
>
> [1] FLIP-375: Built-in cross-platform powerful java profiler on
> taskmanagers - Apache Flink - Apache Software Foundation
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> >
>
> [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP profiler
> for Java featuring AsyncGetCallTrace + perf_events
> <https://github.com/async-profiler/async-profiler>
>
>
>
> Best regards,
>
> Yun Tang and Yu Chen
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Rui Fan <19...@gmail.com>.
Thanks Yun and Yu for driving this proposal!
It's very useful for troubleshooting why the CPU usage is high.
+1
Best,
Rui
On Mon, Oct 9, 2023 at 7:21 PM Zhanghao Chen <zh...@outlook.com>
wrote:
> Hi Yun and Yu,
>
> Thanks for driving this. This would definitely help users identify
> performance bottlenecks, especially for the cases where the bottleneck lies
> in the system stack (e.g. GC), and big +1 for the downloadable flamegraph
> to ease sharing. I'm wondering if we could add this for the job manager as
> well. In the OLAP scenario and sometimes in the streaming scenario (when
> there're some heavy operations during execution plan generation or in
> operator coordinators), the JM can have bottleneck as well.
>
> Best,
> Zhanghao Chen
> ________________________________
> From: Yu Chen <yu...@gmail.com>
> Sent: Monday, October 9, 2023 17:24
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [DISCUSS] FLIP-375: Built-in cross-platform powerful java
> profiler on taskmanagers
>
> Hi all,
>
> Yun Tang and I are opening this thread to discuss our proposal to integrate
> async-profiler's capabilities for profiling taskmananger (e.g., generating
> flame graphs) in the Flink Web [1].
>
>
> Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
> sampling task threads. The results generated in such way missing the
> relevant stack of java threads and system calls. The async-profiler[2] is a
> low-overhead sampling profiler for Java, but the steps to use it in the
> production environment are cumbersome and suffer from permissions and
> security risks.
>
> Therefore, we propose adding rest APIs to provide the capability to invoke
> async-profiler on multiple platforms through JNI, which can be easily
> operated on Web UI. This enhancement will improve the efficiency and
> experience of Flink users in identifying performance bottlenecks.
>
>
>
> Please refer to the FLIP document for more details about the proposed
> design
> and implementation. We welcome any feedback and opinions on this proposal.
>
>
>
> [1] FLIP-375: Built-in cross-platform powerful java profiler on
> taskmanagers - Apache Flink - Apache Software Foundation
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> >
>
> [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP profiler
> for Java featuring AsyncGetCallTrace + perf_events
> <https://github.com/async-profiler/async-profiler>
>
>
>
> Best regards,
>
> Yun Tang and Yu Chen
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Zhanghao Chen <zh...@outlook.com>.
Hi Yun and Yu,
Thanks for driving this. This would definitely help users identify performance bottlenecks, especially for the cases where the bottleneck lies in the system stack (e.g. GC), and big +1 for the downloadable flamegraph to ease sharing. I'm wondering if we could add this for the job manager as well. In the OLAP scenario and sometimes in the streaming scenario (when there're some heavy operations during execution plan generation or in operator coordinators), the JM can have bottleneck as well.
Best,
Zhanghao Chen
________________________________
From: Yu Chen <yu...@gmail.com>
Sent: Monday, October 9, 2023 17:24
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Hi all,
Yun Tang and I are opening this thread to discuss our proposal to integrate
async-profiler's capabilities for profiling taskmananger (e.g., generating
flame graphs) in the Flink Web [1].
Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
sampling task threads. The results generated in such way missing the
relevant stack of java threads and system calls. The async-profiler[2] is a
low-overhead sampling profiler for Java, but the steps to use it in the
production environment are cumbersome and suffer from permissions and
security risks.
Therefore, we propose adding rest APIs to provide the capability to invoke
async-profiler on multiple platforms through JNI, which can be easily
operated on Web UI. This enhancement will improve the efficiency and
experience of Flink users in identifying performance bottlenecks.
Please refer to the FLIP document for more details about the proposed design
and implementation. We welcome any feedback and opinions on this proposal.
[1] FLIP-375: Built-in cross-platform powerful java profiler on
taskmanagers - Apache Flink - Apache Software Foundation
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers>
[2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP profiler
for Java featuring AsyncGetCallTrace + perf_events
<https://github.com/async-profiler/async-profiler>
Best regards,
Yun Tang and Yu Chen
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Rui Fan <19...@gmail.com>.
Thanks Yun for the update, LGTM.
Best,
Rui
On Mon, Oct 16, 2023 at 1:34 PM Yu Chen <yu...@gmail.com> wrote:
> Hi David.
>
> Thanks for your detailed comments.
> The async-profiler has a lot of features, but there are some requirements
> to use it (some cases encountered in production are listed below):
> 1. Prior to JDK 11, alloc mode requires HotSpot debug symbols, and for
> OpenJDK, you need to install them additionally.
> 2. CPU mode requires perf_events support.
> 3. etc (refer to async-profiler's troubleshooting section[1])
>
> However, the addition of modifying system kernel parameters may involve the
> action of invoking shell commands.
> For security considerations, as a workaround, we can display async-profiler
> error messages in the web UI when the task's runtime environment does not
> satisfy the corresponding mode, and the user can manually modify the
> runtime environment (e.g., modify kernel parameters) based on the error
> messages as a bypass solution.
>
> As you recommend, we will revise the FLIP as follows:
> 1. We expose the -e parameter as a dropdown box in the Flink WEB and make
> `itimer` mode the default option. This makes it easy for users to use the
> different profiling modes (wall clock/alloc/cpu/itemer).
> 2. We added a message column on Flink Web to show the call exception so
> that users can adjust the environment according to the exception
> information.
>
> [1] async-profiler/async-profiler: Sampling CPU and HEAP profiler for Java
> featuring AsyncGetCallTrace + perf_events (github.com)
> <https://github.com/async-profiler/async-profiler#troubleshooting>
>
> Best,
> Yu Chen
>
> David Christle <da...@discordapp.com.invalid> 于2023年10月14日周六
> 04:12写道：
>
> > In the Wiki, this FLIP is motivated by:
> >
> > - That the current flamegraph functionality can only see operator-level
> > stack traces, while async-profiler provides CPU/allocation/locks
> > information, along with deeper Java & system call stack information.
> > - Low configurability (e.g. cannot set the sampling interval) + the
> > usability is limited to visual inspection of the flamegraph whenever it
> > happens to read out.
> >
> > The current built-in flamegraph is extremely valuable and easy to use.
> But
> > as a regular user of async-profiler for Flink applications, I agree these
> > deficiencies are worth improving. It's exciting that async-profiler might
> > be built-in, since it will make using it much easier.
> >
> > I'm a little confused, though, about the scope of functionality we'll
> have
> > with this FLIP, in particular the use of itimer only & perf_events
> support.
> > One of the questions near the end of the Wiki is whether sampling with
> > perf_events will be supported. The current answer seems to say that only
> > itimer mode will be supported, as this mode does not rely on perf_events
> > being enabled.
> >
> > However, given that an aim of the FLIP is to support configurability
> (e.g.
> > sampling intervals), is it that much more work to support configurability
> > of the event & the other common options, too? If the '-e' event flag is
> > fixed to itimer only, we can't use wall clock/alloc/cpu profiling modes.
> > The Wiki mentions async-profiler's JNI interface will be used, which has
> > 'event_str' as an input. So, it seems like supporting different event
> types
> > (or even multiple event types in one profile) is possible.
> >
> > Regarding perf_events, it's true that it's disabled in many
> > environments. But it is possible to enable it for debugging purposes. In
> > our Kubernetes workloads, this means adding SYS_PTRACE and SYS_ADMIN to
> the
> > securityContext, deploying the job, and then running:
> >
> > sysctl kernel.kptr_restrict=0
> > sysctl kernel.perf_event_paranoid=1
> >
> > before starting async-profiler.
> >
> > It would be nice if dynamically changing the kernel parameters was
> built-in
> > to this FLIP, somehow, as well, to set these parameters correctly before
> > profiling. If the environment restricts changing these, that's fine. We
> can
> > simply report to the user via the UI that setting them failed, and that
> the
> > choice of profiling configurations is limited without them. I also think
> > it's OK if `itimer` is the default in the UI, as it works under the
> > broadest conditions. But given the motivation in the Wiki is that
> > async-profiler can see detailed system call stack info, allocation, etc.,
> > and that the async-profiler docs describe itimer mode as a "fallback"
> > rather than the way the profiler is best used, it feels like this FLIP
> > should support async-profiler's regular modes of operation & the other
> > most-common configuration options. From my own experience, `cpu`
> (requiring
> > perf_events) is a bit more accurate than `itimer`, and if I recall, and
> > samples once per thread. `wall` is very useful to debug blocks on I/O or
> > locks. Getting per-thread information is nice to drill down into specific
> > parts of the Flink application, e.g. the flame graph lets me ignore the
> > many other tasks running on TM & drill down into just the Source threads,
> > when debugging a Source issue.
> >
> > Kind regards,
> > David
> >
> > On Fri, Oct 13, 2023 at 1:45 AM Rui Fan <19...@gmail.com> wrote:
> >
> > > One minor comment:
> > >
> > > In general, the generic java profiler includes memory analysis,
> > > cpu, thread, deadlock, etc. The FLIP title is java profiler, but
> > > the FLIP just supports flamegraph at process level.
> > > So the `powerful java profiler` title may not be suitable.
> > > Would you mind updating the FLIP title?
> > >
> > > Best,
> > > Rui
> > >
> > > On Fri, Oct 13, 2023 at 4:34 PM Yu Chen <yu...@gmail.com> wrote:
> > >
> > > > Hi all.
> > > > If there are no further questions, we will start a vote on FLIP-375
> > next
> > > > week.
> > > >
> > > > Best regards,
> > > > Yu Chen
> > > >
> > > >
> > > > Yu Chen <yu...@gmail.com> 于2023年10月9日周一 17:24写道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > Yun Tang and I are opening this thread to discuss our proposal to
> > > > > integrate async-profiler's capabilities for profiling taskmananger
> > > (e.g.,
> > > > > generating flame graphs) in the Flink Web [1].
> > > > >
> > > > >
> > > > > Currently, Flink provides ThreadDump and Operator-Level Flame
> Graphs
> > by
> > > > > sampling task threads. The results generated in such way missing
> the
> > > > > relevant stack of java threads and system calls. The
> > async-profiler[2]
> > > > is a
> > > > > low-overhead sampling profiler for Java, but the steps to use it in
> > the
> > > > > production environment are cumbersome and suffer from permissions
> and
> > > > > security risks.
> > > > >
> > > > > Therefore, we propose adding rest APIs to provide the capability to
> > > > invoke
> > > > > async-profiler on multiple platforms through JNI, which can be
> easily
> > > > > operated on Web UI. This enhancement will improve the efficiency
> and
> > > > > experience of Flink users in identifying performance bottlenecks.
> > > > >
> > > > >
> > > > >
> > > > > Please refer to the FLIP document for more details about the
> proposed
> > > > design
> > > > > and implementation. We welcome any feedback and opinions on this
> > > > proposal.
> > > > >
> > > > >
> > > > >
> > > > > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > > > > taskmanagers - Apache Flink - Apache Software Foundation
> > > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> > > > >
> > > > >
> > > > > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> > > > > profiler for Java featuring AsyncGetCallTrace + perf_events
> > > > > <https://github.com/async-profiler/async-profiler>
> > > > >
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Yun Tang and Yu Chen
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Yu Chen <yu...@gmail.com>.
Hi David.
Thanks for your detailed comments.
The async-profiler has a lot of features, but there are some requirements
to use it (some cases encountered in production are listed below):
1. Prior to JDK 11, alloc mode requires HotSpot debug symbols, and for
OpenJDK, you need to install them additionally.
2. CPU mode requires perf_events support.
3. etc (refer to async-profiler's troubleshooting section[1])
However, the addition of modifying system kernel parameters may involve the
action of invoking shell commands.
For security considerations, as a workaround, we can display async-profiler
error messages in the web UI when the task's runtime environment does not
satisfy the corresponding mode, and the user can manually modify the
runtime environment (e.g., modify kernel parameters) based on the error
messages as a bypass solution.
As you recommend, we will revise the FLIP as follows:
1. We expose the -e parameter as a dropdown box in the Flink WEB and make
`itimer` mode the default option. This makes it easy for users to use the
different profiling modes (wall clock/alloc/cpu/itemer).
2. We added a message column on Flink Web to show the call exception so
that users can adjust the environment according to the exception
information.
[1] async-profiler/async-profiler: Sampling CPU and HEAP profiler for Java
featuring AsyncGetCallTrace + perf_events (github.com)
<https://github.com/async-profiler/async-profiler#troubleshooting>
Best,
Yu Chen
David Christle <da...@discordapp.com.invalid> 于2023年10月14日周六
04:12写道：
> In the Wiki, this FLIP is motivated by:
>
> - That the current flamegraph functionality can only see operator-level
> stack traces, while async-profiler provides CPU/allocation/locks
> information, along with deeper Java & system call stack information.
> - Low configurability (e.g. cannot set the sampling interval) + the
> usability is limited to visual inspection of the flamegraph whenever it
> happens to read out.
>
> The current built-in flamegraph is extremely valuable and easy to use. But
> as a regular user of async-profiler for Flink applications, I agree these
> deficiencies are worth improving. It's exciting that async-profiler might
> be built-in, since it will make using it much easier.
>
> I'm a little confused, though, about the scope of functionality we'll have
> with this FLIP, in particular the use of itimer only & perf_events support.
> One of the questions near the end of the Wiki is whether sampling with
> perf_events will be supported. The current answer seems to say that only
> itimer mode will be supported, as this mode does not rely on perf_events
> being enabled.
>
> However, given that an aim of the FLIP is to support configurability (e.g.
> sampling intervals), is it that much more work to support configurability
> of the event & the other common options, too? If the '-e' event flag is
> fixed to itimer only, we can't use wall clock/alloc/cpu profiling modes.
> The Wiki mentions async-profiler's JNI interface will be used, which has
> 'event_str' as an input. So, it seems like supporting different event types
> (or even multiple event types in one profile) is possible.
>
> Regarding perf_events, it's true that it's disabled in many
> environments. But it is possible to enable it for debugging purposes. In
> our Kubernetes workloads, this means adding SYS_PTRACE and SYS_ADMIN to the
> securityContext, deploying the job, and then running:
>
> sysctl kernel.kptr_restrict=0
> sysctl kernel.perf_event_paranoid=1
>
> before starting async-profiler.
>
> It would be nice if dynamically changing the kernel parameters was built-in
> to this FLIP, somehow, as well, to set these parameters correctly before
> profiling. If the environment restricts changing these, that's fine. We can
> simply report to the user via the UI that setting them failed, and that the
> choice of profiling configurations is limited without them. I also think
> it's OK if `itimer` is the default in the UI, as it works under the
> broadest conditions. But given the motivation in the Wiki is that
> async-profiler can see detailed system call stack info, allocation, etc.,
> and that the async-profiler docs describe itimer mode as a "fallback"
> rather than the way the profiler is best used, it feels like this FLIP
> should support async-profiler's regular modes of operation & the other
> most-common configuration options. From my own experience, `cpu` (requiring
> perf_events) is a bit more accurate than `itimer`, and if I recall, and
> samples once per thread. `wall` is very useful to debug blocks on I/O or
> locks. Getting per-thread information is nice to drill down into specific
> parts of the Flink application, e.g. the flame graph lets me ignore the
> many other tasks running on TM & drill down into just the Source threads,
> when debugging a Source issue.
>
> Kind regards,
> David
>
> On Fri, Oct 13, 2023 at 1:45 AM Rui Fan <19...@gmail.com> wrote:
>
> > One minor comment:
> >
> > In general, the generic java profiler includes memory analysis,
> > cpu, thread, deadlock, etc. The FLIP title is java profiler, but
> > the FLIP just supports flamegraph at process level.
> > So the `powerful java profiler` title may not be suitable.
> > Would you mind updating the FLIP title?
> >
> > Best,
> > Rui
> >
> > On Fri, Oct 13, 2023 at 4:34 PM Yu Chen <yu...@gmail.com> wrote:
> >
> > > Hi all.
> > > If there are no further questions, we will start a vote on FLIP-375
> next
> > > week.
> > >
> > > Best regards,
> > > Yu Chen
> > >
> > >
> > > Yu Chen <yu...@gmail.com> 于2023年10月9日周一 17:24写道：
> > >
> > > > Hi all,
> > > >
> > > > Yun Tang and I are opening this thread to discuss our proposal to
> > > > integrate async-profiler's capabilities for profiling taskmananger
> > (e.g.,
> > > > generating flame graphs) in the Flink Web [1].
> > > >
> > > >
> > > > Currently, Flink provides ThreadDump and Operator-Level Flame Graphs
> by
> > > > sampling task threads. The results generated in such way missing the
> > > > relevant stack of java threads and system calls. The
> async-profiler[2]
> > > is a
> > > > low-overhead sampling profiler for Java, but the steps to use it in
> the
> > > > production environment are cumbersome and suffer from permissions and
> > > > security risks.
> > > >
> > > > Therefore, we propose adding rest APIs to provide the capability to
> > > invoke
> > > > async-profiler on multiple platforms through JNI, which can be easily
> > > > operated on Web UI. This enhancement will improve the efficiency and
> > > > experience of Flink users in identifying performance bottlenecks.
> > > >
> > > >
> > > >
> > > > Please refer to the FLIP document for more details about the proposed
> > > design
> > > > and implementation. We welcome any feedback and opinions on this
> > > proposal.
> > > >
> > > >
> > > >
> > > > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > > > taskmanagers - Apache Flink - Apache Software Foundation
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> > > >
> > > >
> > > > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> > > > profiler for Java featuring AsyncGetCallTrace + perf_events
> > > > <https://github.com/async-profiler/async-profiler>
> > > >
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > Yun Tang and Yu Chen
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Yun Tang <my...@live.com>.
Thanks for the response,
@Rui, actually async-profiler also supports memory allocation and wall-clock profiling, and I have updated the FILP to include these profiling options in the next developments. I think it deserves to be described as powerful.
@David, in our internal version, we also support the perf_events option, and I think we can include it as an extension in this FLIP. For the request of dynamically changing the kernel parameters, we would not include this in the FLIP as it might cause permissions issues.
Best
Yun Tang
________________________________
From: David Christle <da...@discordapp.com.INVALID>
Sent: Saturday, October 14, 2023 4:11
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
In the Wiki, this FLIP is motivated by:
- That the current flamegraph functionality can only see operator-level
stack traces, while async-profiler provides CPU/allocation/locks
information, along with deeper Java & system call stack information.
- Low configurability (e.g. cannot set the sampling interval) + the
usability is limited to visual inspection of the flamegraph whenever it
happens to read out.
The current built-in flamegraph is extremely valuable and easy to use. But
as a regular user of async-profiler for Flink applications, I agree these
deficiencies are worth improving. It's exciting that async-profiler might
be built-in, since it will make using it much easier.
I'm a little confused, though, about the scope of functionality we'll have
with this FLIP, in particular the use of itimer only & perf_events support.
One of the questions near the end of the Wiki is whether sampling with
perf_events will be supported. The current answer seems to say that only
itimer mode will be supported, as this mode does not rely on perf_events
being enabled.
However, given that an aim of the FLIP is to support configurability (e.g.
sampling intervals), is it that much more work to support configurability
of the event & the other common options, too? If the '-e' event flag is
fixed to itimer only, we can't use wall clock/alloc/cpu profiling modes.
The Wiki mentions async-profiler's JNI interface will be used, which has
'event_str' as an input. So, it seems like supporting different event types
(or even multiple event types in one profile) is possible.
Regarding perf_events, it's true that it's disabled in many
environments. But it is possible to enable it for debugging purposes. In
our Kubernetes workloads, this means adding SYS_PTRACE and SYS_ADMIN to the
securityContext, deploying the job, and then running:
sysctl kernel.kptr_restrict=0
sysctl kernel.perf_event_paranoid=1
before starting async-profiler.
It would be nice if dynamically changing the kernel parameters was built-in
to this FLIP, somehow, as well, to set these parameters correctly before
profiling. If the environment restricts changing these, that's fine. We can
simply report to the user via the UI that setting them failed, and that the
choice of profiling configurations is limited without them. I also think
it's OK if `itimer` is the default in the UI, as it works under the
broadest conditions. But given the motivation in the Wiki is that
async-profiler can see detailed system call stack info, allocation, etc.,
and that the async-profiler docs describe itimer mode as a "fallback"
rather than the way the profiler is best used, it feels like this FLIP
should support async-profiler's regular modes of operation & the other
most-common configuration options. From my own experience, `cpu` (requiring
perf_events) is a bit more accurate than `itimer`, and if I recall, and
samples once per thread. `wall` is very useful to debug blocks on I/O or
locks. Getting per-thread information is nice to drill down into specific
parts of the Flink application, e.g. the flame graph lets me ignore the
many other tasks running on TM & drill down into just the Source threads,
when debugging a Source issue.
Kind regards,
David
On Fri, Oct 13, 2023 at 1:45 AM Rui Fan <19...@gmail.com> wrote:
> One minor comment:
>
> In general, the generic java profiler includes memory analysis,
> cpu, thread, deadlock, etc. The FLIP title is java profiler, but
> the FLIP just supports flamegraph at process level.
> So the `powerful java profiler` title may not be suitable.
> Would you mind updating the FLIP title?
>
> Best,
> Rui
>
> On Fri, Oct 13, 2023 at 4:34 PM Yu Chen <yu...@gmail.com> wrote:
>
> > Hi all.
> > If there are no further questions, we will start a vote on FLIP-375 next
> > week.
> >
> > Best regards,
> > Yu Chen
> >
> >
> > Yu Chen <yu...@gmail.com> 于2023年10月9日周一 17:24写道：
> >
> > > Hi all,
> > >
> > > Yun Tang and I are opening this thread to discuss our proposal to
> > > integrate async-profiler's capabilities for profiling taskmananger
> (e.g.,
> > > generating flame graphs) in the Flink Web [1].
> > >
> > >
> > > Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
> > > sampling task threads. The results generated in such way missing the
> > > relevant stack of java threads and system calls. The async-profiler[2]
> > is a
> > > low-overhead sampling profiler for Java, but the steps to use it in the
> > > production environment are cumbersome and suffer from permissions and
> > > security risks.
> > >
> > > Therefore, we propose adding rest APIs to provide the capability to
> > invoke
> > > async-profiler on multiple platforms through JNI, which can be easily
> > > operated on Web UI. This enhancement will improve the efficiency and
> > > experience of Flink users in identifying performance bottlenecks.
> > >
> > >
> > >
> > > Please refer to the FLIP document for more details about the proposed
> > design
> > > and implementation. We welcome any feedback and opinions on this
> > proposal.
> > >
> > >
> > >
> > > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > > taskmanagers - Apache Flink - Apache Software Foundation
> > > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> > >
> > >
> > > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> > > profiler for Java featuring AsyncGetCallTrace + perf_events
> > > <https://github.com/async-profiler/async-profiler>
> > >
> > >
> > >
> > > Best regards,
> > >
> > > Yun Tang and Yu Chen
> > >
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by David Christle <da...@discordapp.com.INVALID>.
In the Wiki, this FLIP is motivated by:
- That the current flamegraph functionality can only see operator-level
stack traces, while async-profiler provides CPU/allocation/locks
information, along with deeper Java & system call stack information.
- Low configurability (e.g. cannot set the sampling interval) + the
usability is limited to visual inspection of the flamegraph whenever it
happens to read out.
The current built-in flamegraph is extremely valuable and easy to use. But
as a regular user of async-profiler for Flink applications, I agree these
deficiencies are worth improving. It's exciting that async-profiler might
be built-in, since it will make using it much easier.
I'm a little confused, though, about the scope of functionality we'll have
with this FLIP, in particular the use of itimer only & perf_events support.
One of the questions near the end of the Wiki is whether sampling with
perf_events will be supported. The current answer seems to say that only
itimer mode will be supported, as this mode does not rely on perf_events
being enabled.
However, given that an aim of the FLIP is to support configurability (e.g.
sampling intervals), is it that much more work to support configurability
of the event & the other common options, too? If the '-e' event flag is
fixed to itimer only, we can't use wall clock/alloc/cpu profiling modes.
The Wiki mentions async-profiler's JNI interface will be used, which has
'event_str' as an input. So, it seems like supporting different event types
(or even multiple event types in one profile) is possible.
Regarding perf_events, it's true that it's disabled in many
environments. But it is possible to enable it for debugging purposes. In
our Kubernetes workloads, this means adding SYS_PTRACE and SYS_ADMIN to the
securityContext, deploying the job, and then running:
sysctl kernel.kptr_restrict=0
sysctl kernel.perf_event_paranoid=1
before starting async-profiler.
It would be nice if dynamically changing the kernel parameters was built-in
to this FLIP, somehow, as well, to set these parameters correctly before
profiling. If the environment restricts changing these, that's fine. We can
simply report to the user via the UI that setting them failed, and that the
choice of profiling configurations is limited without them. I also think
it's OK if `itimer` is the default in the UI, as it works under the
broadest conditions. But given the motivation in the Wiki is that
async-profiler can see detailed system call stack info, allocation, etc.,
and that the async-profiler docs describe itimer mode as a "fallback"
rather than the way the profiler is best used, it feels like this FLIP
should support async-profiler's regular modes of operation & the other
most-common configuration options. From my own experience, `cpu` (requiring
perf_events) is a bit more accurate than `itimer`, and if I recall, and
samples once per thread. `wall` is very useful to debug blocks on I/O or
locks. Getting per-thread information is nice to drill down into specific
parts of the Flink application, e.g. the flame graph lets me ignore the
many other tasks running on TM & drill down into just the Source threads,
when debugging a Source issue.
Kind regards,
David
On Fri, Oct 13, 2023 at 1:45 AM Rui Fan <19...@gmail.com> wrote:
> One minor comment:
>
> In general, the generic java profiler includes memory analysis,
> cpu, thread, deadlock, etc. The FLIP title is java profiler, but
> the FLIP just supports flamegraph at process level.
> So the `powerful java profiler` title may not be suitable.
> Would you mind updating the FLIP title?
>
> Best,
> Rui
>
> On Fri, Oct 13, 2023 at 4:34 PM Yu Chen <yu...@gmail.com> wrote:
>
> > Hi all.
> > If there are no further questions, we will start a vote on FLIP-375 next
> > week.
> >
> > Best regards,
> > Yu Chen
> >
> >
> > Yu Chen <yu...@gmail.com> 于2023年10月9日周一 17:24写道：
> >
> > > Hi all,
> > >
> > > Yun Tang and I are opening this thread to discuss our proposal to
> > > integrate async-profiler's capabilities for profiling taskmananger
> (e.g.,
> > > generating flame graphs) in the Flink Web [1].
> > >
> > >
> > > Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
> > > sampling task threads. The results generated in such way missing the
> > > relevant stack of java threads and system calls. The async-profiler[2]
> > is a
> > > low-overhead sampling profiler for Java, but the steps to use it in the
> > > production environment are cumbersome and suffer from permissions and
> > > security risks.
> > >
> > > Therefore, we propose adding rest APIs to provide the capability to
> > invoke
> > > async-profiler on multiple platforms through JNI, which can be easily
> > > operated on Web UI. This enhancement will improve the efficiency and
> > > experience of Flink users in identifying performance bottlenecks.
> > >
> > >
> > >
> > > Please refer to the FLIP document for more details about the proposed
> > design
> > > and implementation. We welcome any feedback and opinions on this
> > proposal.
> > >
> > >
> > >
> > > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > > taskmanagers - Apache Flink - Apache Software Foundation
> > > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> > >
> > >
> > > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> > > profiler for Java featuring AsyncGetCallTrace + perf_events
> > > <https://github.com/async-profiler/async-profiler>
> > >
> > >
> > >
> > > Best regards,
> > >
> > > Yun Tang and Yu Chen
> > >
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Rui Fan <19...@gmail.com>.
One minor comment:
In general, the generic java profiler includes memory analysis,
cpu, thread, deadlock, etc. The FLIP title is java profiler, but
the FLIP just supports flamegraph at process level.
So the `powerful java profiler` title may not be suitable.
Would you mind updating the FLIP title?
Best,
Rui
On Fri, Oct 13, 2023 at 4:34 PM Yu Chen <yu...@gmail.com> wrote:
> Hi all.
> If there are no further questions, we will start a vote on FLIP-375 next
> week.
>
> Best regards,
> Yu Chen
>
>
> Yu Chen <yu...@gmail.com> 于2023年10月9日周一 17:24写道：
>
> > Hi all,
> >
> > Yun Tang and I are opening this thread to discuss our proposal to
> > integrate async-profiler's capabilities for profiling taskmananger (e.g.,
> > generating flame graphs) in the Flink Web [1].
> >
> >
> > Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
> > sampling task threads. The results generated in such way missing the
> > relevant stack of java threads and system calls. The async-profiler[2]
> is a
> > low-overhead sampling profiler for Java, but the steps to use it in the
> > production environment are cumbersome and suffer from permissions and
> > security risks.
> >
> > Therefore, we propose adding rest APIs to provide the capability to
> invoke
> > async-profiler on multiple platforms through JNI, which can be easily
> > operated on Web UI. This enhancement will improve the efficiency and
> > experience of Flink users in identifying performance bottlenecks.
> >
> >
> >
> > Please refer to the FLIP document for more details about the proposed
> design
> > and implementation. We welcome any feedback and opinions on this
> proposal.
> >
> >
> >
> > [1] FLIP-375: Built-in cross-platform powerful java profiler on
> > taskmanagers - Apache Flink - Apache Software Foundation
> > <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers
> >
> >
> > [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> > profiler for Java featuring AsyncGetCallTrace + perf_events
> > <https://github.com/async-profiler/async-profiler>
> >
> >
> >
> > Best regards,
> >
> > Yun Tang and Yu Chen
> >
>
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Posted by Yu Chen <yu...@gmail.com>.
Hi all.
If there are no further questions, we will start a vote on FLIP-375 next
week.
Best regards,
Yu Chen
Yu Chen <yu...@gmail.com> 于2023年10月9日周一 17:24写道：
> Hi all,
>
> Yun Tang and I are opening this thread to discuss our proposal to
> integrate async-profiler's capabilities for profiling taskmananger (e.g.,
> generating flame graphs) in the Flink Web [1].
>
>
> Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by
> sampling task threads. The results generated in such way missing the
> relevant stack of java threads and system calls. The async-profiler[2] is a
> low-overhead sampling profiler for Java, but the steps to use it in the
> production environment are cumbersome and suffer from permissions and
> security risks.
>
> Therefore, we propose adding rest APIs to provide the capability to invoke
> async-profiler on multiple platforms through JNI, which can be easily
> operated on Web UI. This enhancement will improve the efficiency and
> experience of Flink users in identifying performance bottlenecks.
>
>
>
> Please refer to the FLIP document for more details about the proposed design
> and implementation. We welcome any feedback and opinions on this proposal.
>
>
>
> [1] FLIP-375: Built-in cross-platform powerful java profiler on
> taskmanagers - Apache Flink - Apache Software Foundation
> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler+on+taskmanagers>
>
> [2] GitHub - async-profiler/async-profiler: Sampling CPU and HEAP
> profiler for Java featuring AsyncGetCallTrace + perf_events
> <https://github.com/async-profiler/async-profiler>
>
>
>
> Best regards,
>
> Yun Tang and Yu Chen
>