You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Prasad Bhalerao <pr...@gmail.com> on 2022/01/20 05:18:04 UTC
Profiling spark application
Hello,
Is there any way we can profile spark applications which will show no. of
invocations of spark api and their execution time etc etc just the way
jprofiler shows all the details?
Thanks,
Prasad
Re: Profiling spark application
Posted by Wes Peng <we...@stackdev.eu>.
Give a look at this:
https://github.com/LucaCanali/sparkMeasure
On 2022/1/20 1:18, Prasad Bhalerao wrote:
> Is there any way we can profile spark applications which will show no.
> of invocations of spark api and their execution time etc etc just the
> way jprofiler shows all the details?
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Profiling spark application
Posted by Deepak Sharma <de...@gmail.com>.
You can take a look at jvm profiler that was open sourced by uber:
https://github.com/uber-common/jvm-profiler
On Thu, Jan 20, 2022 at 11:20 AM Prasad Bhalerao <
prasadbhalerao1983@gmail.com> wrote:
> Hi,
>
> It will require code changes and I am looking at some third party code , I
> am looking for something which I can just hook to jvm and get the stats..
>
> Thanks,
> Prasad
>
> On Thu, Jan 20, 2022 at 11:00 AM Sonal Goyal <so...@gmail.com>
> wrote:
>
>> Hi Prasad,
>>
>> Have you checked the SparkListener -
>> https://mallikarjuna_g.gitbooks.io/spark/content/spark-SparkListener.html
>> ?
>>
>> Cheers,
>> Sonal
>> https://github.com/zinggAI/zingg
>>
>>
>>
>> On Thu, Jan 20, 2022 at 10:49 AM Prasad Bhalerao <
>> prasadbhalerao1983@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Is there any way we can profile spark applications which will show no.
>>> of invocations of spark api and their execution time etc etc just the way
>>> jprofiler shows all the details?
>>>
>>>
>>> Thanks,
>>> Prasad
>>>
>>
--
Thanks
Deepak
www.bigdatabig.com
www.keosha.net
Re: Profiling spark application
Posted by Prasad Bhalerao <pr...@gmail.com>.
Hi,
It will require code changes and I am looking at some third party code , I
am looking for something which I can just hook to jvm and get the stats..
Thanks,
Prasad
On Thu, Jan 20, 2022 at 11:00 AM Sonal Goyal <so...@gmail.com> wrote:
> Hi Prasad,
>
> Have you checked the SparkListener -
> https://mallikarjuna_g.gitbooks.io/spark/content/spark-SparkListener.html
> ?
>
> Cheers,
> Sonal
> https://github.com/zinggAI/zingg
>
>
>
> On Thu, Jan 20, 2022 at 10:49 AM Prasad Bhalerao <
> prasadbhalerao1983@gmail.com> wrote:
>
>> Hello,
>>
>> Is there any way we can profile spark applications which will show no. of
>> invocations of spark api and their execution time etc etc just the way
>> jprofiler shows all the details?
>>
>>
>> Thanks,
>> Prasad
>>
>
Re: Profiling spark application
Posted by Sonal Goyal <so...@gmail.com>.
Hi Prasad,
Have you checked the SparkListener -
https://mallikarjuna_g.gitbooks.io/spark/content/spark-SparkListener.html ?
Cheers,
Sonal
https://github.com/zinggAI/zingg
On Thu, Jan 20, 2022 at 10:49 AM Prasad Bhalerao <
prasadbhalerao1983@gmail.com> wrote:
> Hello,
>
> Is there any way we can profile spark applications which will show no. of
> invocations of spark api and their execution time etc etc just the way
> jprofiler shows all the details?
>
>
> Thanks,
> Prasad
>