You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Deepak Sharma <de...@gmail.com> on 2016/05/16 05:25:26 UTC

Debug spark core and streaming programs in scala

Hi
I have scala program consisting of spark core and spark streaming APIs
Is there any open source tool that i can use to debug the program for
performance reasons?
My primary interest is to find the block of codes that would be exeuted on
driver and what would go to the executors.
Is there JMX extension of Spark?

-- 
Thanks
Deepak

Re: Debug spark core and streaming programs in scala

Posted by Ted Yu <yu...@gmail.com>.

From https://spark.apache.org/docs/latest/monitoring.html#metrics :

   - JmxSink: Registers metrics for viewing in a JMX console.

FYI

On Sun, May 15, 2016 at 11:54 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> wrote:

> Have you tried Spark GUI on 4040. This will show jobs being executed by
> executors is each stage and the line of code as well.
>
> [image: Inline images 1]
>
> Also command line tools like jps and jmonitor
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 16 May 2016 at 06:25, Deepak Sharma <de...@gmail.com> wrote:
>
>> Hi
>> I have scala program consisting of spark core and spark streaming APIs
>> Is there any open source tool that i can use to debug the program for
>> performance reasons?
>> My primary interest is to find the block of codes that would be exeuted
>> on driver and what would go to the executors.
>> Is there JMX extension of Spark?
>>
>> --
>> Thanks
>> Deepak
>>
>>
>

Re: Debug spark core and streaming programs in scala

Posted by Mich Talebzadeh <mi...@gmail.com>.

Have you tried Spark GUI on 4040. This will show jobs being executed by
executors is each stage and the line of code as well.

[image: Inline images 1]

Also command line tools like jps and jmonitor

HTH


Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 16 May 2016 at 06:25, Deepak Sharma <de...@gmail.com> wrote:

> Hi
> I have scala program consisting of spark core and spark streaming APIs
> Is there any open source tool that i can use to debug the program for
> performance reasons?
> My primary interest is to find the block of codes that would be exeuted on
> driver and what would go to the executors.
> Is there JMX extension of Spark?
>
> --
> Thanks
> Deepak
>
>