You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Reth RM <re...@gmail.com> on 2017/06/23 17:46:21 UTC
Spark job profiler results showing high TCP cpu time
Running a spark job on local machine and profiler results indicate that
highest time spent in
*sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.* Screenshot of
profiler result can be seen here : https://jpst.it/10i-V
Spark job(program) is performing IO (sc.wholeTextFile method of spark
apis), Reads files from local file system and analyses the text to obtain
tokens.
Any thoughts and suggestions?
Thanks.
Re: Spark job profiler results showing high TCP cpu time
Posted by Reth RM <re...@gmail.com>.
I am using visual vm: https://github.com/krasa/VisualVMLauncher
@Marcelo, thank you for the reply, that was helpful.
On Fri, Jun 23, 2017 at 12:48 PM, Eduardo Mello <ee...@gmail.com>
wrote:
> what program do u use to profile Spark?
>
> On Fri, Jun 23, 2017 at 3:07 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
>> That thread looks like the connection between the Spark process and
>> jvisualvm. It's expected to show high up when doing sampling if the
>> app is not doing much else.
>>
>> On Fri, Jun 23, 2017 at 10:46 AM, Reth RM <re...@gmail.com> wrote:
>> > Running a spark job on local machine and profiler results indicate that
>> > highest time spent in sun.rmi.transport.tcp.TCPTrans
>> port$ConnectionHandler.
>> > Screenshot of profiler result can be seen here : https://jpst.it/10i-V
>> >
>> > Spark job(program) is performing IO (sc.wholeTextFile method of spark
>> apis),
>> > Reads files from local file system and analyses the text to obtain
>> tokens.
>> >
>> > Any thoughts and suggestions?
>> >
>> > Thanks.
>> >
>>
>>
>>
>> --
>> Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
>
Re: Spark job profiler results showing high TCP cpu time
Posted by Eduardo Mello <ee...@gmail.com>.
what program do u use to profile Spark?
On Fri, Jun 23, 2017 at 3:07 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> That thread looks like the connection between the Spark process and
> jvisualvm. It's expected to show high up when doing sampling if the
> app is not doing much else.
>
> On Fri, Jun 23, 2017 at 10:46 AM, Reth RM <re...@gmail.com> wrote:
> > Running a spark job on local machine and profiler results indicate that
> > highest time spent in sun.rmi.transport.tcp.TCPTransport$
> ConnectionHandler.
> > Screenshot of profiler result can be seen here : https://jpst.it/10i-V
> >
> > Spark job(program) is performing IO (sc.wholeTextFile method of spark
> apis),
> > Reads files from local file system and analyses the text to obtain
> tokens.
> >
> > Any thoughts and suggestions?
> >
> > Thanks.
> >
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: Spark job profiler results showing high TCP cpu time
Posted by Marcelo Vanzin <va...@cloudera.com>.
That thread looks like the connection between the Spark process and
jvisualvm. It's expected to show high up when doing sampling if the
app is not doing much else.
On Fri, Jun 23, 2017 at 10:46 AM, Reth RM <re...@gmail.com> wrote:
> Running a spark job on local machine and profiler results indicate that
> highest time spent in sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.
> Screenshot of profiler result can be seen here : https://jpst.it/10i-V
>
> Spark job(program) is performing IO (sc.wholeTextFile method of spark apis),
> Reads files from local file system and analyses the text to obtain tokens.
>
> Any thoughts and suggestions?
>
> Thanks.
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org