Debugging Apache Spark clustered application from Eclipse

[nitinkalra2000 <ni...@gmail.com>]

I am trying to debug Spark application running on eclipse in
clustered/distributed environment but not able to succeed. Application is
java based and I am running it through Eclipse. Configurations to spark for
Master/worker is provided through Java only.

Though I can debug the code on driver side but as the code flow moves in
Spark(i.e call to .map(..)), the debugger doesn't stop. Because that code is
running in Workers JVM.

Is there anyway I can achieve this ?

I have tried giving following configurations in Tomcat through Eclipse :
-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=7761,suspend=n

and setting respective port in Debug->remote java application.

But after these settings I get the error: Failed to connect to remote VM.
Connection Refused

Note: I have tried this on Windows as well as on Linux(CentOs) environment.

If anybody has any solution to this, please help.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Debugging-Apache-Spark-clustered-application-from-Eclipse-tp23483.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Debugging Apache Spark clustered application from Eclipse

[Yana Kadiyska <ya...@gmail.com>]

Pass that debug string to your executor like this:  --conf
spark.executor.extraJavaOptions="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=
7761". When your executor is launched it will send debug information on
port 7761. When you attach the Eclipse debugger, you need to have the
IP/host of that executor as the remote process to attach to. This is pretty
standard "remote debugging" procedure .

In the interest of sanity I'd recommend you run with just 1 executor when
you're debugging...makes life easier (of course the easiest thing is if you
can do the debugging in local[*] mode but some bugs don't manifest that way)

On Thu, Jun 25, 2015 at 1:17 AM, nitinkalra2000 <ni...@gmail.com>
wrote:

> I am trying to debug Spark application running on eclipse in
> clustered/distributed environment but not able to succeed. Application is
> java based and I am running it through Eclipse. Configurations to spark for
> Master/worker is provided through Java only.
>
> Though I can debug the code on driver side but as the code flow moves in
> Spark(i.e call to .map(..)), the debugger doesn't stop. Because that code
> is
> running in Workers JVM.
>
> Is there anyway I can achieve this ?
>
> I have tried giving following configurations in Tomcat through Eclipse :
> -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=7761,suspend=n
>
> and setting respective port in Debug->remote java application.
>
> But after these settings I get the error: Failed to connect to remote VM.
> Connection Refused
>
> Note: I have tried this on Windows as well as on Linux(CentOs) environment.
>
> If anybody has any solution to this, please help.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Debugging-Apache-Spark-clustered-application-from-Eclipse-tp23483.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>