Running Spark through a debugger

[Nicholas Chammas <ni...@gmail.com>]

I normally stick to the Python parts of Spark, but I am interested in
walking through the DSv2 code and understanding how it works. I tried
following the "IDE Setup" section of the developer tools
<https://spark.apache.org/developer-tools.html> page, but quickly hit
several problems loading the project.

Can anyone point to a tutorial or guide on running Spark through a
debugger, for folks like me who are not regular IntelliJ or Scala users?

My goal is just to be able to run something like
`spark.read.text(...).show()` and follow it step by step through the
codebase.

Nick

Re: Running Spark through a debugger

[Nicholas Chammas <ni...@gmail.com>]

Finally revisiting this issue. I can now build Spark through IntelliJ (I
had to delete `.idea/` and reload the project, making sure to import the
Maven config), but am struggling to get the breakpoint/debugging to work.

I setup IntelliJ per the instructions under Debug Spark Remotely
<http://spark.apache.org/developer-tools.html>, but the debugger never
seems to connect to the JVM. If I switch the debugger mode from "Listen to
remote JVM" (which is recommended by the guide) to "Attach to remote JVM",
I can get the debugger to talk to the JVM, but my breakpoint still doesn't
fire.

So to elaborate, I have a) a breakpoint set inside CatalogSuite.scala, and
b) the IntelliJ debugger running in "Listen" mode. Then I run this:

```
$ ./build/sbt
> set javaOptions in Test +=
"-agentlib:jdwp=transport=dt_socket,server=n,address=192.168.1.146:5005,suspend=y,onthrow=<FQ
exception class name>,onuncaught=<y/n>"
> testOnly org.apache.spark.sql.internal.CatalogSuite
```

(I copied those Java options from the IntelliJ debug config window.)

But the breakpoint doesn't fire. Any ideas as to what I'm missing?

On Tue, Dec 17, 2019 at 12:36 AM Sean Owen <sr...@gmail.com> wrote:

> I just make a new test suite or something, set breakpoints, and
> execute it in IJ. That generally works fine. You may need to set the
> run configuration to have the right working dir (Spark project root),
> and set the right system property to say 'this is running in a test'
> in some cases. What are you having trouble with, does it build?
>
> On Mon, Dec 16, 2019 at 11:27 PM Nicholas Chammas
> <ni...@gmail.com> wrote:
> >
> > I normally stick to the Python parts of Spark, but I am interested in
> walking through the DSv2 code and understanding how it works. I tried
> following the "IDE Setup" section of the developer tools page, but quickly
> hit several problems loading the project.
> >
> > Can anyone point to a tutorial or guide on running Spark through a
> debugger, for folks like me who are not regular IntelliJ or Scala users?
> >
> > My goal is just to be able to run something like
> `spark.read.text(...).show()` and follow it step by step through the
> codebase.
> >
> > Nick
> >
>

Re: Running Spark through a debugger

[Sean Owen <sr...@gmail.com>]

I just make a new test suite or something, set breakpoints, and
execute it in IJ. That generally works fine. You may need to set the
run configuration to have the right working dir (Spark project root),
and set the right system property to say 'this is running in a test'
in some cases. What are you having trouble with, does it build?

On Mon, Dec 16, 2019 at 11:27 PM Nicholas Chammas
<ni...@gmail.com> wrote:
>
> I normally stick to the Python parts of Spark, but I am interested in walking through the DSv2 code and understanding how it works. I tried following the "IDE Setup" section of the developer tools page, but quickly hit several problems loading the project.
>
> Can anyone point to a tutorial or guide on running Spark through a debugger, for folks like me who are not regular IntelliJ or Scala users?
>
> My goal is just to be able to run something like `spark.read.text(...).show()` and follow it step by step through the codebase.
>
> Nick
>

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org