Re: Running Spark through a debugger

[Nicholas Chammas <ni...@gmail.com>]

Finally revisiting this issue. I can now build Spark through IntelliJ (I
had to delete `.idea/` and reload the project, making sure to import the
Maven config), but am struggling to get the breakpoint/debugging to work.

I setup IntelliJ per the instructions under Debug Spark Remotely
<http://spark.apache.org/developer-tools.html>, but the debugger never
seems to connect to the JVM. If I switch the debugger mode from "Listen to
remote JVM" (which is recommended by the guide) to "Attach to remote JVM",
I can get the debugger to talk to the JVM, but my breakpoint still doesn't
fire.

So to elaborate, I have a) a breakpoint set inside CatalogSuite.scala, and
b) the IntelliJ debugger running in "Listen" mode. Then I run this:

```
$ ./build/sbt
> set javaOptions in Test +=
"-agentlib:jdwp=transport=dt_socket,server=n,address=192.168.1.146:5005,suspend=y,onthrow=<FQ
exception class name>,onuncaught=<y/n>"
> testOnly org.apache.spark.sql.internal.CatalogSuite
```

(I copied those Java options from the IntelliJ debug config window.)

But the breakpoint doesn't fire. Any ideas as to what I'm missing?

On Tue, Dec 17, 2019 at 12:36 AM Sean Owen <sr...@gmail.com> wrote:

> I just make a new test suite or something, set breakpoints, and
> execute it in IJ. That generally works fine. You may need to set the
> run configuration to have the right working dir (Spark project root),
> and set the right system property to say 'this is running in a test'
> in some cases. What are you having trouble with, does it build?
>
> On Mon, Dec 16, 2019 at 11:27 PM Nicholas Chammas
> <ni...@gmail.com> wrote:
> >
> > I normally stick to the Python parts of Spark, but I am interested in
> walking through the DSv2 code and understanding how it works. I tried
> following the "IDE Setup" section of the developer tools page, but quickly
> hit several problems loading the project.
> >
> > Can anyone point to a tutorial or guide on running Spark through a
> debugger, for folks like me who are not regular IntelliJ or Scala users?
> >
> > My goal is just to be able to run something like
> `spark.read.text(...).show()` and follow it step by step through the
> codebase.
> >
> > Nick
> >
>