Newbie: Help debugging Drill

[Matthew Mucker <ma...@mucker.net>]

I became a new Drill user last week only to discover that Drill would crash
with an IndexOutOfBounds exception on one of my queries. Some searching and
testing later, my best guess is that I'm hitting DRILL-5451.

 

Since this is currently a showstopper for me, and since I might learn
something by doing so, I thought I'd give it a go and try to debug this
problem and see if I might be able to contribute back.

 

I'm finding that when it comes to debugging, I really don't know what I'm
doing, and could use some help. 

 

Preferably, help made up of small words.

 

I'm running Drill 1.11.0 on Windows 7. To start Drill in debug mode, my best
guess was to edit line 182 of sqlline.bat to read:

 

set DRILL_SHELL_JAVA_OPTS=%DRILL_SHELL_JAVA_OPTS%
-Dlog.path="%DRILL_LOG_DIR%\sqlline.log"
-Dlog.query.path="%DRILL_LOG_DIR%\sqlline_queries.log" -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000

 

When I run sqlline.bat -u "jdbc:drill:sz=local" now, I get a message
"Listening for transport dt_socket at address: 8000" which, I expect, means
that the runtime is waiting for the debugger to attach.

 

I downloaded the Drill source and built with Maven. Using Eclipse Neon, I
imported the project. I created a debug configuration for a Remote Java
Application for project drill-common, on machine localhost:8000.

 

When I start the debug session, sqlline finishes launching and I get a
prompt at which I can enter a SQL command. Which suggests to me that the
debugger is in fact attached.

 

Inside Eclipse, I set breakpoints on lines 114 and 122 of
drill-java-exec/src/main/java/io.netty.buffer/DrillBuf.java.

 

However, when I repro my issue, I get the IndexOutOfBoundsException in
sqlline, but there's no indication in Eclipse that the debugger has broken
in, and I see no facilities to examine the stack trace or the local
variables.

 

What do I do next?

Re: Newbie: Help debugging Drill

[Charles Givre <cg...@gmail.com>]

HI Matthew, 
Can you describe the data you are querying and the query you are trying to execute?
— C


> On Oct 2, 2017, at 17:19, Matthew Mucker <ma...@mucker.net> wrote:
> 
> I became a new Drill user last week only to discover that Drill would crash
> with an IndexOutOfBounds exception on one of my queries. Some searching and
> testing later, my best guess is that I'm hitting DRILL-5451.
> 
> 
> 
> Since this is currently a showstopper for me, and since I might learn
> something by doing so, I thought I'd give it a go and try to debug this
> problem and see if I might be able to contribute back.
> 
> 
> 
> I'm finding that when it comes to debugging, I really don't know what I'm
> doing, and could use some help. 
> 
> 
> 
> Preferably, help made up of small words.
> 
> 
> 
> I'm running Drill 1.11.0 on Windows 7. To start Drill in debug mode, my best
> guess was to edit line 182 of sqlline.bat to read:
> 
> 
> 
> set DRILL_SHELL_JAVA_OPTS=%DRILL_SHELL_JAVA_OPTS%
> -Dlog.path="%DRILL_LOG_DIR%\sqlline.log"
> -Dlog.query.path="%DRILL_LOG_DIR%\sqlline_queries.log" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000
> 
> 
> 
> When I run sqlline.bat -u "jdbc:drill:sz=local" now, I get a message
> "Listening for transport dt_socket at address: 8000" which, I expect, means
> that the runtime is waiting for the debugger to attach.
> 
> 
> 
> I downloaded the Drill source and built with Maven. Using Eclipse Neon, I
> imported the project. I created a debug configuration for a Remote Java
> Application for project drill-common, on machine localhost:8000.
> 
> 
> 
> When I start the debug session, sqlline finishes launching and I get a
> prompt at which I can enter a SQL command. Which suggests to me that the
> debugger is in fact attached.
> 
> 
> 
> Inside Eclipse, I set breakpoints on lines 114 and 122 of
> drill-java-exec/src/main/java/io.netty.buffer/DrillBuf.java.
> 
> 
> 
> However, when I repro my issue, I get the IndexOutOfBoundsException in
> sqlline, but there's no indication in Eclipse that the debugger has broken
> in, and I see no facilities to examine the stack trace or the local
> variables.
> 
> 
> 
> What do I do next?
> 

Re: Newbie: Help debugging Drill

[Paul Rogers <pr...@mapr.com>]

Hi Matthew,

Debugging Drill is a bit of an art.

Your direction is fine. However, the source code you forked from Apache Drill has evolved significantly since the 1.11 build, so you’ll want to build your own Drill if you want to do remote debugging.

> mvn clean install -DskipTests

Will get you started.

You added remote debug options to the sqlline (client) script. This works only if you are running Drill embedded in Sqlline, which is one of two modes that sqlline can run. Be sure you are, in fact, running Drill embedded in this case.

You can also launch Drill as a server (change the DRILLBIT_JAVA_OPTS in drill-env.sh to include remote debugging). Then, attach the debugger, and lastly fire up sqlline to connect remotely.

Note that Drill is a highly multi-threaded server. You really need to be familiar with code structure to know exactly where to look. The CSV area is a bit complex, with many areas interacting, so it will take some sleuthing to track down the problem.

It is also possible to debug Drill directly from Eclipse, running embedded. See the history of this list fro some hints if you want to go down that path.

Thanks,

- Paul

> On Oct 2, 2017, at 2:19 PM, Matthew Mucker <ma...@mucker.net> wrote:
> 
> I became a new Drill user last week only to discover that Drill would crash
> with an IndexOutOfBounds exception on one of my queries. Some searching and
> testing later, my best guess is that I'm hitting DRILL-5451.
> 
> 
> 
> Since this is currently a showstopper for me, and since I might learn
> something by doing so, I thought I'd give it a go and try to debug this
> problem and see if I might be able to contribute back.
> 
> 
> 
> I'm finding that when it comes to debugging, I really don't know what I'm
> doing, and could use some help. 
> 
> 
> 
> Preferably, help made up of small words.
> 
> 
> 
> I'm running Drill 1.11.0 on Windows 7. To start Drill in debug mode, my best
> guess was to edit line 182 of sqlline.bat to read:
> 
> 
> 
> set DRILL_SHELL_JAVA_OPTS=%DRILL_SHELL_JAVA_OPTS%
> -Dlog.path="%DRILL_LOG_DIR%\sqlline.log"
> -Dlog.query.path="%DRILL_LOG_DIR%\sqlline_queries.log" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000
> 
> 
> 
> When I run sqlline.bat -u "jdbc:drill:sz=local" now, I get a message
> "Listening for transport dt_socket at address: 8000" which, I expect, means
> that the runtime is waiting for the debugger to attach.
> 
> 
> 
> I downloaded the Drill source and built with Maven. Using Eclipse Neon, I
> imported the project. I created a debug configuration for a Remote Java
> Application for project drill-common, on machine localhost:8000.
> 
> 
> 
> When I start the debug session, sqlline finishes launching and I get a
> prompt at which I can enter a SQL command. Which suggests to me that the
> debugger is in fact attached.
> 
> 
> 
> Inside Eclipse, I set breakpoints on lines 114 and 122 of
> drill-java-exec/src/main/java/io.netty.buffer/DrillBuf.java.
> 
> 
> 
> However, when I repro my issue, I get the IndexOutOfBoundsException in
> sqlline, but there's no indication in Eclipse that the debugger has broken
> in, and I see no facilities to examine the stack trace or the local
> variables.
> 
> 
> 
> What do I do next?
>