[GitHub] [spark-website] srowen commented on a change in pull request #280: Add documentation for debugging with Maven and IntelliJ.

[GitBox <gi...@apache.org>]

srowen commented on a change in pull request #280:
URL: https://github.com/apache/spark-website/pull/280#discussion_r453363004



##########
File path: developer-tools.md
##########
@@ -435,6 +435,44 @@ Error:(147, 9) value q is not a member of StringContext
         q"""
         ^ 
 ```
+
+<h4>Debugging Spark unit tests with Intellj and Maven</h4>
+This part will show how to debug spark locally with maven.
+
+- Once the spark project in imported as a maven project install spark using the following command.

Review comment:
       Likewise these as unordered lists is distracting, just paragraphs. This needs formatting and proofreading

##########
File path: developer-tools.md
##########
@@ -435,6 +435,44 @@ Error:(147, 9) value q is not a member of StringContext
         q"""
         ^ 
 ```
+
+<h4>Debugging Spark unit tests with Intellj and Maven</h4>
+This part will show how to debug spark locally with maven.

Review comment:
       Fix things like spark -> Spark, maven -> Maven
   Add a blank line above.

##########
File path: developer-tools.md
##########
@@ -435,6 +435,44 @@ Error:(147, 9) value q is not a member of StringContext
         q"""
         ^ 
 ```
+
+<h4>Debugging Spark unit tests with Intellj and Maven</h4>
+This part will show how to debug spark locally with maven.
+
+- Once the spark project in imported as a maven project install spark using the following command.
+    ```
+    $ ./build/mvn -DskipTests clean install
+    ``` 
+    Then open the maven tab in Intellij(usually present in the top right corner) and reload all maven projects.
+    Now, open any unit test and right click on the play button for debugging options. A good way to test 
+    your changes or understand some part of the code is to write a dummy unit test, add a few breakpoints
+    to the part of the code that you are interested in and run the debugger. You could right click on a test Suite
+    and debug/run all tests in that Suite
+
+- To keep your development spark from polluting your .m2 cache you can specify a custom maven repository location
+during install.
+    ```
+    $ ./build/mvn -DskipTests clean install -Dmaven.repo.local=/Users/dummyuser/repos/Jiras/SPARK-xyz/.m2
+    ```
+    Accordingly Intellij has to be configured to point to this custom location of m2 cache. To do this
+
+    Go to Preferences> search for maven > Local Repository > Click the override button> update the new location 
+
+-  It is sometimes necessary to stop only a specific thread at a break point. For example, you might want to see the 
+Spark UI at particular break point during your debugging session. In that case right click on the break point and select
+"Thread" option. This will cause the execution in the current thread to pause and will let the other threads continue.   
+
+- When running unit tests for the whole project or a module you might run something like this in a terminal
+    ```
+    $ ./build/mvn -pl sql/hive clean test
+    ``` 
+    You can run the same in Intellij instead of terminal. 
+    Go to Maven Tab > Click on the 'm'(execute maven goal) icon and paste your maven command.
+    The advantage of running there maven command here instead of the terminal is that you get a clickable stacktrace 
+    when there are test failures. This helps in navigating the code easily. You might want to increase the size limit
+    of your console output so that the logs will not be swallowed. Typically, setting it to 10MB is good enough. 

Review comment:
       I'm not sure how much this adds. You don't need to do this with IJ for example. Just modify and run tests inside the IDE?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org