You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/10 23:04:43 UTC

[GitHub] [spark] dilipbiswal opened a new pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

dilipbiswal opened a new pull request #28493:
URL: https://github.com/apache/spark/pull/28493


   ### What changes were proposed in this pull request?
   Currently QueryExecution.debug.toFile dumps the query plan information in a fixed format. This PR adds an additional explain mode parameter that writes the debug information as per the user supplied format.
   ```
   df.queryExecution.debug.toFile("/tmp/plan.txt", explainMode = ExplainMode.fromString("formatted"))
   ```
   ```
   == Physical Plan ==
   * Filter (2)
   +- Scan hive default.s1 (1)
   
   
   (1) Scan hive default.s1
   Output [2]: [c1#15, c2#16]
   Arguments: [c1#15, c2#16], HiveTableRelation `default`.`s1`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#15, c2#16]
   
   (2) Filter [codegen id : 1]
   Input [2]: [c1#15, c2#16]
   Condition : (isnotnull(c1#15) AND (c1#15 > 0))
   
   
   == Whole Stage Codegen ==
   Found 1 WholeStageCodegen subtrees.
   == Subtree 1 / 1 (maxMethodCodeSize:220; maxConstantPoolSize:105(0.16% used); numInnerClasses:0) ==
   *(1) Filter (isnotnull(c1#15) AND (c1#15 > 0))
   +- Scan hive default.s1 [c1#15, c2#16], HiveTableRelation `default`.`s1`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#15, c2#16]
   
   Generated code:
   /* 001 */ public Object generate(Object[] references) {
   /* 002 */   return new GeneratedIteratorForCodegenStage1(references);
   /* 003 */ }
   /* 004 */
   /* 005 */ // codegenStageId=1
   /* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
   /* 007 */   private Object[] references;
   /* 008 */   private scala.collection.Iterator[] inputs;
   /* 009 */   private scala.collection.Iterator inputadapter_input_0;
   /* 010 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] filter_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1];
   /* 011 */
   /* 012 */   public GeneratedIteratorForCodegenStage1(Object[] references) {
   /* 013 */     this.references = references;
   /* 014 */   }
   /* 015 */
   /* 016 */   public void init(int index, scala.collection.Iterator[] inputs) {
   /* 017 */     partitionIndex = index;
   /* 018 */     this.inputs = inputs;
   /* 019 */     inputadapter_input_0 = inputs[0];
   /* 020 */     filter_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0);
   /* 021 */
   /* 022 */   }
   /* 023 */
   /* 024 */   protected void processNext() throws java.io.IOException {
   /* 025 */     while ( inputadapter_input_0.hasNext()) {
   /* 026 */       InternalRow inputadapter_row_0 = (InternalRow) inputadapter_input_0.next();
   /* 027 */
   /* 028 */       do {
   /* 029 */         boolean inputadapter_isNull_0 = inputadapter_row_0.isNullAt(0);
   /* 030 */         int inputadapter_value_0 = inputadapter_isNull_0 ?
   /* 031 */         -1 : (inputadapter_row_0.getInt(0));
   /* 032 */
   /* 033 */         boolean filter_value_2 = !inputadapter_isNull_0;
   /* 034 */         if (!filter_value_2) continue;
   /* 035 */
   /* 036 */         boolean filter_value_3 = false;
   /* 037 */         filter_value_3 = inputadapter_value_0 > 0;
   /* 038 */         if (!filter_value_3) continue;
   /* 039 */
   /* 040 */         ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
   /* 041 */
   /* 042 */         boolean inputadapter_isNull_1 = inputadapter_row_0.isNullAt(1);
   /* 043 */         int inputadapter_value_1 = inputadapter_isNull_1 ?
   /* 044 */         -1 : (inputadapter_row_0.getInt(1));
   /* 045 */         filter_mutableStateArray_0[0].reset();
   /* 046 */
   /* 047 */         filter_mutableStateArray_0[0].zeroOutNullBytes();
   /* 048 */
   /* 049 */         filter_mutableStateArray_0[0].write(0, inputadapter_value_0);
   /* 050 */
   /* 051 */         if (inputadapter_isNull_1) {
   /* 052 */           filter_mutableStateArray_0[0].setNullAt(1);
   /* 053 */         } else {
   /* 054 */           filter_mutableStateArray_0[0].write(1, inputadapter_value_1);
   /* 055 */         }
   /* 056 */         append((filter_mutableStateArray_0[0].getRow()));
   /* 057 */
   /* 058 */       } while(false);
   /* 059 */       if (shouldStop()) return;
   /* 060 */     }
   /* 061 */   }
   /* 062 */
   /* 063 */ }
   ```
   ### Why are the changes needed?
   Hopefully enhances the usability of debug.toFile(..)
   
   ### Does this PR introduce any user-facing change?
   No
   
   ### How was this patch tested?
   Added a test in QueryExecutionSuite
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-626403637






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633184530






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633668937






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-626572248


   +1 too


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633668501


   **[Test build #123095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123095/testReport)** for PR 28493 at commit [`1fea0e2`](https://github.com/apache/spark/commit/1fea0e2f1c7909083973027acd46b1e37eb41d21).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28493:
URL: https://github.com/apache/spark/pull/28493#discussion_r422892179



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -249,18 +277,25 @@ class QueryExecution(
      * Dumps debug information about query execution into the specified file.
      *
      * @param maxFields maximum number of fields converted to string representation.
+     * @param explainMode the explain mode to be used to generate the string
+     *                    representation of the plan.
      */
-    def toFile(path: String, maxFields: Int = Int.MaxValue): Unit = {
+    def toFile(path: String,
+        maxFields: Int = Int.MaxValue,
+        explainMode: ExplainMode = ExtendedMode): Unit = {

Review comment:
       I would let it take a string instead consistently by `ExplainMode.fromString(mode)` since `ExtendedMode` under `execution` is strictly private.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633668937






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maropu commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-626485974


   The change looks useful to me. cc: @MaxGekk 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633189875






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633219204


   **[Test build #123054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123054/testReport)** for PR 28493 at commit [`e5e360f`](https://github.com/apache/spark/commit/e5e360fa21a2be5f4389c7982e83c3fa169ccefc).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633189877


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123048/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633191517






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-626450849






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633184986


   **[Test build #123048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123048/testReport)** for PR 28493 at commit [`e5e360f`](https://github.com/apache/spark/commit/e5e360fa21a2be5f4389c7982e83c3fa169ccefc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #28493:
URL: https://github.com/apache/spark/pull/28493


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-626450849






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633191517






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dilipbiswal commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
dilipbiswal commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633845206


   Thank you very much @HyukjinKwon @maropu @MaxGekk 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633184986


   **[Test build #123048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123048/testReport)** for PR 28493 at commit [`e5e360f`](https://github.com/apache/spark/commit/e5e360fa21a2be5f4389c7982e83c3fa169ccefc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633191409


   **[Test build #123054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123054/testReport)** for PR 28493 at commit [`e5e360f`](https://github.com/apache/spark/commit/e5e360fa21a2be5f4389c7982e83c3fa169ccefc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633191409


   **[Test build #123054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123054/testReport)** for PR 28493 at commit [`e5e360f`](https://github.com/apache/spark/commit/e5e360fa21a2be5f4389c7982e83c3fa169ccefc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633184490


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633668501






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dilipbiswal commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
dilipbiswal commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-626473070


   cc @gatorsmile @maropu 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-626450430


   **[Test build #122484 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122484/testReport)** for PR 28493 at commit [`62dbc83`](https://github.com/apache/spark/commit/62dbc8320641e791b06ba989b029e4d20d724407).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633219415






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dilipbiswal commented on a change in pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
dilipbiswal commented on a change in pull request #28493:
URL: https://github.com/apache/spark/pull/28493#discussion_r422896114



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -249,18 +277,25 @@ class QueryExecution(
      * Dumps debug information about query execution into the specified file.
      *
      * @param maxFields maximum number of fields converted to string representation.
+     * @param explainMode the explain mode to be used to generate the string
+     *                    representation of the plan.
      */
-    def toFile(path: String, maxFields: Int = Int.MaxValue): Unit = {
+    def toFile(path: String,
+        maxFields: Int = Int.MaxValue,
+        explainMode: ExplainMode = ExtendedMode): Unit = {

Review comment:
       @HyukjinKwon Makes sense. Thank you. I will make the change.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-626403637






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633817832


   Merged to master.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633189875


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-626403485


   **[Test build #122484 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122484/testReport)** for PR 28493 at commit [`62dbc83`](https://github.com/apache/spark/commit/62dbc8320641e791b06ba989b029e4d20d724407).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633332300


   @MaxGekk did you find some time to take a look? If not, I will just merge after double checking.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-626403485


   **[Test build #122484 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122484/testReport)** for PR 28493 at commit [`62dbc83`](https://github.com/apache/spark/commit/62dbc8320641e791b06ba989b029e4d20d724407).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633219415






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633184530






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maropu commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633191204


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28493: [SPARK-31673][SQL] QueryExection.debug.toFile() to take an addtional explain mode param

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28493:
URL: https://github.com/apache/spark/pull/28493#issuecomment-633189836


   **[Test build #123048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123048/testReport)** for PR 28493 at commit [`e5e360f`](https://github.com/apache/spark/commit/e5e360fa21a2be5f4389c7982e83c3fa169ccefc).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org